I have no special insight in pandas development, so this is just my guess.
I believe this was because Pandas' initial primary use case was manipulating time series (datetime-indexed numerical vectors), which are used extensively in financial institutions such as hedge funds and trading firms (Pandas was initiated as a skunkwork project in AQR Capoital Management). You can see the lineage in its very extensive collection of convenience methods for manipulating time series (pd.Series where the index is some datetime type). A pd.Series is a numpy array with a meaningful index, and a pd.DataFrame is a collection of pd.Series with a shared index. If you use Dataframe to store and manipulate multivariate time series, the api is quite sensible. So pd.Series and pd.DataFrame are probably the datatype that you'd design to store time series of (a portfolio of) stock returns. (Old foggies who'd use something like Matlab before know that just being sure your vector/matrix calculations are using correctly aligned dates was not a given.)
Dplyr and its R data.frame heritage are what statisticians would probably use to record measurements / experimental outcomes on individuals. There is usually no meaningful index/natural primary key, and the order usually doesn't matter. It's much closer to a relational database table (unordered collection of tuples), but for analytics rather than transactions so column- rather than row-oriented.
For data tables without a meaningful natural index, the Pandas api is much more confusing and cumbersome than needed. It happens that a lot of ML applications fall in that category, but during the early 2010s when Panndas took of it had very little competition.
I concur wholeheartedly. The use of indexes in Pandas makes working with timerseries data so much nicer and that's why indexing is an important part of the API.
Really interesting. I would argue that a lot of datapoints (maybe all) in fact do have a natural index. It is the fact if you are talking about Relational databases (every tables has a primary key). In a lot of scenarios it helped me a lot to think as the index as an associated pk of some database table.
Comments
I have no special insight in pandas development, so this is just my guess.
I believe this was because Pandas' initial primary use case was manipulating time series (datetime-indexed numerical vectors), which are used extensively in financial institutions such as hedge funds and trading firms (Pandas was initiated as a skunkwork project in AQR Capoital Management). You can see the lineage in its very extensive collection of convenience methods for manipulating time series (pd.Series where the index is some datetime type). A pd.Series is a numpy array with a meaningful index, and a pd.DataFrame is a collection of pd.Series with a shared index. If you use Dataframe to store and manipulate multivariate time series, the api is quite sensible. So pd.Series and pd.DataFrame are probably the datatype that you'd design to store time series of (a portfolio of) stock returns. (Old foggies who'd use something like Matlab before know that just being sure your vector/matrix calculations are using correctly aligned dates was not a given.)
Dplyr and its R data.frame heritage are what statisticians would probably use to record measurements / experimental outcomes on individuals. There is usually no meaningful index/natural primary key, and the order usually doesn't matter. It's much closer to a relational database table (unordered collection of tuples), but for analytics rather than transactions so column- rather than row-oriented.
For data tables without a meaningful natural index, the Pandas api is much more confusing and cumbersome than needed. It happens that a lot of ML applications fall in that category, but during the early 2010s when Panndas took of it had very little competition.
The name pandas was derived from panel data (data sets with observations over multiple time periods).
https://www.dlr.de/sc/Portaldata/15/Resources/dokumente/pyhp...
I concur wholeheartedly. The use of indexes in Pandas makes working with timerseries data so much nicer and that's why indexing is an important part of the API.
Really interesting. I would argue that a lot of datapoints (maybe all) in fact do have a natural index. It is the fact if you are talking about Relational databases (every tables has a primary key). In a lot of scenarios it helped me a lot to think as the index as an associated pk of some database table.