Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/static-frame/static-frame
Immutable and statically-typeable DataFrames with runtime type and data validation
https://github.com/static-frame/static-frame
arrays dataframes immutable-collections immutable-data-structures python
Last synced: 6 days ago
JSON representation
Immutable and statically-typeable DataFrames with runtime type and data validation
- Host: GitHub
- URL: https://github.com/static-frame/static-frame
- Owner: static-frame
- License: other
- Created: 2018-01-03T15:07:52.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2024-10-29T17:30:36.000Z (about 2 months ago)
- Last Synced: 2024-10-29T18:54:58.460Z (about 2 months ago)
- Topics: arrays, dataframes, immutable-collections, immutable-data-structures, python
- Language: Python
- Homepage: https://staticframe.dev
- Size: 29.7 MB
- Stars: 437
- Watchers: 11
- Forks: 31
- Open Issues: 59
-
Metadata Files:
- Readme: README.rst
- Contributing: CONTRIBUTING.rst
- License: LICENSE.txt
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
- best-of-python - GitHub - 7% open · ⏱️ 21.05.2024): (Data Containers & Dataframes)
README
.. figure:: https://raw.githubusercontent.com/static-frame/static-frame/master/doc/images/sf-logo-web_icon-small.png
:align: center
.. image:: https://img.shields.io/pypi/pyversions/static-frame.svg
:target: https://pypi.org/project/static-frame
.. image:: https://img.shields.io/pypi/v/static-frame.svg
:target: https://pypi.org/project/static-frame
.. image:: https://img.shields.io/conda/vn/conda-forge/static-frame.svg
:target: https://anaconda.org/conda-forge/static-frame
.. image:: https://img.shields.io/codecov/c/github/static-frame/static-frame.svg
:target: https://codecov.io/gh/static-frame/static-frame
.. image:: https://img.shields.io/github/actions/workflow/status/static-frame/static-frame/ci.yml?branch=master&label=test&logo=Github
:target: https://github.com/static-frame/static-frame/actions/workflows/ci.yml
.. image:: https://img.shields.io/readthedocs/static-frame.svg
:target: https://static-frame.readthedocs.io/en/latest
.. image:: https://img.shields.io/badge/hypothesis-tested-brightgreen.svg
:target: https://hypothesis.readthedocs.io
.. image:: https://img.shields.io/pypi/status/static-frame.svg
:target: https://pypi.org/project/static-frame
.. image:: https://img.shields.io/badge/launch-binder-579ACA.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFkAAABZCAMAAABi1XidAAAB8lBMVEX///9XmsrmZYH1olJXmsr1olJXmsrmZYH1olJXmsr1olJXmsrmZYH1olL1olJXmsr1olJXmsrmZYH1olL1olJXmsrmZYH1olJXmsr1olL1olJXmsrmZYH1olL1olJXmsrmZYH1olL1olL0nFf1olJXmsrmZYH1olJXmsq8dZb1olJXmsrmZYH1olJXmspXmspXmsr1olL1olJXmsrmZYH1olJXmsr1olL1olJXmsrmZYH1olL1olLeaIVXmsrmZYH1olL1olL1olJXmsrmZYH1olLna31Xmsr1olJXmsr1olJXmsrmZYH1olLqoVr1olJXmsr1olJXmsrmZYH1olL1olKkfaPobXvviGabgadXmsqThKuofKHmZ4Dobnr1olJXmsr1olJXmspXmsr1olJXmsrfZ4TuhWn1olL1olJXmsqBi7X1olJXmspZmslbmMhbmsdemsVfl8ZgmsNim8Jpk8F0m7R4m7F5nLB6jbh7jbiDirOEibOGnKaMhq+PnaCVg6qWg6qegKaff6WhnpKofKGtnomxeZy3noG6dZi+n3vCcpPDcpPGn3bLb4/Mb47UbIrVa4rYoGjdaIbeaIXhoWHmZYHobXvpcHjqdHXreHLroVrsfG/uhGnuh2bwj2Hxk17yl1vzmljzm1j0nlX1olL3AJXWAAAAbXRSTlMAEBAQHx8gICAuLjAwMDw9PUBAQEpQUFBXV1hgYGBkcHBwcXl8gICAgoiIkJCQlJicnJ2goKCmqK+wsLC4usDAwMjP0NDQ1NbW3Nzg4ODi5+3v8PDw8/T09PX29vb39/f5+fr7+/z8/Pz9/v7+zczCxgAABC5JREFUeAHN1ul3k0UUBvCb1CTVpmpaitAGSLSpSuKCLWpbTKNJFGlcSMAFF63iUmRccNG6gLbuxkXU66JAUef/9LSpmXnyLr3T5AO/rzl5zj137p136BISy44fKJXuGN/d19PUfYeO67Znqtf2KH33Id1psXoFdW30sPZ1sMvs2D060AHqws4FHeJojLZqnw53cmfvg+XR8mC0OEjuxrXEkX5ydeVJLVIlV0e10PXk5k7dYeHu7Cj1j+49uKg7uLU61tGLw1lq27ugQYlclHC4bgv7VQ+TAyj5Zc/UjsPvs1sd5cWryWObtvWT2EPa4rtnWW3JkpjggEpbOsPr7F7EyNewtpBIslA7p43HCsnwooXTEc3UmPmCNn5lrqTJxy6nRmcavGZVt/3Da2pD5NHvsOHJCrdc1G2r3DITpU7yic7w/7Rxnjc0kt5GC4djiv2Sz3Fb2iEZg41/ddsFDoyuYrIkmFehz0HR2thPgQqMyQYb2OtB0WxsZ3BeG3+wpRb1vzl2UYBog8FfGhttFKjtAclnZYrRo9ryG9uG/FZQU4AEg8ZE9LjGMzTmqKXPLnlWVnIlQQTvxJf8ip7VgjZjyVPrjw1te5otM7RmP7xm+sK2Gv9I8Gi++BRbEkR9EBw8zRUcKxwp73xkaLiqQb+kGduJTNHG72zcW9LoJgqQxpP3/Tj//c3yB0tqzaml05/+orHLksVO+95kX7/7qgJvnjlrfr2Ggsyx0eoy9uPzN5SPd86aXggOsEKW2Prz7du3VID3/tzs/sSRs2w7ovVHKtjrX2pd7ZMlTxAYfBAL9jiDwfLkq55Tm7ifhMlTGPyCAs7RFRhn47JnlcB9RM5T97ASuZXIcVNuUDIndpDbdsfrqsOppeXl5Y+XVKdjFCTh+zGaVuj0d9zy05PPK3QzBamxdwtTCrzyg/2Rvf2EstUjordGwa/kx9mSJLr8mLLtCW8HHGJc2R5hS219IiF6PnTusOqcMl57gm0Z8kanKMAQg0qSyuZfn7zItsbGyO9QlnxY0eCuD1XL2ys/MsrQhltE7Ug0uFOzufJFE2PxBo/YAx8XPPdDwWN0MrDRYIZF0mSMKCNHgaIVFoBbNoLJ7tEQDKxGF0kcLQimojCZopv0OkNOyWCCg9XMVAi7ARJzQdM2QUh0gmBozjc3Skg6dSBRqDGYSUOu66Zg+I2fNZs/M3/f/Grl/XnyF1Gw3VKCez0PN5IUfFLqvgUN4C0qNqYs5YhPL+aVZYDE4IpUk57oSFnJm4FyCqqOE0jhY2SMyLFoo56zyo6becOS5UVDdj7Vih0zp+tcMhwRpBeLyqtIjlJKAIZSbI8SGSF3k0pA3mR5tHuwPFoa7N7reoq2bqCsAk1HqCu5uvI1n6JuRXI+S1Mco54YmYTwcn6Aeic+kssXi8XpXC4V3t7/ADuTNKaQJdScAAAAAElFTkSuQmCC
:target: https://mybinder.org/v2/gh/static-frame/static-frame-ftgu/default?urlpath=tree/index.ipynb
static-frame
=============
Immutable and statically-typeable DataFrames with runtime type and data validation.
Among the many Python DataFrame libraries, StaticFrame is an alternative that prioritizes correctness, maintainability, and reducing opportunities for error. Key features include:
* 🛡️ Immutable Data: Provides memory efficiency, excellent performance, and prohibits side effects.
* 🗜️ Static Typing: Use Python type-hints to statically type index, columns, and columnar types.
* 🚦 Runtime Validation: Use type hints and specialized validators for runtime type and data checks.
* 🧭 Consistent Interface: An easy-to-learn, hierarchical, and intuitive API that avoids the many inconsistencies of Pandas.
* 🧬 Comprehensive ``dtype`` Support: Full compatibility with all NumPy dtypes and datetime64 units.
* 🔗 Broad Interoperability: Translate between Pandas, DuckDB, Arrow, Parquet, CSV, TSV, JSON, MessagePack, Excel XLSX, SQLite, HDF5, and NumPy; output to xarray, VisiData, HTML, RST, Markdown, LaTeX, and Jupyter notebooks.
* 🚀 Optimized Serialization & Memory Mapping: Fast disk I/O with custom NPZ and NPY encodings.
* 💼 Multi-Table Containers: The ``Bus`` and ``Yarn`` provide interfaces to collections of tables with lazy data loading, well-suited for large datasets.
* ⏳ Deferred Processing: The ``Batch`` provides a common interface for deferred processing of groups, windows, or any iterator.
* 🪶 Lean Dependencies: Core functionality relies only on NumPy and team-maintained C-extensions.
* 📚 Comprehensive Documentation: All API endpoints documented with thousands of easily runnable examples.
Code: https://github.com/static-frame/static-frame
Docs: http://static-frame.readthedocs.io
Packages: https://pypi.org/project/static-frame
API Search: https://staticframe.dev
Jupyter Notebook Tutorial: `Launch Binder `_
Installation via ``pip``
-------------------------------
Install StaticFrame with ``pip``. Note that pre-built wheels are published for all supported Python versions and platforms (including Apple Silicon platforms)::
pip install static-frame
To install optional dependencies for full support of input and output formats (such as XLSX and HDF5) via ``pip``::
pip install static-frame [extras]
Installation via ``conda``
-------------------------------
StaticFrame can be installed via ``conda`` with the ``conda-forge`` channel. Note that pre-built wheels of StaticFrame and all compiled dependencies are available through ``pip`` and may offer more compatibility than a ``conda``-based installation ::
conda install -c conda-forge static-frame
Installation via Pyodide
-------------------------------
StaticFrame can be run in the browser via Pyodide with the ``static_frame_pyodide`` package: https://github.com/static-frame/static-frame-pyodide
Dependencies
--------------
Core StaticFrame requires the following:
- Python>=3.9
- numpy>=1.23.5 (numpy>=2 is supported)
- arraymap==0.4.0
- arraykit==0.10.0
- typing-extensions>=4.12.0
For extended input and output, the following packages are required:
- pandas>=1.1.5
- duckdb>=1.0.0
- xlsxwriter>=1.1.2
- openpyxl>=3.0.9
- xarray>=0.13.0
- tables>=3.9.1
- pyarrow>=3.0.0
- visidata>=2.4
Quick-Start Guide
---------------------
To get startred quickly, let's download the classic iris (flower) characteristics data set and build a simple naive Bayes classifier that can predict species from iris petal characteristics.
While StaticFrame's API has over 7,500 endpoints, much will be familiar to users of Pandas or other DataFrame libraries. Rather than offering fewer interfaces with greater configurability, StaticFrame favors more numerous interfaces with more narrow parameters and functionality. This design leads to more maintainable code. (Read more about differences between Pandas and StaticFrame `here `__.)
We can download the data set from the UCI Machine Learning Repository and create a ``Frame``. StaticFrame exposes all constructors on the class: here, we will use the ``Frame.from_csv()`` constructor. To download a file from the internet and provide it to a constructor, we can use StaticFrame's ``WWW.from_file()`` interface::
>>> import static_frame as sf
>>> data = sf.Frame.from_csv(sf.WWW.from_file('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'), columns_depth=0)
Each record (or row) in this dataset describes observations of an iris flower, including its sepal and petal characteristics, as well as its species (of which there are three). To display just the first few rows, we can use the ``head()`` method. Notice that StaticFrame's default display makes it very clear what type of ``Frame``, ``Index``, and NumPy datatypes are present::
>>> data.head()
0 1 2 3 4
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
<
As the columns are unlabelled, let's next add column labels. StaticFrame supports reindexing (conforming existing axis labels to new labels, potentially changing the size and ordering) and relabeling (simply applying new labels without regard to existing labels). As we can ignore the default column labels (auto-incremented integers), the ``relabel()`` method is used to provide new labels.
Note that while ``relabel()`` creates a new ``Frame``, underlying NumPy data is not copied. As all NumPy data is immutable in StaticFrame, we can reuse it in our new container, making such operations very efficient::
>>> data = data.relabel(columns=('sepal_l', 'sepal_w', 'petal_l', 'petal_w', 'species'))
>>> data.head()
sepal_l sepal_w petal_l petal_w species <
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
<
(Read more about no-copy operations `here `__.)
For this example, eighty percent of the data will be used to train the classifier; the remaining twenty percent will be used to test the classifier. As all records are labelled with the known species, we can conclude by measuring the effectiveness of the classifier on the test data.
To divide the data into two groups, we create a ``Series`` of contiguous integers and then extract a random selection of 80% of the values into a new ``Series``, here named ``sel_train``. This will be used to select our traning data. As the ``sample()`` method, given a count, randomly samples that many values, your results will be different unless use the same ``seed`` argument::
>>> sel = sf.Series(np.arange(len(data)))
>>> sel_train = sel.sample(round(len(data) * .8), seed=42)
>>> sel_train.head()
0 0
2 2
3 3
4 4
5 5
We will create another ``Series`` to select the test data. The ``drop[]`` interface can be used to create a new ``Series`` that excludes the training selections, leaving just the testing selections. As with many interfaces in StaticFrame (such as ``astype`` and ``assign``), brackets can be used to do ``loc[]`` style selections::
>>> sel_test = sel.drop[sel_train]
>>> sel_test.head()
1 1
14 14
20 20
21 21
37 37
To select a subset of the data for training, the ``sel_train`` ``Series`` can be passed to ``loc[]`` to select just those rows::
>>> data_train = data.loc[sel_train]
>>> data_train.head()
sepal_l sepal_w petal_l petal_w species <
0 5.1 3.5 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
5 5.4 3.9 1.7 0.4 Iris-setosa
<
With our data divided into two randomly-selected, non-overlapping groups, we can proceed to implement the naive Bayes classifier. We will compute the ``posterior`` of the test data by multiplying the ``prior`` and the ``likelihood``. With the ``posterior``, we can determine which species the classifier has calculated is most likely. (More on naive Bayes classifiers can be found `here `__.)
The ``prior`` is calculated as the percentage of samples of each species in the training data. This is the "normalized" count per species. To get a ``Series`` of counts per species, we can select the species column, iterate over groups based on species name, and count the size of each group.
In StaticFrame, this can be done by calling ``Series.iter_group_items()`` to get an iterator of pairs of group label, group (where the group is a ``Series``). This iterator (or any similar iterator) can be given to a ``Batch``, a chaining processor of ``Frame`` or ``Series``, to perform operations on each group. (For more on the ``Batch`` and other higher-order containers in StaticFrame, see `here `__.)
Once the ``Batch`` is created, selections, method calls, and operator expressions can be chained as if they were being called on a single container. Processing happens to every contained container, and a container is returned, only when a finalizer method, such as ``to_series()``, is called::
>>> counts = sf.Batch(data_train['species'].iter_group_items()).count().to_series()
>>> counts
Iris-setosa 43
Iris-versicolor 39
Iris-virginica 38
<
As with NumPy, StaticFrame containers can be used in expressions with binary operators. The ``prior`` can be derived by dividing ``counts`` by the size of the training data. This returns a ``Series`` of the percentage of records per species::
>>> prior = counts / len(data_train)
>>> prior
Iris-setosa 0.35833333333333334
Iris-versicolor 0.325
Iris-virginica 0.31666666666666665
<
Having calculated the ``prior``, we can calculate ``likelihood`` next. To calculate ``likelihood``, we will call a probability distribution function (imported from SciPy) with the test data, once for each species, given the characteristics (mean and standard deviation) observed in the test data for that species.
The ``Batch`` can again be used to calculate the mean and standard deviation, per species, from the training data. With the ``Frame`` of training data, we call ``iter_group_items()`` to group by species and, passing that iterator to ``Batch``, call ``mean()`` (assigned to ``mu``) or ``std()`` (assigned to ``sigma``). Note that ``iter_group_items()`` has an optional ``drop`` parameter to remove the column used for grouping from subsequent operations::
>>> mu = sf.Batch(data_train[['sepal_l', 'sepal_w', 'species']].iter_group_items('species', drop=True)).mean().to_frame()
>>> mu
sepal_l sepal_w <
Iris-setosa 4.986046511627907 3.434883720930233
Iris-versicolor 5.920512820512819 2.771794871794872
Iris-virginica 6.6078947368421055 2.9763157894736842
<
>>> sigma = sf.Batch(data_train[['sepal_l', 'sepal_w', 'species']].iter_group_items('species', drop=True)).std(ddof=1).to_frame()
>>> sigma
sepal_l sepal_w <
Iris-setosa 0.3419700595003668 0.3477024733400345
Iris-versicolor 0.508444214804487 0.33082728674826684
Iris-virginica 0.6055516042229233 0.3513942965328924
<
For a unified display of these characteristics, we can build a hierarchical index on each ``Frame`` with ``relabel_level_add()`` (adding the "mu" or "sigma" labels), then vertically concatenate the tables. As StaticFrame always requires unique labels in indices, adding an additional label is required before concatenation. The built-in ``round`` function can be used for more tidy display::
>>> stats = sf.Frame.from_concat((mu.relabel_level_add('mu'), sigma.relabel_level_add('sigma')))
>>> round(stats, 2)
sepal_l sepal_w <
mu Iris-setosa 4.99 3.43
mu Iris-versicolor 5.92 2.77
mu Iris-virginica 6.61 2.98
sigma Iris-setosa 0.34 0.35
sigma Iris-versicolor 0.51 0.33
sigma Iris-virginica 0.61 0.35
< <
We can now move on to processing the test data with the characteristics derived from the training data. To do that, we will extract our previously selected test records with ``sel_test`` into a new ``Frame``, to which we can add our ``posterior`` predictions and final species classifications.
It is common to process data in table by adding columns from left to right. StaticFrame permits this limited form of mutability with the grow-only ``FrameGO``. While underlying NumPy arrays are still always immutable, columns can be added to a ``FrameGO`` with bracket-style assignments. A ``FrameGO`` can be created from a ``Frame`` with the ``to_frame_go()`` method. As mentioned elsewhere, underlying immutable NumPy arrays are not copied: this is an efficient, no-copy operation.
Passing two arguments to ``loc[]``, we can select rows with the values from ``sel_test``, and we can select columns with a list of labels for the sepal length and sepal width::
>>> data_test = data.loc[sel_test.values, ['sepal_l', 'sepal_w']].to_frame_go()
>>> data_test.head()
sepal_l sepal_w <
1 4.9 3.0
14 5.8 4.0
20 5.4 3.4
21 5.1 3.7
37 4.9 3.1
StaticFrame interfaces make extensive use of iterators and generators. As used below, the ``Frame.from_fields()`` constructor will create a ``Frame`` from any iterable (or generator) of column arrays.
The ``likelihood_of_species()`` function (defined below), for each index label in ``mu`` (which provides each unique iris species), calculates a probability density function for the test data, given the ``mu`` (mean) and ``sigma`` (standard deviation) for the species. An array of the sum of the log is yielded::
>>> from scipy.stats import norm
>>> def likelihood_of_species():
... for label in mu.index:
... pdf = norm.pdf(data_test.values, mu.loc[label], sigma.loc[label])
... yield np.log(pdf).sum(axis=1)
While the generator function above is easy to read, it is hard to copy and paste. If you are following along, using the one-line generator expression, below, will be easier. The two are equivalent:
>>> likelihood_of_species = (np.log(norm.pdf(data_test.values, mu.loc[label], sigma.loc[label])).sum(axis=1) for label in mu.index)
With this generator expression defined, we call the ``from_fields`` constructor to produce the ``likelihood`` table, providing column labels from ``mu.index`` and index labels from ``data_test.index``. For each test record row we now have a likelihood per species::
>>> likelihood = sf.Frame.from_fields(likelihood_of_species, columns=mu.index, index=data_test.index)
>>> round(likelihood.head(), 2)
Iris-setosa Iris-versicolor Iris-virginica <
1 -0.52 -2.31 -4.27
14 -3.86 -6.97 -5.42
20 -0.45 -2.38 -3.01
21 -0.05 -5.29 -5.51
37 -0.2 -2.56 -4.33
We can calculate the ``posterior`` by multiplying ``likelihood`` by ``prior``. Whenever performing binary operations on ``Frame`` and ``Series``, indices will be aligned and, if necessary, reindexed before processing::
>>> posterior = likelihood * prior
>>> round(posterior.head(), 2)
Iris-setosa Iris-versicolor Iris-virginica <
1 -0.19 -0.75 -1.35
14 -1.38 -2.27 -1.72
20 -0.16 -0.77 -0.95
21 -0.02 -1.72 -1.75
37 -0.07 -0.83 -1.37
We can now add columns to our ``data_test`` ``FrameGO``. To determine our best prediction of species for each row of the test data, the column label (the species) of the maximum a posteriori estimate is selected with ``loc_max()``::
>>> data_test['predict'] = posterior.loc_max(axis=1)
>>> data_test.head()
sepal_l sepal_w predict <
1 4.9 3.0 Iris-setosa
14 5.8 4.0 Iris-setosa
20 5.4 3.4 Iris-setosa
21 5.1 3.7 Iris-setosa
37 4.9 3.1 Iris-setosa
<
We can add two additional columns to evaluate the effectivess of the classifier. First, we can add an "observed" column by adding the original "species" column from the original ``data`` ``Frame``. In assigning a ``Series`` to a ``Frame``, only values found in the intersection of the indices will be added as a column::
>>> data_test['observed'] = data['species']
>>> data_test.head()
sepal_l sepal_w predict observed <
1 4.9 3.0 Iris-setosa Iris-setosa
14 5.8 4.0 Iris-setosa Iris-setosa
20 5.4 3.4 Iris-setosa Iris-setosa
21 5.1 3.7 Iris-setosa Iris-setosa
37 4.9 3.1 Iris-setosa Iris-setosa
< <
Having populated a column of predicted and observed values, we can compare the two to get a Boolean column indicating when the classifier calculated a correct predicton::
>>> data_test['correct'] = data_test['predict'] == data_test['observed']
>>> data_test.tail()
sepal_l sepal_w predict observed correct <
129 7.2 3.0 Iris-virginica Iris-virginica True
130 7.4 2.8 Iris-virginica Iris-virginica True
140 6.7 3.1 Iris-virginica Iris-virginica True
144 6.7 3.3 Iris-virginica Iris-virginica True
149 5.9 3.0 Iris-versicolor Iris-virginica False
< <
To find the percentage of correct classifications among the test data, we can sum the ``correct`` Boolean column and divide that by the size of the test data::
>>> data_test["correct"].sum() / len(data_test)
0.7333333333333333
This simple naive Bayes classifier can predict iris species correctly about 73% of the time.
For further introduction to StaticFrame, including links to articles, videos, and documentation, see `here `__.