{"id":13532903,"url":"https://github.com/static-frame/static-frame","last_synced_at":"2025-04-14T03:56:16.456Z","repository":{"id":37381945,"uuid":"116150224","full_name":"static-frame/static-frame","owner":"static-frame","description":"Immutable and statically-typeable DataFrames with runtime type and data validation","archived":false,"fork":false,"pushed_at":"2025-04-03T22:07:25.000Z","size":29599,"stargazers_count":457,"open_issues_count":50,"forks_count":34,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-10T04:02:06.770Z","etag":null,"topics":["arrays","dataframes","immutable-collections","immutable-data-structures","python"],"latest_commit_sha":null,"homepage":"https://staticframe.dev","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/static-frame.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-03T15:07:52.000Z","updated_at":"2025-04-09T03:23:22.000Z","dependencies_parsed_at":"2023-10-02T20:16:01.596Z","dependency_job_id":"5bc6ad3c-1e15-4a2e-9dea-0cb81dcee3c2","html_url":"https://github.com/static-frame/static-frame","commit_stats":{"total_commits":5499,"total_committers":26,"mean_commits":211.5,"dds":"0.10038188761593014","last_synced_commit":"07764ae03172f7d23efcb81d996407070cf0d035"},"previous_names":["investmentsystems/static-frame"],"tags_count":193,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/static-frame%2Fstatic-frame","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/static-frame%2Fstatic-frame/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/static-frame%2Fstatic-frame/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/static-frame%2Fstatic-frame/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/static-frame","download_url":"https://codeload.github.com/static-frame/static-frame/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248819378,"owners_count":21166476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arrays","dataframes","immutable-collections","immutable-data-structures","python"],"created_at":"2024-08-01T07:01:14.763Z","updated_at":"2025-04-14T03:56:16.424Z","avatar_url":"https://github.com/static-frame.png","language":"Python","readme":".. figure:: https://raw.githubusercontent.com/static-frame/static-frame/master/doc/images/sf-logo-web_icon-small.png\n   :align: center\n\n\n.. image:: https://img.shields.io/pypi/pyversions/static-frame.svg\n  :target: https://pypi.org/project/static-frame\n\n.. image:: https://img.shields.io/pypi/v/static-frame.svg\n  :target: https://pypi.org/project/static-frame\n\n.. image:: https://img.shields.io/conda/vn/conda-forge/static-frame.svg\n  :target: https://anaconda.org/conda-forge/static-frame\n\n\n.. image:: https://img.shields.io/codecov/c/github/static-frame/static-frame.svg\n  :target: https://codecov.io/gh/static-frame/static-frame\n\n\n.. image:: https://img.shields.io/github/actions/workflow/status/static-frame/static-frame/ci.yml?branch=master\u0026label=test\u0026logo=Github\n  :target: https://github.com/static-frame/static-frame/actions/workflows/ci.yml\n\n\n.. image:: https://img.shields.io/readthedocs/static-frame.svg\n  :target: https://static-frame.readthedocs.io/en/latest\n\n.. image:: https://img.shields.io/badge/hypothesis-tested-brightgreen.svg\n  :target: https://hypothesis.readthedocs.io\n\n.. image:: https://img.shields.io/pypi/status/static-frame.svg\n  :target: https://pypi.org/project/static-frame\n\n\n.. image:: https://img.shields.io/badge/launch-binder-579ACA.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFkAAABZCAMAAABi1XidAAAB8lBMVEX///9XmsrmZYH1olJXmsr1olJXmsrmZYH1olJXmsr1olJXmsrmZYH1olL1olJXmsr1olJXmsrmZYH1olL1olJXmsrmZYH1olJXmsr1olL1olJXmsrmZYH1olL1olJXmsrmZYH1olL1olL0nFf1olJXmsrmZYH1olJXmsq8dZb1olJXmsrmZYH1olJXmspXmspXmsr1olL1olJXmsrmZYH1olJXmsr1olL1olJXmsrmZYH1olL1olLeaIVXmsrmZYH1olL1olL1olJXmsrmZYH1olLna31Xmsr1olJXmsr1olJXmsrmZYH1olLqoVr1olJXmsr1olJXmsrmZYH1olL1olKkfaPobXvviGabgadXmsqThKuofKHmZ4Dobnr1olJXmsr1olJXmspXmsr1olJXmsrfZ4TuhWn1olL1olJXmsqBi7X1olJXmspZmslbmMhbmsdemsVfl8ZgmsNim8Jpk8F0m7R4m7F5nLB6jbh7jbiDirOEibOGnKaMhq+PnaCVg6qWg6qegKaff6WhnpKofKGtnomxeZy3noG6dZi+n3vCcpPDcpPGn3bLb4/Mb47UbIrVa4rYoGjdaIbeaIXhoWHmZYHobXvpcHjqdHXreHLroVrsfG/uhGnuh2bwj2Hxk17yl1vzmljzm1j0nlX1olL3AJXWAAAAbXRSTlMAEBAQHx8gICAuLjAwMDw9PUBAQEpQUFBXV1hgYGBkcHBwcXl8gICAgoiIkJCQlJicnJ2goKCmqK+wsLC4usDAwMjP0NDQ1NbW3Nzg4ODi5+3v8PDw8/T09PX29vb39/f5+fr7+/z8/Pz9/v7+zczCxgAABC5JREFUeAHN1ul3k0UUBvCb1CTVpmpaitAGSLSpSuKCLWpbTKNJFGlcSMAFF63iUmRccNG6gLbuxkXU66JAUef/9LSpmXnyLr3T5AO/rzl5zj137p136BISy44fKJXuGN/d19PUfYeO67Znqtf2KH33Id1psXoFdW30sPZ1sMvs2D060AHqws4FHeJojLZqnw53cmfvg+XR8mC0OEjuxrXEkX5ydeVJLVIlV0e10PXk5k7dYeHu7Cj1j+49uKg7uLU61tGLw1lq27ugQYlclHC4bgv7VQ+TAyj5Zc/UjsPvs1sd5cWryWObtvWT2EPa4rtnWW3JkpjggEpbOsPr7F7EyNewtpBIslA7p43HCsnwooXTEc3UmPmCNn5lrqTJxy6nRmcavGZVt/3Da2pD5NHvsOHJCrdc1G2r3DITpU7yic7w/7Rxnjc0kt5GC4djiv2Sz3Fb2iEZg41/ddsFDoyuYrIkmFehz0HR2thPgQqMyQYb2OtB0WxsZ3BeG3+wpRb1vzl2UYBog8FfGhttFKjtAclnZYrRo9ryG9uG/FZQU4AEg8ZE9LjGMzTmqKXPLnlWVnIlQQTvxJf8ip7VgjZjyVPrjw1te5otM7RmP7xm+sK2Gv9I8Gi++BRbEkR9EBw8zRUcKxwp73xkaLiqQb+kGduJTNHG72zcW9LoJgqQxpP3/Tj//c3yB0tqzaml05/+orHLksVO+95kX7/7qgJvnjlrfr2Ggsyx0eoy9uPzN5SPd86aXggOsEKW2Prz7du3VID3/tzs/sSRs2w7ovVHKtjrX2pd7ZMlTxAYfBAL9jiDwfLkq55Tm7ifhMlTGPyCAs7RFRhn47JnlcB9RM5T97ASuZXIcVNuUDIndpDbdsfrqsOppeXl5Y+XVKdjFCTh+zGaVuj0d9zy05PPK3QzBamxdwtTCrzyg/2Rvf2EstUjordGwa/kx9mSJLr8mLLtCW8HHGJc2R5hS219IiF6PnTusOqcMl57gm0Z8kanKMAQg0qSyuZfn7zItsbGyO9QlnxY0eCuD1XL2ys/MsrQhltE7Ug0uFOzufJFE2PxBo/YAx8XPPdDwWN0MrDRYIZF0mSMKCNHgaIVFoBbNoLJ7tEQDKxGF0kcLQimojCZopv0OkNOyWCCg9XMVAi7ARJzQdM2QUh0gmBozjc3Skg6dSBRqDGYSUOu66Zg+I2fNZs/M3/f/Grl/XnyF1Gw3VKCez0PN5IUfFLqvgUN4C0qNqYs5YhPL+aVZYDE4IpUk57oSFnJm4FyCqqOE0jhY2SMyLFoo56zyo6becOS5UVDdj7Vih0zp+tcMhwRpBeLyqtIjlJKAIZSbI8SGSF3k0pA3mR5tHuwPFoa7N7reoq2bqCsAk1HqCu5uvI1n6JuRXI+S1Mco54YmYTwcn6Aeic+kssXi8XpXC4V3t7/ADuTNKaQJdScAAAAAElFTkSuQmCC\n   :target: https://mybinder.org/v2/gh/static-frame/static-frame-ftgu/default?urlpath=tree/index.ipynb\n\n\n\nstatic-frame\n=============\n\nImmutable and statically-typeable DataFrames with runtime type and data validation.\n\nAmong the many Python DataFrame libraries, StaticFrame is an alternative that prioritizes correctness, maintainability, and reducing opportunities for error. Key features include:\n\n* 🛡️ Immutable Data: Provides memory efficiency, excellent performance, and prohibits side effects.\n* 🗜️ Static Typing: Use Python type-hints to statically type index, columns, and columnar types.\n* 🚦 Runtime Validation: Use type hints and specialized validators for runtime type and data checks.\n* 🧭 Consistent Interface: An easy-to-learn, hierarchical, and intuitive API that avoids the many inconsistencies of Pandas.\n* 🧬 Comprehensive ``dtype`` Support: Full compatibility with all NumPy dtypes and datetime64 units.\n* 🔗 Broad Interoperability: Translate between Pandas, Arrow, Parquet, CSV, TSV, JSON, Excel XLSX, SQLite, and NumPy; output to xarray, VisiData, HTML, RST, Markdown, LaTeX, and Jupyter notebooks.\n* 🚀 Optimized Serialization \u0026 Memory Mapping: Fast disk I/O with custom NPZ and NPY encodings.\n* 💼 Multi-Table Containers: The ``Bus`` and ``Yarn`` provide interfaces to collections of tables with lazy data loading, well-suited for large datasets.\n* ⏳ Deferred Processing: The ``Batch`` provides a common interface for deferred processing of groups, windows, or any iterator.\n* 🪶 Lean Dependencies: Core functionality relies only on NumPy and team-maintained C-extensions.\n* 📚 Comprehensive Documentation: All API endpoints documented with thousands of easily runnable examples.\n\n\nCode: https://github.com/static-frame/static-frame\n\nDocs: http://static-frame.readthedocs.io\n\nPackages: https://pypi.org/project/static-frame\n\nAPI Search: https://staticframe.dev\n\nJupyter Notebook Tutorial: `Launch Binder \u003chttps://mybinder.org/v2/gh/static-frame/static-frame-ftgu/default?urlpath=tree/index.ipynb\u003e`_\n\n\n\nInstallation via ``pip``\n-------------------------------\n\nInstall StaticFrame with ``pip``. Note that pre-built wheels are published for all supported Python versions and platforms (including Apple Silicon platforms)::\n\n    pip install static-frame\n\nTo install optional dependencies for full support of input and output formats (such as XLSX and HDF5) via ``pip``::\n\n    pip install static-frame [extras]\n\n\n\nInstallation via ``conda``\n-------------------------------\n\nStaticFrame can be installed via ``conda`` with the ``conda-forge`` channel. Note that pre-built wheels of StaticFrame and all compiled dependencies are available through ``pip`` and may offer more compatibility than a ``conda``-based installation ::\n\n    conda install -c conda-forge static-frame\n\n\nInstallation via Pyodide\n-------------------------------\n\nStaticFrame can be run in the browser via Pyodide with the ``static_frame_pyodide`` package: https://github.com/static-frame/static-frame-pyodide\n\n\nDependencies\n--------------\n\nCore StaticFrame requires the following:\n\n- Python\u003e=3.9\n- numpy\u003e=1.23.5 (numpy\u003e=2 is supported)\n- arraymap==0.4.0\n- arraykit==0.10.0\n- typing-extensions\u003e=4.12.0\n\nFor extended input and output, the following packages are required:\n\n- pandas\u003e=1.1.5\n- xlsxwriter\u003e=1.1.2\n- openpyxl\u003e=3.0.9\n- xarray\u003e=0.13.0\n- pyarrow\u003e=3.0.0\n- visidata\u003e=2.4\n\n\nQuick-Start Guide\n---------------------\n\nTo get startred quickly, let's download the classic iris (flower) characteristics data set and build a simple naive Bayes classifier that can predict species from iris petal characteristics.\n\nWhile StaticFrame's API has over 7,500 endpoints, much will be familiar to users of Pandas or other DataFrame libraries. Rather than offering fewer interfaces with greater configurability, StaticFrame favors more numerous interfaces with more narrow parameters and functionality. This design leads to more maintainable code. (Read more about differences between Pandas and StaticFrame `here \u003chttps://static-frame.readthedocs.io/en/latest/articles/upgrade.html\u003e`__.)\n\n\nWe can download the data set from the UCI Machine Learning Repository and create a ``Frame``. StaticFrame exposes all constructors on the class: here, we will use the ``Frame.from_csv()`` constructor. To download a file from the internet and provide it to a constructor, we can use StaticFrame's ``WWW.from_file()`` interface::\n\n    \u003e\u003e\u003e import static_frame as sf\n    \u003e\u003e\u003e data = sf.Frame.from_csv(sf.WWW.from_file('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'), columns_depth=0)\n\n\nEach record (or row) in this dataset describes observations of an iris flower, including its sepal and petal characteristics, as well as its species (of which there are three). To display just the first few rows, we can use the ``head()`` method. Notice that StaticFrame's default display makes it very clear what type of ``Frame``, ``Index``, and NumPy datatypes are present::\n\n    \u003e\u003e\u003e data.head()\n    \u003cFrame\u003e\n    \u003cIndex\u003e 0         1         2         3         4           \u003cint64\u003e\n    \u003cIndex\u003e\n    0       5.1       3.5       1.4       0.2       Iris-setosa\n    1       4.9       3.0       1.4       0.2       Iris-setosa\n    2       4.7       3.2       1.3       0.2       Iris-setosa\n    3       4.6       3.1       1.5       0.2       Iris-setosa\n    4       5.0       3.6       1.4       0.2       Iris-setosa\n    \u003cint64\u003e \u003cfloat64\u003e \u003cfloat64\u003e \u003cfloat64\u003e \u003cfloat64\u003e \u003c\u003cU15\u003e\n\n\nAs the columns are unlabelled, let's next add column labels. StaticFrame supports reindexing (conforming existing axis labels to new labels, potentially changing the size and ordering) and relabeling (simply applying new labels without regard to existing labels). As we can ignore the default column labels (auto-incremented integers), the ``relabel()`` method is used to provide new labels.\n\nNote that while ``relabel()`` creates a new ``Frame``, underlying NumPy data is not copied. As all NumPy data is immutable in StaticFrame, we can reuse it in our new container, making such operations very efficient::\n\n    \u003e\u003e\u003e data = data.relabel(columns=('sepal_l', 'sepal_w', 'petal_l', 'petal_w', 'species'))\n    \u003e\u003e\u003e data.head()\n    \u003cFrame\u003e\n    \u003cIndex\u003e sepal_l   sepal_w   petal_l   petal_w   species     \u003c\u003cU7\u003e\n    \u003cIndex\u003e\n    0       5.1       3.5       1.4       0.2       Iris-setosa\n    1       4.9       3.0       1.4       0.2       Iris-setosa\n    2       4.7       3.2       1.3       0.2       Iris-setosa\n    3       4.6       3.1       1.5       0.2       Iris-setosa\n    4       5.0       3.6       1.4       0.2       Iris-setosa\n    \u003cint64\u003e \u003cfloat64\u003e \u003cfloat64\u003e \u003cfloat64\u003e \u003cfloat64\u003e \u003c\u003cU15\u003e\n\n\n(Read more about no-copy operations `here \u003chttps://static-frame.readthedocs.io/en/latest/articles/no_copy.html\u003e`__.)\n\nFor this example, eighty percent of the data will be used to train the classifier; the remaining twenty percent will be used to test the classifier. As all records are labelled with the known species, we can conclude by measuring the effectiveness of the classifier on the test data.\n\nTo divide the data into two groups, we create a ``Series`` of contiguous integers and then extract a random selection of 80% of the values into a new ``Series``, here named ``sel_train``. This will be used to select our traning data. As the ``sample()`` method, given a count, randomly samples that many values, your results will be different unless use the same ``seed`` argument::\n\n    \u003e\u003e\u003e sel = sf.Series(np.arange(len(data)))\n    \u003e\u003e\u003e sel_train = sel.sample(round(len(data) * .8), seed=42)\n    \u003e\u003e\u003e sel_train.head()\n    \u003cSeries\u003e\n    \u003cIndex\u003e\n    0        0\n    2        2\n    3        3\n    4        4\n    5        5\n    \u003cint64\u003e  \u003cint64\u003e\n\n\nWe will create another ``Series`` to select the test data. The ``drop[]`` interface can be used to create a new ``Series`` that excludes the training selections, leaving just the testing selections. As with many interfaces in StaticFrame (such as ``astype`` and ``assign``), brackets can be used to do ``loc[]`` style selections::\n\n    \u003e\u003e\u003e sel_test = sel.drop[sel_train]\n    \u003e\u003e\u003e sel_test.head()\n    \u003cSeries\u003e\n    \u003cIndex\u003e\n    1        1\n    14       14\n    20       20\n    21       21\n    37       37\n    \u003cint64\u003e  \u003cint64\u003e\n\n\nTo select a subset of the data for training, the ``sel_train`` ``Series`` can be passed to ``loc[]`` to select just those rows::\n\n    \u003e\u003e\u003e data_train = data.loc[sel_train]\n    \u003e\u003e\u003e data_train.head()\n    \u003cFrame\u003e\n    \u003cIndex\u003e sepal_l   sepal_w   petal_l   petal_w   species     \u003c\u003cU7\u003e\n    \u003cIndex\u003e\n    0       5.1       3.5       1.4       0.2       Iris-setosa\n    2       4.7       3.2       1.3       0.2       Iris-setosa\n    3       4.6       3.1       1.5       0.2       Iris-setosa\n    4       5.0       3.6       1.4       0.2       Iris-setosa\n    5       5.4       3.9       1.7       0.4       Iris-setosa\n    \u003cint64\u003e \u003cfloat64\u003e \u003cfloat64\u003e \u003cfloat64\u003e \u003cfloat64\u003e \u003c\u003cU15\u003e\n\n\nWith our data divided into two randomly-selected, non-overlapping groups, we can proceed to implement the naive Bayes classifier. We will compute the ``posterior`` of the test data by multiplying the ``prior`` and the ``likelihood``. With the ``posterior``, we can determine which species the classifier has calculated is most likely. (More on naive Bayes classifiers can be found `here \u003chttps://en.wikipedia.org/wiki/Naive_Bayes_classifier\u003e`__.)\n\nThe ``prior`` is calculated as the percentage of samples of each species in the training data. This is the \"normalized\" count per species. To get a ``Series`` of counts per species, we can select the species column, iterate over groups based on species name, and count the size of each group.\n\nIn StaticFrame, this can be done by calling ``Series.iter_group_items()`` to get an iterator of pairs of group label, group (where the group is a ``Series``). This iterator (or any similar iterator) can be given to a ``Batch``, a chaining processor of ``Frame`` or ``Series``, to perform operations on each group. (For more on the ``Batch`` and other higher-order containers in StaticFrame, see `here \u003chttps://static-frame.readthedocs.io/en/latest/articles/uhoc.html\u003e`__.)\n\nOnce the ``Batch`` is created, selections, method calls, and operator expressions can be chained as if they were being called on a single container. Processing happens to every contained container, and a container is returned, only when a finalizer method, such as ``to_series()``, is called::\n\n    \u003e\u003e\u003e counts = sf.Batch(data_train['species'].iter_group_items()).count().to_series()\n    \u003e\u003e\u003e counts\n    \u003cSeries\u003e\n    \u003cIndex\u003e\n    Iris-setosa     43\n    Iris-versicolor 39\n    Iris-virginica  38\n    \u003c\u003cU15\u003e          \u003cint64\u003e\n\n\nAs with NumPy, StaticFrame containers can be used in expressions with binary operators. The ``prior`` can be derived by dividing ``counts`` by the size of the training data. This returns a ``Series`` of the percentage of records per species::\n\n    \u003e\u003e\u003e prior = counts / len(data_train)\n    \u003e\u003e\u003e prior\n    \u003cSeries\u003e\n    \u003cIndex\u003e\n    Iris-setosa     0.35833333333333334\n    Iris-versicolor 0.325\n    Iris-virginica  0.31666666666666665\n    \u003c\u003cU15\u003e          \u003cfloat64\u003e\n\n\nHaving calculated the ``prior``, we can calculate ``likelihood`` next. To calculate ``likelihood``, we will call a probability distribution function (imported from SciPy) with the test data, once for each species, given the characteristics (mean and standard deviation) observed in the test data for that species.\n\nThe ``Batch`` can again be used to calculate the mean and standard deviation, per species, from the training data. With the ``Frame`` of training data, we call ``iter_group_items()`` to group by species and, passing that iterator to ``Batch``, call ``mean()`` (assigned to ``mu``) or ``std()`` (assigned to ``sigma``). Note that ``iter_group_items()`` has an optional ``drop`` parameter to remove the column used for grouping from subsequent operations::\n\n\n    \u003e\u003e\u003e mu = sf.Batch(data_train[['sepal_l', 'sepal_w', 'species']].iter_group_items('species', drop=True)).mean().to_frame()\n    \u003e\u003e\u003e mu\n    \u003cFrame\u003e\n    \u003cIndex\u003e         sepal_l            sepal_w            \u003c\u003cU7\u003e\n    \u003cIndex\u003e\n    Iris-setosa     4.986046511627907  3.434883720930233\n    Iris-versicolor 5.920512820512819  2.771794871794872\n    Iris-virginica  6.6078947368421055 2.9763157894736842\n    \u003c\u003cU15\u003e          \u003cfloat64\u003e          \u003cfloat64\u003e\n\n    \u003e\u003e\u003e sigma = sf.Batch(data_train[['sepal_l', 'sepal_w', 'species']].iter_group_items('species', drop=True)).std(ddof=1).to_frame()\n    \u003e\u003e\u003e sigma\n    \u003cFrame\u003e\n    \u003cIndex\u003e         sepal_l            sepal_w             \u003c\u003cU7\u003e\n    \u003cIndex\u003e\n    Iris-setosa     0.3419700595003668 0.3477024733400345\n    Iris-versicolor 0.508444214804487  0.33082728674826684\n    Iris-virginica  0.6055516042229233 0.3513942965328924\n    \u003c\u003cU15\u003e          \u003cfloat64\u003e          \u003cfloat64\u003e\n\n\nFor a unified display of these characteristics, we can build a hierarchical index on each ``Frame`` with ``relabel_level_add()`` (adding the \"mu\" or \"sigma\" labels), then vertically concatenate the tables. As StaticFrame always requires unique labels in indices, adding an additional label is required before concatenation. The built-in ``round`` function can be used for more tidy display::\n\n    \u003e\u003e\u003e stats = sf.Frame.from_concat((mu.relabel_level_add('mu'), sigma.relabel_level_add('sigma')))\n    \u003e\u003e\u003e round(stats, 2)\n    \u003cFrame\u003e\n    \u003cIndex\u003e                          sepal_l   sepal_w   \u003c\u003cU7\u003e\n    \u003cIndexHierarchy\u003e\n    mu               Iris-setosa     4.99      3.43\n    mu               Iris-versicolor 5.92      2.77\n    mu               Iris-virginica  6.61      2.98\n    sigma            Iris-setosa     0.34      0.35\n    sigma            Iris-versicolor 0.51      0.33\n    sigma            Iris-virginica  0.61      0.35\n    \u003c\u003cU5\u003e            \u003c\u003cU15\u003e          \u003cfloat64\u003e \u003cfloat64\u003e\n\n\nWe can now move on to processing the test data with the characteristics derived from the training data. To do that, we will extract our previously selected test records with ``sel_test`` into a new ``Frame``, to which we can add our ``posterior`` predictions and final species classifications.\n\nIt is common to process data in table by adding columns from left to right. StaticFrame permits this limited form of mutability with the grow-only ``FrameGO``. While underlying NumPy arrays are still always immutable, columns can be added to a ``FrameGO`` with bracket-style assignments. A ``FrameGO`` can be created from a ``Frame`` with the ``to_frame_go()`` method. As mentioned elsewhere, underlying immutable NumPy arrays are not copied: this is an efficient, no-copy operation.\n\nPassing two arguments to ``loc[]``, we can select rows with the values from ``sel_test``, and we can select columns with a list of labels for the sepal length and sepal width::\n\n    \u003e\u003e\u003e data_test = data.loc[sel_test.values, ['sepal_l', 'sepal_w']].to_frame_go()\n    \u003e\u003e\u003e data_test.head()\n    \u003cFrameGO\u003e\n    \u003cIndexGO\u003e sepal_l   sepal_w   \u003c\u003cU7\u003e\n    \u003cIndex\u003e\n    1         4.9       3.0\n    14        5.8       4.0\n    20        5.4       3.4\n    21        5.1       3.7\n    37        4.9       3.1\n    \u003cint64\u003e   \u003cfloat64\u003e \u003cfloat64\u003e\n\n\nStaticFrame interfaces make extensive use of iterators and generators. As used below, the ``Frame.from_fields()`` constructor will create a ``Frame`` from any iterable (or generator) of column arrays.\n\nThe ``likelihood_of_species()`` function (defined below), for each index label in ``mu`` (which provides each unique iris species), calculates a probability density function for the test data, given the ``mu`` (mean) and ``sigma`` (standard deviation) for the species. An array of the sum of the log is yielded::\n\n    \u003e\u003e\u003e from scipy.stats import norm\n    \u003e\u003e\u003e def likelihood_of_species():\n    ...     for label in mu.index:\n    ...             pdf = norm.pdf(data_test.values, mu.loc[label], sigma.loc[label])\n    ...             yield np.log(pdf).sum(axis=1)\n\n\nWhile the generator function above is easy to read, it is hard to copy and paste. If you are following along, using the one-line generator expression, below, will be easier. The two are equivalent:\n\n\u003e\u003e\u003e likelihood_of_species = (np.log(norm.pdf(data_test.values, mu.loc[label], sigma.loc[label])).sum(axis=1) for label in mu.index)\n\n\nWith this generator expression defined, we call the ``from_fields`` constructor to produce the ``likelihood`` table, providing column labels from ``mu.index`` and index labels from ``data_test.index``. For each test record row we now have a likelihood per species::\n\n    \u003e\u003e\u003e likelihood = sf.Frame.from_fields(likelihood_of_species, columns=mu.index, index=data_test.index)\n    \u003e\u003e\u003e round(likelihood.head(), 2)\n    \u003cFrame\u003e\n    \u003cIndex\u003e Iris-setosa Iris-versicolor Iris-virginica \u003c\u003cU15\u003e\n    \u003cIndex\u003e\n    1       -0.52       -2.31           -4.27\n    14      -3.86       -6.97           -5.42\n    20      -0.45       -2.38           -3.01\n    21      -0.05       -5.29           -5.51\n    37      -0.2        -2.56           -4.33\n    \u003cint64\u003e \u003cfloat64\u003e   \u003cfloat64\u003e       \u003cfloat64\u003e\n\n\nWe can calculate the ``posterior`` by multiplying ``likelihood`` by ``prior``. Whenever performing binary operations on ``Frame`` and ``Series``, indices will be aligned and, if necessary, reindexed before processing::\n\n    \u003e\u003e\u003e posterior = likelihood * prior\n    \u003e\u003e\u003e round(posterior.head(), 2)\n    \u003cFrame\u003e\n    \u003cIndex\u003e Iris-setosa Iris-versicolor Iris-virginica \u003c\u003cU15\u003e\n    \u003cIndex\u003e\n    1       -0.19       -0.75           -1.35\n    14      -1.38       -2.27           -1.72\n    20      -0.16       -0.77           -0.95\n    21      -0.02       -1.72           -1.75\n    37      -0.07       -0.83           -1.37\n    \u003cint64\u003e \u003cfloat64\u003e   \u003cfloat64\u003e       \u003cfloat64\u003e\n\n\nWe can now add columns to our ``data_test`` ``FrameGO``. To determine our best prediction of species for each row of the test data, the column label (the species) of the maximum a posteriori estimate is selected with ``loc_max()``::\n\n    \u003e\u003e\u003e data_test['predict'] = posterior.loc_max(axis=1)\n    \u003e\u003e\u003e data_test.head()\n    \u003cFrameGO\u003e\n    \u003cIndexGO\u003e sepal_l   sepal_w   predict     \u003c\u003cU7\u003e\n    \u003cIndex\u003e\n    1         4.9       3.0       Iris-setosa\n    14        5.8       4.0       Iris-setosa\n    20        5.4       3.4       Iris-setosa\n    21        5.1       3.7       Iris-setosa\n    37        4.9       3.1       Iris-setosa\n    \u003cint64\u003e   \u003cfloat64\u003e \u003cfloat64\u003e \u003c\u003cU15\u003e\n\n\nWe can add two additional columns to evaluate the effectivess of the classifier. First, we can add an \"observed\" column by adding the original \"species\" column from the original ``data`` ``Frame``. In assigning a ``Series`` to a ``Frame``, only values found in the intersection of the indices will be added as a column::\n\n    \u003e\u003e\u003e data_test['observed'] = data['species']\n    \u003e\u003e\u003e data_test.head()\n    \u003cFrameGO\u003e\n    \u003cIndexGO\u003e sepal_l   sepal_w   predict     observed    \u003c\u003cU8\u003e\n    \u003cIndex\u003e\n    1         4.9       3.0       Iris-setosa Iris-setosa\n    14        5.8       4.0       Iris-setosa Iris-setosa\n    20        5.4       3.4       Iris-setosa Iris-setosa\n    21        5.1       3.7       Iris-setosa Iris-setosa\n    37        4.9       3.1       Iris-setosa Iris-setosa\n    \u003cint64\u003e   \u003cfloat64\u003e \u003cfloat64\u003e \u003c\u003cU15\u003e      \u003c\u003cU15\u003e\n\n\nHaving populated a column of predicted and observed values, we can compare the two to get a Boolean column indicating when the classifier calculated a correct predicton::\n\n    \u003e\u003e\u003e data_test['correct'] = data_test['predict'] == data_test['observed']\n    \u003e\u003e\u003e data_test.tail()\n    \u003cFrameGO\u003e\n    \u003cIndexGO\u003e sepal_l   sepal_w   predict         observed       correct \u003c\u003cU8\u003e\n    \u003cIndex\u003e\n    129       7.2       3.0       Iris-virginica  Iris-virginica True\n    130       7.4       2.8       Iris-virginica  Iris-virginica True\n    140       6.7       3.1       Iris-virginica  Iris-virginica True\n    144       6.7       3.3       Iris-virginica  Iris-virginica True\n    149       5.9       3.0       Iris-versicolor Iris-virginica False\n    \u003cint64\u003e   \u003cfloat64\u003e \u003cfloat64\u003e \u003c\u003cU15\u003e          \u003c\u003cU15\u003e         \u003cbool\u003e\n\n\nTo find the percentage of correct classifications among the test data, we can sum the ``correct`` Boolean column and divide that by the size of the test data::\n\n    \u003e\u003e\u003e data_test[\"correct\"].sum() / len(data_test)\n    0.7333333333333333\n\nThis simple naive Bayes classifier can predict iris species correctly about 73% of the time.\n\nFor further introduction to StaticFrame, including links to articles, videos, and documentation, see `here \u003chttps://static-frame.readthedocs.io/en/latest/intro.html\u003e`__.\n\n\n\n","funding_links":[],"categories":["Python","Data Containers \u0026 Dataframes"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstatic-frame%2Fstatic-frame","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstatic-frame%2Fstatic-frame","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstatic-frame%2Fstatic-frame/lists"}