Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/shawnbrown/datatest

Tools for test driven data-wrangling and data validation.
https://github.com/shawnbrown/datatest

data-wrangling pytest-plugin python quality-assurance testing unittest

Last synced: 3 months ago
JSON representation

Tools for test driven data-wrangling and data validation.

Host: GitHub
URL: https://github.com/shawnbrown/datatest
Owner: shawnbrown
License: other
Created: 2016-05-12T13:16:27.000Z (over 8 years ago)
Default Branch: master
Last Pushed: 2021-12-05T17:44:33.000Z (almost 3 years ago)
Last Synced: 2024-07-08T02:31:37.262Z (4 months ago)
Topics: data-wrangling, pytest-plugin, python, quality-assurance, testing, unittest
Language: Python
Homepage:
Size: 3.27 MB
Stars: 291
Watchers: 12
Forks: 14
Open Issues: 14
Metadata Files:
- Readme: README.rst
- Changelog: CHANGELOG
- License: LICENSE

Awesome Lists containing this project

awesome-python-machine-learning-resources - GitHub - 21% open · ⏱️ 05.12.2021): (数据读写与提取)
jimsghstars - shawnbrown/datatest - Tools for test driven data-wrangling and data validation. (Python)

README

        
********************************************************

datatest: Test driven data-wrangling and data validation

********************************************************

|licensebadge| |pythonbadge| |requiresbadge|

|repobadge| |buildbadge| |statusbadge| |stabledocsbadge| |latestdocsbadge|

Datatest helps to speed up and formalize data-wrangling and data

validation tasks. It implements a system of validation methods,

difference classes, and acceptance managers. Datatest can help you:

* Clean and wrangle data faster and more accurately.

* Maintain a record of checks and decisions regarding important data sets.

* Distinguish between ideal criteria and acceptible deviation.

* Validate the input and output of data pipeline components.

* Measure progress of data preparation tasks.

* On-board new team members with an explicit and structured process.

Datatest can be used directly in your own projects or as part of a testing

framework like pytest_ or unittest_. It has no hard dependencies; it's

tested on Python 2.6, 2.7, 3.2 through 3.10, PyPy, and PyPy3; and is freely

available under the Apache License, version 2.

.. _pytest: https://pytest.org

.. _unittest: https://docs.python.org/library/unittest.html

:Documentation:

    | https://datatest.readthedocs.io/ (stable)

    | https://datatest.readthedocs.io/en/latest/ (latest)

:Official:

    | https://pypi.org/project/datatest/

Code Examples

=============

Validating a Dictionary of Lists

--------------------------------

.. code-block:: python

    from datatest import validate, accepted, Invalid

    data = {

        'A': [1, 2, 3, 4],

        'B': ['x', 'y', 'x', 'x'],

        'C': ['foo', 'bar', 'baz', 'EMPTY']

    }

    validate(data.keys(), {'A', 'B', 'C'})

    validate(data['A'], int)

    validate(data['B'], {'x', 'y'})

    with accepted(Invalid('EMPTY')):

        validate(data['C'], str.islower)

Validating a Pandas DataFrame

-----------------------------

.. code-block:: python

    import pandas as pd

    from datatest import register_accessors, accepted, Invalid

    register_accessors()

    df = pd.read_csv('data.csv')

    df.columns.validate({'A', 'B', 'C'})

    df['A'].validate(int)

    df['B'].validate({'x', 'y'})

    with accepted(Invalid('EMPTY')):

        df['C'].validate(str.islower)

Installation

============

.. start-inclusion-marker-install

The easiest way to install datatest is to use `pip `_:

.. code-block:: console

    pip install datatest

If you are upgrading from version 0.11.0 or newer, use the ``--upgrade``

option:

.. code-block:: console

    pip install --upgrade datatest

Upgrading From Version 0.9.6

----------------------------

If you have an existing codebase of older datatest scripts, you should

upgrade using the following steps:

* Install datatest 0.10.0 first:

  .. code-block:: console

      pip install --force-reinstall datatest==0.10.0

* Run your existing code and check for DeprecationWarnings.

* Update the parts of your code that use deprecated features.

* Once your code is running without DeprecationWarnings,

  install the latest version of datatest:

  .. code-block:: console

      pip install --upgrade datatest

Stuntman Mike

-------------

If you need bug-fixes or features that are not available

in the current stable release, you can "pip install" the

development version directly from GitHub:

.. code-block:: console

    pip install --upgrade https://github.com/shawnbrown/datatest/archive/master.zip

All of the usual caveats for a development install should

apply---only use this version if you can risk some instability

or if you know exactly what you're doing. While care is taken

to never break the build, it can happen.

Safety-first Clyde

------------------

If you need to review and test packages before installing, you can

install datatest manually.

Download the latest **source** distribution from the Python Package

Index (PyPI):

    https://pypi.org/project/datatest/#files

Unpack the file (replacing X.Y.Z with the appropriate version number)

and review the source code:

.. code-block:: console

    tar xvfz datatest-X.Y.Z.tar.gz

Change to the unpacked directory and run the tests:

.. code-block:: console

    cd datatest-X.Y.Z

    python setup.py test

Don't worry if some of the tests are skipped. Tests for optional data

sources (like pandas DataFrames or NumPy arrays) are skipped when the

related third-party packages are not installed.

If the source code and test results are satisfactory, install the

package:

.. code-block:: console

    python setup.py install

.. end-inclusion-marker-install

Supported Versions

==================

Tested on Python 2.6, 2.7, 3.2 through 3.10, PyPy, and PyPy3.

Datatest is pure Python and may also run on other implementations

as well (check using "setup.py test" before installing).

Backward Compatibility

======================

If you have existing tests that use API features which have

changed since 0.9.0, you can still run your old code by

adding the following import to the beginning of each file:

.. code-block:: python

    from datatest.__past__ import api09

To maintain existing test code, this project makes a best-effort

attempt to provide backward compatibility support for older

features. The API will be improved in the future but only in

measured and sustainable ways.

All of the data used at the `National Committee for an Effective

Congress `_ has been checked with

datatest for several years so there is, already, a large and

growing codebase that relies on current features and must be

maintained into the future.

Soft Dependencies

=================

Datatest has no hard, third-party dependencies. But if you want

to interface with pandas DataFrames, NumPy arrays, or other

optional data sources, you will need to install the relevant

packages (``pandas``, ``numpy``, etc.).

Development Repository

======================

The development repository for ``datatest`` is hosted on

`GitHub `_.

----------

Freely licensed under the Apache License, Version 2.0

Copyright 2014 - 2021 National Committee for an Effective Congress, et al.

.. start-inclusion-marker-badge-substitutions

.. |buildbadge| image:: https://img.shields.io/travis/shawnbrown/datatest?logo=travis-ci&logoColor=white&style=flat-square

    :target: https://travis-ci.org/shawnbrown/datatest

    :alt: Current Build Status

.. |pypibadge| image:: https://img.shields.io/pypi/v/datatest?logo=pypi&logoColor=white&style=flat-square

    :target: https://pypi.org/project/datatest/

    :alt: Current PyPI Version

.. |commitsbadge| image:: https://img.shields.io/github/commits-since/shawnbrown/datatest/latest?color=informational&logo=github&logoColor=white&style=flat-square

    :target: https://github.com/shawnbrown/datatest/

    :alt: Commits Since Last Release

.. |statusbadge| image:: https://img.shields.io/pypi/status/datatest?label=PyPI%20status&logo=pypi&logoColor=white&style=flat-square

    :target: https://pypi.org/project/datatest/

    :alt: Development Status

.. |licensebadge| image:: https://img.shields.io/badge/license-Apache_2-informational?logo=open-source-initiative&logoColor=white&style=flat-square

    :target: https://opensource.org/licenses/Apache-2.0

    :alt: Apache 2.0 License

.. |pythonbadge| image:: https://img.shields.io/badge/python-2.6_|_2.7_|_3.2_through_3.10_|_PyPy_|_PyPy3-informational?logo=python&logoColor=white&style=flat-square

    :target: https://pypi.org/project/datatest/#supported-versions

    :alt: Supported Python Versions

.. |requiresbadge| image:: https://img.shields.io/badge/install_requires-no_dependencies-informational?logo=pypi&logoColor=white&style=flat-square

    :target: https://pypi.org/project/datatest/#installation

    :alt: Installation Requirements

.. |repobadge| image:: https://img.shields.io/badge/repo-GitHub-informational?logo=github&logoColor=white&style=flat-square

    :target: https://github.com/shawnbrown/datatest/

    :alt: Development Repository

.. |stabledocsbadge| image:: https://img.shields.io/badge/docs_(stable)-Read_the_Docs-informational?logo=read-the-docs&logoColor=white&style=flat-square

    :target: https://datatest.readthedocs.io/en/stable/

    :alt: Documentation (stable)

.. |latestdocsbadge| image:: https://img.shields.io/badge/docs_(latest)-Read_the_Docs-informational?logo=read-the-docs&logoColor=white&style=flat-square

    :target: https://datatest.readthedocs.io/en/latest/

    :alt: Documentation (latest)

.. |starsbadge| image:: https://img.shields.io/github/stars/shawnbrown/datatest.svg?logo=github&logoColor=white&style=flat-square

    :target: https://github.com/shawnbrown/datatest/stargazers

    :alt: GitHub users who have starred this project

.. end-inclusion-marker-badge-substitutions