{"id":15688075,"url":"https://github.com/rodrigobaron/qafs","last_synced_at":"2025-08-17T13:07:14.898Z","repository":{"id":43667748,"uuid":"399997046","full_name":"rodrigobaron/qafs","owner":"rodrigobaron","description":"Quality Aware Feature Store","archived":false,"fork":false,"pushed_at":"2024-04-23T18:59:38.000Z","size":104,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-08T05:07:23.296Z","etag":null,"topics":["dataquality","feature-engineering","feature-store"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/qafs/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rodrigobaron.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-26T00:51:03.000Z","updated_at":"2023-02-27T04:47:19.000Z","dependencies_parsed_at":"2024-10-22T12:54:20.749Z","dependency_job_id":null,"html_url":"https://github.com/rodrigobaron/qafs","commit_stats":{"total_commits":20,"total_committers":2,"mean_commits":10.0,"dds":"0.050000000000000044","last_synced_commit":"6deda1c997d12771c502ef772814f447090aae80"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rodrigobaron%2Fqafs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rodrigobaron%2Fqafs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rodrigobaron%2Fqafs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rodrigobaron%2Fqafs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rodrigobaron","download_url":"https://codeload.github.com/rodrigobaron/qafs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243025464,"owners_count":20223805,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataquality","feature-engineering","feature-store"],"created_at":"2024-10-03T17:54:23.910Z","updated_at":"2025-03-11T11:32:14.325Z","avatar_url":"https://github.com/rodrigobaron.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Quality Aware Feature Store\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/rodrigobaron/qafs/actions/workflows/build.yaml\"\u003e\n        \u003cimg alt=\"Build\" src=\"https://github.com/rodrigobaron/qafs/actions/workflows/build.yaml/badge.svg\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/rodrigobaron/qafs/blob/main/LICENSE\"\u003e\n        \u003cimg alt=\"GitHub\" src=\"https://img.shields.io/github/license/rodrigobaron/qafs.svg?color=blue\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/rodrigobaron/qafs/releases\"\u003e\n        \u003cimg alt=\"GitHub release\" src=\"https://img.shields.io/github/release/rodrigobaron/qafs.svg\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\u003ch3 align=\"center\"\u003e\n    Simple and scalable feature store with data quality checks.\n\u003c/h3\u003e\n\n## WARN: API WILL BE REWRITE FOR SIMPLICITY\n\nfeature store aim to solve the data management problems when building Machine Learning applications. However the data quality is a component which data teams need integrate and handle as separated component. This project join both concepts keeping the data quality closely coupled with data transformations making necessary a minimal data verification check and possibiliting the data/transformations check evolve during the projects.\n\nFor that **qafs** have a strong dependecy with [pandera](https://pandera.readthedocs.io/) to build the data validations.\n\n\n## Features\n\n* Pandas-like API\n* Features information stored in database along with metadata.\n* Dask to process large datasets in a cluster enviroment.\n* Data is stored as timeseries in [Parquet format](https://parquet.apache.org/), store in filesystem or object storage services.\n* Store transformations as feature.\n\n\n## Get Started\n\nInstalling the python package through pip:  \n\n```bash\n$ pip install qafs\n```\n\nBellow is an example of usage **qafs** where we'll create a feature store and register `numbers` feature and an `squared` feature transformation. First we need import the packages and create the feature store, for this example we are using sqlite database and persisting the features in the filesystem:  \n\n```python\nimport qafs\nimport pandas as pd\nimport pandera as pa\nfrom pandera import Check, Column, DataFrameSchema\nfrom pandera import io\n\n\nfs = qafs.FeatureStore(\n    connection_string='sqlite:///test.sqlite',\n    url='/tmp/featurestore/example'\n)\n```\n\nFeatures could be stored in namespaces, it help organize the data. When creating `numbers` we specify the `'example/numbers'` feature to point the feature `numbers`at that namespace `example` however we can use the arguments `name='numbers', namespace='example'` as well. The we specify the data validation using **pandera** telling that feature is `Integer` and the values should be `greater than 0`:\n\n```python\nfs.create_namespace('example', description='Example datasets')\nfs.create_feature(\n    'example/numbers',\n    description='Timeseries of numbers',\n    check=Column(pa.Int, Check.greater_than(0))\n)\n\n\ndts = pd.date_range('2020-01-01', '2021-02-09')\ndf = pd.DataFrame({'time': dts, 'numbers': list(range(1, len(dts) + 1))})\n\nfs.save_dataframe(df, name='numbers', namespace='example')\n\n```\n\nTo register our `squared` transformation feature we're using the annotation `fs.transform` and fetching the data from the `numbers` feature applying the same data validation from `numbers`:\n```python\n@fs.transform(\n    'example/squared',\n    from_features=['example/numbers'],\n    check=Column(pa.Int, Check.greater_than(0))\n)\ndef squared(df):\n    return df ** 2\n\n```\n\nWhen fetch our features we should see:\n```python\ndf_query = fs.load_dataframe(\n    ['example/numbers', 'example/squared'], \n    from_date='2021-01-01',\n    to_date='2021-01-31'\n)\nprint(df_query.tail(1))\n##----\n#             example/numbers  example/squared\n# time                                        \n# 2021-01-31              397           157609\n##----\n```\n\n## Contributing\n\nPlease follow the [Contributing](CONTRIBUTING.md) guide.\n\n## License\n\n[GPL-3.0 License](LICENSE)  \n\nThis project started using the as base [bytehub feature store](https://github.com/bytehub-ai/bytehub) and is under the same license.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frodrigobaron%2Fqafs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frodrigobaron%2Fqafs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frodrigobaron%2Fqafs/lists"}