{"id":19643784,"url":"https://github.com/logicalclocks/feature-store-api","last_synced_at":"2025-04-07T06:10:23.668Z","repository":{"id":37085886,"uuid":"232286451","full_name":"logicalclocks/feature-store-api","owner":"logicalclocks","description":"Python - Java/Scala API for the Hopsworks feature store","archived":false,"fork":false,"pushed_at":"2024-04-12T12:38:57.000Z","size":130734,"stargazers_count":51,"open_issues_count":49,"forks_count":43,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-04-12T16:36:29.250Z","etag":null,"topics":["feature-store","hopsworks","hsfs","python","scala","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/logicalclocks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-01-07T09:10:14.000Z","updated_at":"2024-04-15T11:41:35.268Z","dependencies_parsed_at":"2022-07-12T16:13:50.209Z","dependency_job_id":"c681c436-c031-48e9-8f62-e35c17f1fc5d","html_url":"https://github.com/logicalclocks/feature-store-api","commit_stats":{"total_commits":658,"total_committers":34,"mean_commits":"19.352941176470587","dds":0.7507598784194529,"last_synced_commit":"564d2740435adc5fae15bc0acd76a6c21e55e182"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Ffeature-store-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Ffeature-store-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Ffeature-store-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/logicalclocks%2Ffeature-store-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/logicalclocks","download_url":"https://codeload.github.com/logicalclocks/feature-store-api/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247601448,"owners_count":20964864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["feature-store","hopsworks","hsfs","python","scala","spark"],"created_at":"2024-11-11T14:23:59.633Z","updated_at":"2025-04-07T06:10:23.629Z","avatar_url":"https://github.com/logicalclocks.png","language":"Python","funding_links":[],"categories":["Feature Store"],"sub_categories":[],"readme":"# Hopsworks Feature Store\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://community.hopsworks.ai\"\u003e\u003cimg\n    src=\"https://img.shields.io/discourse/users?label=Hopsworks%20Community\u0026server=https%3A%2F%2Fcommunity.hopsworks.ai\"\n    alt=\"Hopsworks Community\"\n  /\u003e\u003c/a\u003e\n    \u003ca href=\"https://docs.hopsworks.ai\"\u003e\u003cimg\n    src=\"https://img.shields.io/badge/docs-HSFS-orange\"\n    alt=\"Hopsworks Feature Store Documentation\"\n  /\u003e\u003c/a\u003e\n  \u003ca\u003e\u003cimg\n    src=\"https://img.shields.io/badge/python-3.8+-blue\"\n    alt=\"python\"\n  /\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/hsfs/\"\u003e\u003cimg\n    src=\"https://img.shields.io/pypi/v/hsfs?color=blue\"\n    alt=\"PyPiStatus\"\n  /\u003e\u003c/a\u003e\n  \u003ca href=\"https://archiva.hops.works/#artifact/com.logicalclocks/hsfs\"\u003e\u003cimg\n    src=\"https://img.shields.io/badge/java-HSFS-green\"\n    alt=\"Scala/Java Artifacts\"\n  /\u003e\u003c/a\u003e\n  \u003ca href=\"https://pepy.tech/project/hsfs/month\"\u003e\u003cimg\n    src=\"https://pepy.tech/badge/hsfs/month\"\n    alt=\"Downloads\"\n  /\u003e\u003c/a\u003e\n  \u003ca href=https://github.com/astral-sh/ruff\u003e\u003cimg\n    src=\"https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json\"\n    alt=\"Ruff\"\n  /\u003e\u003c/a\u003e\n  \u003ca\u003e\u003cimg\n    src=\"https://img.shields.io/pypi/l/hsfs?color=green\"\n    alt=\"License\"\n  /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nHSFS is the library to interact with the Hopsworks Feature Store. The library makes creating new features, feature groups and training datasets easy.\n\nThe library is environment independent and can be used in two modes:\n\n- Spark mode: For data engineering jobs that create and write features into the feature store or generate training datasets. It requires a Spark environment such as the one provided in the Hopsworks platform or Databricks. In Spark mode, HSFS provides bindings both for Python and JVM languages.\n\n- Python mode: For data science jobs to explore the features available in the feature store, generate training datasets and feed them in a training pipeline. Python mode requires just a Python interpreter and can be used both in Hopsworks from Python Jobs/Jupyter Kernels, Amazon SageMaker or KubeFlow.\n\nThe library automatically configures itself based on the environment it is run.\nHowever, to connect from an external environment such as Databricks or AWS Sagemaker,\nadditional connection information, such as host and port, is required. For more information checkout the [Hopsworks documentation](https://docs.hopsworks.ai/latest/).\n\n## Getting Started On Hopsworks\n\nGet started easily by registering an account on [Hopsworks Serverless](https://app.hopsworks.ai/). Create your project and a [new Api key](https://docs.hopsworks.ai/latest/user_guides/projects/api_key/create_api_key/). In a new python environment with Python 3.8 or higher, install the [client library](https://docs.hopsworks.ai/latest/user_guides/client_installation/) using pip:\n\n```bash\n# Get all Hopsworks SDKs: Feature Store, Model Serving and Platform SDK\npip install hopsworks\n# or minimum install with the Feature Store SDK\npip install hsfs[python]\n# if using zsh don't forget the quotes\npip install 'hsfs[python]'\n```\n\nYou can start a notebook and instantiate a connection and get the project feature store handler.\n\n```python\nimport hopsworks\n\nproject = hopsworks.login() # you will be prompted for your api key\nfs = project.get_feature_store()\n```\n\nor using `hsfs` directly:\n\n```python\nimport hsfs\n\nconnection = hsfs.connection(\n    host=\"c.app.hopsworks.ai\", #\n    project=\"your-project\",\n    api_key_value=\"your-api-key\",\n)\nfs = connection.get_feature_store()\n```\n\nCreate a new feature group to start inserting feature values.\n```python\nfg = fs.create_feature_group(\"rain\",\n                        version=1,\n                        description=\"Rain features\",\n                        primary_key=['date', 'location_id'],\n                        online_enabled=True)\n\nfg.save(dataframe)\n```\n\nUpsert new data in to the feature group with `time_travel_format=\"HUDI\"`\".\n```python\nfg.insert(upsert_df)\n```\n\nRetrieve commit timeline metdata of the feature group with `time_travel_format=\"HUDI\"`\".\n```python\nfg.commit_details()\n```\n\n\"Reading feature group as of specific point in time\".\n```python\nfg = fs.get_feature_group(\"rain\", 1)\nfg.read(\"2020-10-20 07:34:11\").show()\n```\n\nRead updates  that occurred between specified points in time.\n```python\nfg = fs.get_feature_group(\"rain\", 1)\nfg.read_changes(\"2020-10-20 07:31:38\", \"2020-10-20 07:34:11\").show()\n```\n\nJoin features together\n```python\nfeature_join = rain_fg.select_all()\n                    .join(temperature_fg.select_all(), on=[\"date\", \"location_id\"])\n                    .join(location_fg.select_all())\nfeature_join.show(5)\n```\n\njoin feature groups that correspond to specific point in time\n```python\nfeature_join = rain_fg.select_all()\n                    .join(temperature_fg.select_all(), on=[\"date\", \"location_id\"])\n                    .join(location_fg.select_all())\n                    .as_of(\"2020-10-31\")\nfeature_join.show(5)\n```\n\njoin feature groups that correspond to different time\n```python\nrain_fg_q = rain_fg.select_all().as_of(\"2020-10-20 07:41:43\")\ntemperature_fg_q = temperature_fg.select_all().as_of(\"2020-10-20 07:32:33\")\nlocation_fg_q = location_fg.select_all().as_of(\"2020-10-20 07:33:08\")\njoined_features_q = rain_fg_q.join(temperature_fg_q).join(location_fg_q)\n```\n\nUse the query object to create a training dataset:\n```python\ntd = fs.create_training_dataset(\"rain_dataset\",\n                                version=1,\n                                data_format=\"tfrecords\",\n                                description=\"A test training dataset saved in TfRecords format\",\n                                splits={'train': 0.7, 'test': 0.2, 'validate': 0.1})\n\ntd.save(feature_join)\n```\n\nA short introduction to the Scala API:\n```scala\nimport com.logicalclocks.hsfs._\nval connection = HopsworksConnection.builder().build()\nval fs = connection.getFeatureStore();\nval attendances_features_fg = fs.getFeatureGroup(\"games_features\", 1);\nattendances_features_fg.show(1)\n```\n\nYou can find more examples on how to use the library in our [hops-examples](https://github.com/logicalclocks/hops-examples) repository.\n\n## Usage\n\nUsage data is collected for improving quality of the library. It is turned on by default if the backend\nis \"c.app.hopsworks.ai\". To turn it off, use one of the following way:\n```python\n# use environment variable\nimport os\nos.environ[\"ENABLE_HOPSWORKS_USAGE\"] = \"false\"\n\n# use `disable_usage_logging`\nimport hsfs\nhsfs.disable_usage_logging()\n```\n\nThe source code can be found in python/hsfs/usage.py.\n\n## Documentation\n\nDocumentation is available at [Hopsworks Feature Store Documentation](https://docs.hopsworks.ai/).\n\n## Issues\n\nFor general questions about the usage of Hopsworks and the Feature Store please open a topic on [Hopsworks Community](https://community.hopsworks.ai/).\n\nPlease report any issue using [Github issue tracking](https://github.com/logicalclocks/feature-store-api/issues).\n\nPlease attach the client environment from the output below in the issue:\n```python\nimport hopsworks\nimport hsfs\nhopsworks.login().get_feature_store()\nprint(hsfs.get_env())\n```\n\n## Contributing\n\nIf you would like to contribute to this library, please see the [Contribution Guidelines](CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flogicalclocks%2Ffeature-store-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flogicalclocks%2Ffeature-store-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flogicalclocks%2Ffeature-store-api/lists"}