https://github.com/feast-dev/feast-custom-online-store-demo
https://github.com/feast-dev/feast-custom-online-store-demo
Last synced: 9 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/feast-dev/feast-custom-online-store-demo
- Owner: feast-dev
- Created: 2021-08-20T16:57:19.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2021-10-24T06:15:27.000Z (about 4 years ago)
- Last Synced: 2025-03-26T10:11:17.343Z (10 months ago)
- Language: Python
- Size: 52.7 KB
- Stars: 1
- Watchers: 3
- Forks: 7
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Feast Custom Online Store
[](https://github.com/feast-dev/feast-custom-online-store-demo/actions/workflows/test_custom_online_store.yml)
### Overview
This repository demonstrates how developers can create their own custom `online store`s for Feast. Custom online stores allow users to use any underlying
data store to store features for low-latency retrieval, typically needed during model inference.
### Why create a custom online store?
Feast materializes data to online stores for low-latency lookup at model inference time. Typically, key-value stores are used for
the online stores, however relational databases can be used for this purpose as well.
Feast comes with some online stores built in, e.g, Sqlite, Redis, DynamoDB and Datastore. However, users can develop their
own online stores by creating a class that implements the contract in the [OnlineStore class](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/online_stores/online_store.py#L26).
### What is included in this repository?
* [feast_custom_online_store/](feast_custom_online_store): An example of a custom online store, `MySQLOnlineStore`, which implements `OnlineStore`. This example online store uses MySQL as the backing database.
* [feature_repo/](feature_repo): A simple feature repository that is used to test the custom online store. The repository has been configured to use the `MySQLOnlineStore` as part of it's `feature_store.yaml`
* [test_custom_online_store.py](test_custom_online_store.py): A test case that uses `MySQLOnlineStore` through the `feature_repo/`
### Testing the custom online store in this repository
Run the following commands to test the custom online store ([MySQLOnlineStore](https://github.com/feast-dev/feast-custom-online-store-demo/blob/master/feast_custom_online_store/mysql.py))
```bash
pip install -r requirements.txt
```
```
pytest test_custom_online_store.py
```
It is also possible to run a Feast CLI command, which interacts with the online store. It may be necessary to add the
`PYTHONPATH` to the path where your online store module is stored.
```
PYTHONPATH=$PYTHONPATH:/$(pwd) feast -c basic_feature_repo apply
```
```
Registered entity driver_id
Registered feature view driver_hourly_stats
Deploying infrastructure for driver_hourly_stats
```
```
$ PYTHONPATH=$PYTHONPATH:/$(pwd) feast -c feature_repo materialize-incremental 2021-08-19T22:29:28
```
```Materializing 1 feature views to 2021-08-19 15:29:28-07:00 into the feast_custom_online_store.mysql.MySQLOnlineStore online store.
driver_hourly_stats from 2020-08-24 05:23:49-07:00 to 2021-08-19 15:29:28-07:00:
100%|████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 120.59it/s]
```
### Testing against the Feast test suite
A subset of the Feast test suite, called "universal tests", are designed to test the core behavior of offline and online stores. A custom online store implementation can use the universal tests as follows.
First, this repository contains Feast as a submodule. To fetch and populate the directory, run
```
git submodule update --init --recursive
```
Next, install Feast following the instructions [here](https://github.com/feast-dev/feast/blob/master/CONTRIBUTING.md)
```
cd feast
pip install -e "sdk/python[ci]"
```
and confirm that the Feast unit tests run as expected:
```
make test
```
The Feast universal tests can be run with the command
```
make test-python-universal
```
If the command is run immediately, the tests should fail. The tests are parametrized based on the `FULL_REPO_CONFIGS` variable defined in `sdk/python/tests/integration/feature_repos/repo_configuration.py`. To overwrite these configurations, you can simply create your own file that contains a `FULL_REPO_CONFIGS`, and point Feast to that file by setting the environment variable `FULL_REPO_CONFIGS_MODULE` to point to that file. In this repo, the file that overwrites `FULL_REPO_CONFIGS` is `feast_custom_online_store/feast_tests.py`, so you would run
```
export FULL_REPO_CONFIGS_MODULE='feast_custom_online_store.feast_tests'
make test-python-universal
```
to test the MySQL online store against the Feast universal tests. You should notice that some of the tests actually fail; this indicates that there is a mistake in the implementation of this online store!