https://github.com/paulescu/backfill-feature-store-with-prefect
Backfill historical OHLC feature in a Feature Store (Hopsworks) using an orchestration tool (Prefect).
https://github.com/paulescu/backfill-feature-store-with-prefect
backfill dataengineering hopsworks machine-learning ml mlops prefect
Last synced: about 2 months ago
JSON representation
Backfill historical OHLC feature in a Feature Store (Hopsworks) using an orchestration tool (Prefect).
- Host: GitHub
- URL: https://github.com/paulescu/backfill-feature-store-with-prefect
- Owner: Paulescu
- Created: 2023-04-26T09:25:55.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-04-28T15:07:40.000Z (over 2 years ago)
- Last Synced: 2025-04-14T06:40:45.632Z (8 months ago)
- Topics: backfill, dataengineering, hopsworks, machine-learning, ml, mlops, prefect
- Language: Python
- Homepage: https://www.realworldml.net/subscribe
- Size: 142 KB
- Stars: 5
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
#### Table of contents
1. [What is this repo about?](#what-is-this-repo-about)
2. [How to run this code](#how-to-run-this-code)
3. [Wannna build real-world ML products?](#wannna-build-real-world-ml-products)
## What is this repo about?
This repository shows how to backfill historical OHLC feature in a Feature Store (Hopsworks) using an orchestration tool (Prefect).
## How to run this code
- Create a Python virtual environment with the project dependencies with
```
$ make init
```
- Connect to your Prefect Cloud
```
$ prefect cloud login
```
- Set environment variables necessary to talk to your Hopsworks feature store
```
$ . ./set_hopsworks_credentials.sh
```
- Backfill OHLC data for a range of dates (e.g. from `2023-01-01` to `2023-01-31`)
```
$ make from_day=2023-01-01 to_day=2023-01-31 backfill
```
## Wannna build real-world ML products?
Check the [Real-World ML Program](https://realworldmachinelearning.carrd.co/), a hands-on, 3-hour course where you will learn
how to design, build, [deploy](https://taxi-demand-predictor.streamlit.app/), and [monitor](https://taxi-demand-predictor-monitoring.streamlit.app/) complete ML products.