https://github.com/kuutsav/MLOps

MLOps template with examples for Data pipelines, ML workflow management, API development and Monitoring
https://github.com/kuutsav/MLOps

dagster fastapi minio mlflow mlops

Last synced: 3 months ago
JSON representation

MLOps template with examples for Data pipelines, ML workflow management, API development and Monitoring

• Host: GitHub
• URL: https://github.com/kuutsav/MLOps
• Owner: kuutsav
• Archived: true
• Created: 2021-10-09T09:50:22.000Z (about 3 years ago)
• Default Branch: master
• Last Pushed: 2023-12-23T19:04:47.000Z (11 months ago)
• Last Synced: 2024-04-24T12:48:18.339Z (7 months ago)
• Topics: dagster, fastapi, minio, mlflow, mlops
• Language: Python
• Homepage:
• Size: 5.61 MB
• Stars: 43
• Watchers: 7
• Forks: 11
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# MLOps

[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)

A project based example of Data pipelines, ML workflow management, API endpoints

and Monitoring.

![MLops](docs/images/mlops.png)

## Tools used

- Data Pipeline: [Dagster](https://github.com/dagster-io/dagster)

- ML workflow: [MLflow](https://github.com/mlflow/mlflow)

- API Deployment: [FastAPI](https://github.com/tiangolo/fastapi)

- Monitoring: [ElasticAPM](https://www.elastic.co/apm/)

## Requirements

### Poetry (dependency management)

```bash

$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

$ poetry --version

# Poetry version 1.1.10

```

### pre-commit (static code analysis)

```bash

$ pip install pre-commit

$ pre-commit --version

# pre-commit 2.15.0

```

### Minio (s3 compatible object storage)

Follow the instructions here - https://min.io/download

## Setup

### Environment setup

```bash

$ poetry install

```

### MLflow

```bash

$ poetry shell

$ export MLFLOW_S3_ENDPOINT_URL=http://127.0.0.1:9000

$ export AWS_ACCESS_KEY_ID=minioadmin

$ export AWS_SECRET_ACCESS_KEY=minioadmin

# make sure that the backend store and artifact locations are same in the .env file as well

$ mlflow server \

    --backend-store-uri sqlite:///mlflow.db \

    --default-artifact-root s3://mlflow \

    --host 0.0.0.0

```

### Minio

```bash

$ export MINIO_ROOT_USER=minioadmin

$ export MINIO_ROOT_PASSWORD=minioadmin

$ mkdir minio_data

$ minio server minio_data --console-address ":9001"

# API: http://192.168.29.103:9000  http://10.119.80.13:9000  http://127.0.0.1:9000

# RootUser: minioadmin

# RootPass: minioadmin

# Console: http://192.168.29.103:9001 http://10.119.80.13:9001 http://127.0.0.1:9001

# RootUser: minioadmin

# RootPass: minioadmin

# Command-line: https://docs.min.io/docs/minio-client-quickstart-guide

#    $ mc alias set myminio http://192.168.29.103:9000 minioadmin minioadmin

# Documentation: https://docs.min.io

```

Go to http://127.0.0.1:9001/buckets/ and create a bucket called `mlflow`.

### Dagster

```bash

$ poetry shell

$ dagit -f mlops/pipeline.py

```

### ElasticAPM

```bash

$ docker-compose -f docker-compose-monitoring.yaml up

```

### FastAPI

```bash

$ poetry shell

$ export PYTHONPATH=.

$ python mlops/app/application.py

```