{"id":13738470,"url":"https://github.com/kuutsav/MLOps","last_synced_at":"2025-05-08T16:34:05.826Z","repository":{"id":127629448,"uuid":"415267833","full_name":"kuutsav/mlops","owner":"kuutsav","description":"Template for data pipelines, ML workflows, API dev and monitoring","archived":true,"fork":false,"pushed_at":"2023-12-23T19:04:47.000Z","size":5881,"stargazers_count":44,"open_issues_count":0,"forks_count":11,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-08-04T03:12:35.629Z","etag":null,"topics":["dagster","fastapi","minio","mlflow","mlops"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kuutsav.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-10-09T09:50:22.000Z","updated_at":"2024-07-14T02:59:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"c86b65b0-d3d2-4189-b3ea-b6c9fceef65f","html_url":"https://github.com/kuutsav/mlops","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuutsav%2Fmlops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuutsav%2Fmlops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuutsav%2Fmlops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuutsav%2Fmlops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kuutsav","download_url":"https://codeload.github.com/kuutsav/mlops/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224746763,"owners_count":17363108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dagster","fastapi","minio","mlflow","mlops"],"created_at":"2024-08-03T03:02:23.373Z","updated_at":"2024-11-15T07:31:10.360Z","avatar_url":"https://github.com/kuutsav.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# MLOps\n\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)\n\nA project based example of Data pipelines, ML workflow management, API endpoints\nand Monitoring.\n\n![MLops](docs/images/mlops.png)\n\n## Tools used\n\n- Data Pipeline: [Dagster](https://github.com/dagster-io/dagster)\n- ML workflow: [MLflow](https://github.com/mlflow/mlflow)\n- API Deployment: [FastAPI](https://github.com/tiangolo/fastapi)\n- Monitoring: [ElasticAPM](https://www.elastic.co/apm/)\n\n## Requirements\n\n### Poetry (dependency management)\n\n```bash\n$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -\n$ poetry --version\n# Poetry version 1.1.10\n```\n\n### pre-commit (static code analysis)\n\n```bash\n$ pip install pre-commit\n$ pre-commit --version\n# pre-commit 2.15.0\n```\n\n### Minio (s3 compatible object storage)\n\nFollow the instructions here - https://min.io/download\n\n## Setup\n\n### Environment setup\n\n```bash\n$ poetry install\n```\n\n### MLflow\n\n```bash\n$ poetry shell\n$ export MLFLOW_S3_ENDPOINT_URL=http://127.0.0.1:9000\n$ export AWS_ACCESS_KEY_ID=minioadmin\n$ export AWS_SECRET_ACCESS_KEY=minioadmin\n\n# make sure that the backend store and artifact locations are same in the .env file as well\n$ mlflow server \\\n    --backend-store-uri sqlite:///mlflow.db \\\n    --default-artifact-root s3://mlflow \\\n    --host 0.0.0.0\n```\n\n### Minio\n\n```bash\n$ export MINIO_ROOT_USER=minioadmin\n$ export MINIO_ROOT_PASSWORD=minioadmin\n\n$ mkdir minio_data\n$ minio server minio_data --console-address \":9001\"\n\n# API: http://192.168.29.103:9000  http://10.119.80.13:9000  http://127.0.0.1:9000\n# RootUser: minioadmin\n# RootPass: minioadmin\n\n# Console: http://192.168.29.103:9001 http://10.119.80.13:9001 http://127.0.0.1:9001\n# RootUser: minioadmin\n# RootPass: minioadmin\n\n# Command-line: https://docs.min.io/docs/minio-client-quickstart-guide\n#    $ mc alias set myminio http://192.168.29.103:9000 minioadmin minioadmin\n\n# Documentation: https://docs.min.io\n```\n\nGo to http://127.0.0.1:9001/buckets/ and create a bucket called `mlflow`.\n\n### Dagster\n\n```bash\n$ poetry shell\n$ dagit -f mlops/pipeline.py\n```\n\n### ElasticAPM\n\n```bash\n$ docker-compose -f docker-compose-monitoring.yaml up\n```\n\n### FastAPI\n\n```bash\n$ poetry shell\n$ export PYTHONPATH=.\n$ python mlops/app/application.py\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuutsav%2FMLOps","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkuutsav%2FMLOps","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuutsav%2FMLOps/lists"}