{"id":24750569,"url":"https://github.com/fabioba/mlops-architecture","last_synced_at":"2025-09-08T12:42:37.925Z","repository":{"id":231612405,"uuid":"550383832","full_name":"fabioba/mlops-architecture","owner":"fabioba","description":"This is an overview of a MLOps architecture that includes both Airflow and MLflow running on separate Docker containers.","archived":false,"fork":false,"pushed_at":"2022-10-18T20:49:26.000Z","size":145,"stargazers_count":21,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T02:41:30.414Z","etag":null,"topics":["airflow","docker","docker-compose","machine-learning","mlflow","mlops","python","workflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fabioba.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-10-12T17:10:21.000Z","updated_at":"2025-01-28T18:21:59.000Z","dependencies_parsed_at":"2024-04-04T22:45:21.808Z","dependency_job_id":null,"html_url":"https://github.com/fabioba/mlops-architecture","commit_stats":null,"previous_names":["fabioba/mlops-architecture"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/fabioba/mlops-architecture","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabioba%2Fmlops-architecture","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabioba%2Fmlops-architecture/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabioba%2Fmlops-architecture/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabioba%2Fmlops-architecture/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fabioba","download_url":"https://codeload.github.com/fabioba/mlops-architecture/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabioba%2Fmlops-architecture/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274187838,"owners_count":25237845,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-08T02:00:09.813Z","response_time":121,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","docker","docker-compose","machine-learning","mlflow","mlops","python","workflow"],"created_at":"2025-01-28T09:08:20.797Z","updated_at":"2025-09-08T12:42:37.877Z","avatar_url":"https://github.com/fabioba.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AIRFLOW_MLFLOW_DOCKER\n![test](https://user-images.githubusercontent.com/31510474/196541725-ed4c7fca-4521-48d8-95e0-c7beaa6aa662.png)\n\n\n## Table of content\n- [Background](#background)\n- [Tools Overview](#tools_overview)\n- [Getting started](#getting_started)\n    * [Docker Compose configuration](#docker_config)\n    * [Airflow](#airflow)\n    * [MLflow](#mlflow)\n- [Connect Airflow to MLflow](#airflow_and_mlflow)\n- [References](#references)\n\n\n\u003ca name=\"background\"/\u003e\n\n## Background\nThe goal of this project is to create an ecosystem where to run **Data Pipelines** and monitor **Machine Learning Experiments**.\n\n\u003ca name=\"tools_overview\"/\u003e\n\n## Tools Overview\nFrom `Airflow` documentation:\n```\nApache Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows\n```\n\nFrom `MLflow` documentation:\n```\nMLflow is an open source platform for managing the end-to-end machine learning lifecycle\n```\n\nFrom `Docker` documentation:\n```\nDocker Compose is a tool for defining and running multi-container Docker applications.\n```\n\n\u003ca name=\"getting_started\"/\u003e\n\n## Getting Started\nThe first step to structure this project is connecting `Airflow` and `MLflow` together: `docker compose`.\n\n\n\u003ca name=\"docker_config\"/\u003e\n\n### Docker Compose Configuration\nCreate `docker-compose.yaml`, which contains the configuration of those docker containers responsible for running `Airflow` and `MLflow` services. \nEach of those services runs on a different container:\n* airflow-webserver\n* airflow-scheduler\n* airflow-worker\n* airflow-triggerer\n* mlflow \n\nTo create and start multiple container, from terminal run the following command:\n```\ndocker compose up -d\n```\n\n\u003ca name=\"airflow\"/\u003e\n\n### Airflow\nIn order to access to `Airflow server` visit the page: `localhost:8080`\n\n![img](docs/imgs/airflow_home.png)\n\nAnd take a step into `Airflow` world!\n\nTo start creating DAGS initialize an empty folder named `dags` and populate it with as many scripts as you need.\n```bash\n└── dags\n     └── example_dag.py\n```\n\n\u003ca name=\"mlflow\"/\u003e\n\n### MLFlow\nIn order to monitor `MLflow experiments` through its server, visit the page: `localhost:600`\n\n![img](docs/imgs/mlflow_home.png)\n\n\u003ca name=\"airflow_and_mlflow\"/\u003e\n\n## Connect Airflow to MLflow \nTo establish a connection between `Airflow` and `MLflow`, define the URI of the `MLflow server`:\n```\nmlflow.set_tracking_uri('http://mlflow:600')\n```\n\nAfter that, create a new connection on `Airflow` that points to that port.\n\u003cimg width=\"1418\" alt=\"image\" src=\"https://user-images.githubusercontent.com/31510474/196529307-3f2df479-a11b-4a0c-bf73-090b82024907.png\"\u003e\n\n\n\u003ca name=\"references\"/\u003e\n\n## References\n* [Airflow Docker](https://airflow.apache.org/docs/apache-airflow/2.0.1/start/docker.html)\n* [What is Airflow?](https://airflow.apache.org/docs/apache-airflow/stable/index.html)\n* [MLflow](https://mlflow.org/docs/latest/index.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabioba%2Fmlops-architecture","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffabioba%2Fmlops-architecture","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabioba%2Fmlops-architecture/lists"}