{"id":13719984,"url":"https://github.com/dmesquita/dvc_pipelines_and_experiments_tutorial","last_synced_at":"2025-05-07T12:30:36.175Z","repository":{"id":71848861,"uuid":"277385962","full_name":"dmesquita/dvc_pipelines_and_experiments_tutorial","owner":"dmesquita","description":"Building a maintainable Machine Learning pipeline using DVC","archived":false,"fork":false,"pushed_at":"2020-07-07T13:38:57.000Z","size":39,"stargazers_count":16,"open_issues_count":1,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-14T09:39:03.835Z","etag":null,"topics":["dvc","dvc-for-experiment-management","experiments","machine-learning","pipelines","project-management","reproducibility"],"latest_commit_sha":null,"homepage":"https://towardsdatascience.com/the-ultimate-guide-to-building-maintainable-machine-learning-pipelines-using-dvc-a976907b2a1b","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmesquita.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-07-05T21:26:01.000Z","updated_at":"2024-04-18T10:56:47.000Z","dependencies_parsed_at":"2023-02-27T23:30:47.605Z","dependency_job_id":null,"html_url":"https://github.com/dmesquita/dvc_pipelines_and_experiments_tutorial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmesquita%2Fdvc_pipelines_and_experiments_tutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmesquita%2Fdvc_pipelines_and_experiments_tutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmesquita%2Fdvc_pipelines_and_experiments_tutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmesquita%2Fdvc_pipelines_and_experiments_tutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmesquita","download_url":"https://codeload.github.com/dmesquita/dvc_pipelines_and_experiments_tutorial/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252876284,"owners_count":21818155,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dvc","dvc-for-experiment-management","experiments","machine-learning","pipelines","project-management","reproducibility"],"created_at":"2024-08-03T01:00:58.576Z","updated_at":"2025-05-07T12:30:35.035Z","avatar_url":"https://github.com/dmesquita.png","language":"Python","funding_links":[],"categories":["Tutorials"],"sub_categories":[],"readme":"# Building a maintainable Machine Learning pipeline using DVC\n\nThis guides uses the [DVC Get Started Guide](https://github.com/iterative/example-get-started)\nas a starting point and takes you on **how\nto build maintainable Machine Learning pipelines using DVC**.\n\nIf you have some time you can check the full article [here](https://towardsdatascience.com/the-ultimate-guide-to-building-maintainable-machiane-learning-pipelines-using-dvc-a976907b2a1b) (it has more in depth explanations than this readme :wink:)\n\nThe principles are:\n- **Write a python script** for each pipeline step\n- **Save the parameters** each script uses in a `yaml` file\n- Specify the files each script **depends on**\n- Specify the files each script **generates**\n\nIn this tutorial we're going to build a model to classify the 20newsgroups dataset.\n\n*Environment*: Linux with **Python 3**, **pip** and **Git** installed\n\n## First: installing DVC as a Python library\n```console\n$ mkdir dvc_tutorial\n$ cd dvc_tutorial\n$ python3 -m venv .env\n$ source .env/bin/activate\n(.env)$ pip3 install dvc\n(.env)$ git init\n(.env)$ dvc init\n```\n\n## 1 - Create a `params.yaml` file\n```\n# file params.yaml\nprepare:\n    categories:\n        - comp.graphics\n        - sci.space\n```\n\n## 2 - Create the `prepare.py` script\nSave the file  `prepare.py` file (it's available here on this repo) inside `/src`. Your folder structure should look like this:\n```\n├── params.yaml\n└── src\n    └── prepare.py\n```\n\n## 3 - Create the `prepare.py` stage usinf DVC\nThe steps for doing that are:\n- Write a python script: `prepare.py`\n- Save the parameters: `categories` inside `params.yaml`\n- Specify the files the script depends on: `prepare.py`\n- Specify the files the script generates: the folder `data/prepared`\n- Defined the command line instruction to run this step\n\n```console\n(.env)$ pip install pyyaml scikit-learn pandas\n\n(.env)$ dvc run -n prepare -p prepare.categories -d src/prepare.py -o data/prepared python3 src/prepare.py\n```\n\n## 4 - Create the scripts and the stages for all the other steps\n```\n(.env)$ dvc run -n featurize -d src/featurize.py -d data/prepared -o data/features python3 src/featurize.py data/prepared data/features\n\n(.env)$ dvc run -n train -p train.alpha -d src/train.py -d data/features -o model.pkl python3 src/train.py data/features model.pkl\n\n(.env)$ dvc run -n evaluate -d src/evaluate.py -d model.pkl -d data/features --metrics-no-cache scores.json --plots-no-cache plots.json python3 src/evaluate.py model.pkl data/features scores.json plots.json\n```\n\n## 5 - Change parameters\n```\n# file params.yaml\nprepare:\n    categories:\n        - comp.graphics\n        - rec.sport.baseball\ntrain:\n    alpha: 0.9\n```\n## 6 - Run the pipeline\n```console\n(.env)$ dvc repro\n```\n\n## 7 -  Compare the metrics\n```console\n(.env)$ dvc params diff\n\n(.env)$ dvc metrics diff\n```\n\n## 8 - Visualize and compare metrics using plots\n```console\n(.env)$ dvc plots show -y precision -x recall plots.json\n\n(.env)$ dvc plots diff --targets plots.json -y precision\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmesquita%2Fdvc_pipelines_and_experiments_tutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmesquita%2Fdvc_pipelines_and_experiments_tutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmesquita%2Fdvc_pipelines_and_experiments_tutorial/lists"}