{"id":16926893,"url":"https://github.com/shcheklein/ensemble-dvc-template","last_synced_at":"2025-04-11T17:41:58.854Z","repository":{"id":133373955,"uuid":"559309448","full_name":"shcheklein/ensemble-dvc-template","owner":"shcheklein","description":"Skeleton for DVC pipeline to evaluate multiple models together","archived":false,"fork":false,"pushed_at":"2023-12-15T05:49:07.000Z","size":38,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T13:45:33.250Z","etag":null,"topics":["data-science","dvc","dvc-pipeline","ensemble","ensemble-machine-learning","ensemble-model","machine-learning","pipeline"],"latest_commit_sha":null,"homepage":"https://dvc.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shcheklein.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-10-29T17:59:55.000Z","updated_at":"2024-06-18T11:21:47.000Z","dependencies_parsed_at":"2023-11-09T13:00:41.075Z","dependency_job_id":"129a2463-48db-4730-bd56-edbee1a31aae","html_url":"https://github.com/shcheklein/ensemble-dvc-template","commit_stats":{"total_commits":22,"total_committers":1,"mean_commits":22.0,"dds":0.0,"last_synced_commit":"32410f4d2ed58d416f1f211d46e6ed7c196b051b"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shcheklein%2Fensemble-dvc-template","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shcheklein%2Fensemble-dvc-template/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shcheklein%2Fensemble-dvc-template/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shcheklein%2Fensemble-dvc-template/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shcheklein","download_url":"https://codeload.github.com/shcheklein/ensemble-dvc-template/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248451568,"owners_count":21105891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","dvc","dvc-pipeline","ensemble","ensemble-machine-learning","ensemble-model","machine-learning","pipeline"],"created_at":"2024-10-13T20:32:04.335Z","updated_at":"2025-04-11T17:41:58.834Z","avatar_url":"https://github.com/shcheklein.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DVC Model Ensemble\n\nSkeleton for DVC pipeline to evaluate multiple models together.\nAn experiment in response to this\n[SO question](https://stackoverflow.com/questions/74236782/experiment-tracking-for-multiple-ml-independent-models-using-wandb-in-a-single-m).\nThere is no actual ML model train code, no frameworks, no actual data. This is\nmade to showcase the possible project layout, how to run this pipeline to see\nmetrics, plots, etc.\n\nExplore [CLI](#cli), [VS Code extension](#vs-code-extension),\n[Studio](#studio), and [Codespaces](#codespaces) tools to experiments,\nvisualize, and share the results.\n\n## Stucture\n\n```\n.\n├── LICENSE\n├── README.md\n\n# Project / pipeline definition (dvc.yaml) and\n# project artifacts and dependecies snapshot (dvc.lock)\n\n├── dvc.lock\n├── dvc.yaml\n\n# Metrics, plots that are logger via `dvclive` logger\n# https://dvc.org/doc/dvclive\n# This is similar to TB, W\u0026B, MLFlow, etc loggers\n\n├── dvclive\n│   ├── model-1\n│   │   ├── metrics.json\n│   │   └── plots\n│   │       └── metrics\n│   │           └── acc.tsv\n\n...  # Any number of models\n\n│   └── model-N\n\n# Evaluation script that can read multiple models or their metrics\n# and dumps aggregare metrics\n\n├── evaluate.py\n├── evaluation\n│   └── metrics.json\n\n# Mode data, train, and model itself.\n# If data and model are large they could be tracked and saved to any\n# supported remote storage. Here we put them in Git for simplicity.\n\n├── model-1\n│   ├── data\n│   │   └── data.csv\n│   ├── data.dvc\n│   ├── model.pkl\n│   ├── params.yaml\n│   └── train.py\n\n...  # Any number of model\n\n├── model-N\n    ...\n└── requirements.txt\n```\n\n## CLI\n\nUsing set of the\n[`dvc exp`](https://dvc.org/doc/start/experiment-management/experiments)\ncommands it's possible to iterate on models and compare different iterations\nwith each other.\n\n```\nvim model-1/params.yaml. # change params\ndvc exp run              # run an experiment \ndvc exp show             # show all experiments\n\n# Queue multiple experiments and run them:\ndvc exp run --queue -S model-1/params.yaml:res=0.8,0.82,0.84,0.86\ndvc exp run --run-all\n\n# Show experimnents again:\ndvc exp show\n```\n\nhttps://user-images.githubusercontent.com/3659196/198862257-ad0e4cc0-8265-47a2-98ac-cae590f25373.mp4\n\nAlso, it's possible to show plots with:\n\n```\ndvc plots show\n# or\ndvc plots diff\n```\n\n\u003cimg width=\"1156\" alt=\"Screen Shot 2022-10-30 at 3 35 23 PM\" src=\"https://user-images.githubusercontent.com/3659196/198904947-02b339eb-8573-4ffe-9430-5c382020a463.png\"\u003e\n\n## VS Code Extension\n\nCan be installed from the\n[marketplace](https://marketplace.visualstudio.com/items?itemName=Iterative.dvc)\nand provides a visual layer for DVC experiments, plots, and common actions for\ndata management.\n\nhttps://user-images.githubusercontent.com/3659196/198889808-488193f9-c4fa-4f7f-9287-dbc4bf8774be.mp4\n\n\n## Studio\n\nOpen\n[public project](https://studio.iterative.ai/user/shcheklein/projects/ensemble-dvc-template-7u1t1fys3z)\nfor this repository.\n\n[Studio](https://studio.iterative.ai/) (see docs [here](https://dvc.org/doc/studio))\nprovides a collaborative interface to share experiments, see and manage ML models in\n[model registry](https://dvc.org/doc/studio/user-guide/model-registry/what-is-a-model-registry).\n\n\nhttps://user-images.githubusercontent.com/3659196/198904752-807a9c8e-7f80-4fea-ba21-efaac4bd91ef.mp4\n\n## Codespaces\n\nProject also can be run in the\n[GitHub Codespaces](https://github.com/features/codespaces) for the in browser or descktop\nplatform that is deployed with one click:\n\nhttps://user-images.githubusercontent.com/3659196/198906563-b451b951-13f2-45ac-883c-8f03709622c0.mp4\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshcheklein%2Fensemble-dvc-template","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshcheklein%2Fensemble-dvc-template","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshcheklein%2Fensemble-dvc-template/lists"}