{"id":21070695,"url":"https://github.com/pyronear/pyro-mlops","last_synced_at":"2025-05-16T05:30:59.985Z","repository":{"id":164461448,"uuid":"635654024","full_name":"pyronear/pyro-mlops","owner":"pyronear","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-14T11:09:11.000Z","size":163034,"stargazers_count":6,"open_issues_count":1,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-08T07:23:29.144Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pyronear.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-03T07:00:09.000Z","updated_at":"2025-02-14T11:09:13.000Z","dependencies_parsed_at":"2024-04-12T08:16:43.036Z","dependency_job_id":"b3d951a5-c004-4494-a31d-f4a7c41eb9ca","html_url":"https://github.com/pyronear/pyro-mlops","commit_stats":{"total_commits":13,"total_committers":3,"mean_commits":4.333333333333333,"dds":0.5384615384615384,"last_synced_commit":"80b47841bc0d09ac662f5164dfbed86e193cc642"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyronear%2Fpyro-mlops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyronear%2Fpyro-mlops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyronear%2Fpyro-mlops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyronear%2Fpyro-mlops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pyronear","download_url":"https://codeload.github.com/pyronear/pyro-mlops/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254473844,"owners_count":22077185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T18:47:51.980Z","updated_at":"2025-05-16T05:30:59.979Z","avatar_url":"https://github.com/pyronear.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pyronear: Machine Learning Pipeline for Wildfire Detection 🔥\n\n![Pyronear Logo](./docs/assets/images/pyronear_logo.png)\n\nMachine Learning Pipeline for Wildfire Detection.\n\n![Pipeline Overview](./docs/assets/images/pipeline.png)\n\n## Setup\n\n### Dependencies\n\n- [Poetry](https://python-poetry.org/): Python packaging and dependency\nmanagement - Install it with something like `pipx`\n- [Git LFS](https://git-lfs.com/): Git Large File Storage replaces large\nfiles such as jupyter notebooks with text pointers inside Git while\nstoring the file contents on a remote server like github.com\n- [DVC](https://dvc.org/): Data Version Control  - This will get\ninstalled automatically\n- [MLFlow](https://mlflow.org/): ML Experiment Tracking - This will get\ninstalled automatically\n\n### Install\n\n#### Poetry\n\nFollow the [official documentation](https://python-poetry.org/docs/) to install `poetry`.\n\n#### Git LFS\n\nMake sure [`git-lfs`](https://git-lfs.com/) is installed on your system.\n\nRun the following command to check:\n\n```sh\ngit lfs install\n```\n\nIf not installed, one can install it with the following:\n\n##### Linux\n\n```sh\nsudo apt install git-lfs\ngit-lfs install\n```\n\n##### Mac\n\n```sh\nbrew install git-lfs\ngit-lfs install\n```\n\n##### Windows\n\nDownload and run the latest [windows installer](https://github.com/git-lfs/git-lfs/releases).\n\n#### Project Dependencies\n\nCreate a virtualenv and install python version with conda - or use a\ncombination of pyenv and venv:\n\n```sh\nconda create -n pyronear-mlops python=3.12\n```\n\nActivate the virtual environment:\n\n```sh\nconda activate pyronear-mlops\n```\n\nInstall python dependencies\n\n```sh\npoetry install\n```\n\n#### Data Dependencies\n\nTo get the data dependencies one can use DVC - To fully use this\nrepository you would need access to our DVC remote storage which is\ncurrently reserved for Pyronear members. On request, you will be provided with\nAWS credentials to access our remote storage.\n\nOnce setup, run the following command:\n\n```sh\ndvc pull\n```\n\n![Random batch sample from the dataset](./docs/assets/images/batch.jpg)\n\n##### Setup S3 access\n\nCreate the following file `~/.aws/config`:\n\n```toml\n[profile pyronear]\nregion = eu-west-3\n```\n\nAdd your credentials in the file `~/.aws/credentials` - replace `XXX`\nwith your access key id and your secret access key:\n\n```toml\n[pyronear]\naws_access_key_id = XXX\naws_secret_access_key = XXX\n```\n\nMake sure you use the AWS `pyronear` profile:\n\n```bash\nexport AWS_PROFILE=pyronear\n```\n\n## Project structure and conventions\n\nThe project is organized following mostly the [cookie-cutter-datascience\nguideline](https://drivendata.github.io/cookiecutter-data-science/#directory-structure).\n\n### Data\n\nAll the data lives in the `data` folder and follows some [data engineering\nconventions](https://docs.kedro.org/en/stable/faq/faq.html#what-is-data-engineering-convention).\n\n### Library Code\n\nThe library code is available under the `pyronear_mlops` folder.\n\n### Notebooks\n\nThe notebooks live in the `notebooks` folder. They are automatically synced to\nthe Git LFS storage.\nPlease follow [this\nconvention](https://drivendata.github.io/cookiecutter-data-science/#notebooks-are-for-exploration-and-communication)\nto name your Notebooks.\n\n`\u003cstep\u003e-\u003cghuser\u003e-\u003cdescription\u003e.ipynb` - e.g., `0.3-mateo-visualize-distributions.ipynb`.\n\n### Scripts\n\nThe scripts live in the `scripts` folder, they are\ncommonly CLI interfaces to the library\ncode.\n\n## DVC\n\nDVC is used to track and define data pipelines and make them\nreproducible. See `dvc.yaml`.\n\nTo get an overview of the pipeline DAG:\n\n```sh\ndvc dag\n```\n\nTo run the full pipeline:\n\n```sh\ndvc repro\n```\n\n## MLFlow\n\nAn MLFlow server is running when running ML experiments to track\nhyperparameters and performances and to streamline model\nselection.\n\nTo start the mlflow UI server, run the following command:\n\n```sh\nmake mlflow_start\n```\n\nTo stop the mlflow UI server, run the following command:\n\n```sh\nmake mlflow_stop\n```\n\nTo browse the different runs, open your browser and navigate to the URL:\n[http://localhost:5000](http://localhost:5000)\n\n## Contribute to the project\n\n### New ML experiments\n\nFollow the steps:\n\n1. Work on a separate git branch: `git checkout -b \"\u003cuser\u003e/\u003cexperiment-name\u003e\"`\n2. Modify and iterate on the code, then run `dvc repro`. It will rerun\n   parts of the pipeline that have been updated.\n3. Commit your changes and open a Pull Request to get your changes\n   approved and merged.\n\n### Run Random Hyperparameter Search\n\n#### YOLOv8\n\nUse the following commands to run random hyperparameter search:\n\n```sh\nmake run_yolov8_hyperparameter_search\n```\n\nIt will run 10 random training runs with hyperparameters drawn from the\nhyperparameter space defined in\n`pyronear_mlops/model/yolo/hyperparameters/yolov8.py`\n\n#### YOLOv9\n\nUse the following commands to run random hyperparameter search:\n\n```sh\nmake run_yolov9_hyperparameter_search\n```\n\nIt will run 10 random training runs with hyperparameters drawn from the\nhyperparameter space defined in\n`pyronear_mlops/model/yolo/hyperparameters/yolov9.py`\n\n### Generate a benchmark CSV file\n\n#### YOLOv8\n\n```sh\nmake yolov8_benchmark\n```\n\n#### YOLOv9\n\n```sh\nmake yolov9_benchmark\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyronear%2Fpyro-mlops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpyronear%2Fpyro-mlops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyronear%2Fpyro-mlops/lists"}