{"id":29102922,"url":"https://github.com/pyronear/pyro-eval","last_synced_at":"2025-09-12T22:33:15.828Z","repository":{"id":301178590,"uuid":"1008406829","full_name":"pyronear/pyro-eval","owner":"pyronear","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-30T10:48:22.000Z","size":2518,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-31T12:38:03.520Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pyronear.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-25T13:50:17.000Z","updated_at":"2025-07-30T10:48:26.000Z","dependencies_parsed_at":"2025-07-14T12:25:53.451Z","dependency_job_id":"a7279293-68c7-4c51-8b46-824dd1fd9f51","html_url":"https://github.com/pyronear/pyro-eval","commit_stats":null,"previous_names":["pyronear/pyro-eval"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pyronear/pyro-eval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyronear%2Fpyro-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyronear%2Fpyro-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyronear%2Fpyro-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyronear%2Fpyro-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pyronear","download_url":"https://codeload.github.com/pyronear/pyro-eval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyronear%2Fpyro-eval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274886944,"owners_count":25368227,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-12T02:00:09.324Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-28T22:44:44.549Z","updated_at":"2025-09-12T22:33:15.813Z","avatar_url":"https://github.com/pyronear.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pyro-eval\n\nLibrary to evaluate Pyronear ML models 🔥\n\n## Context\n\nThis module aims at providing an evaluation pipeline to measure and commpare\nthe performance of pyronear algorithms. It is split in two parts:\n\n- Dataset management\n- Metrics computation\n\n## Installation\n\n### Python dependencies\n\nMake sure you have [uv](https://docs.astral.sh/uv/getting-started/installation/) installed, then clone this repo and install dependencies:\n\n```bash\ngit clone git@github.com:earthtoolsmaker/pyro-eval.git\nuv sync\n```\n\n__Note__:\n\nThis repo the [pyro-engine](https://github.com/pyronear/pyro-engine) repo as a\ndependency: Make sure to run `uv sync` to retrieve changes made on this\nrepo.\n\n### Data dependencies\n\nTo get the data dependencies one can use DVC - To fully use this\nrepository you would need access to our DVC remote storage which is\ncurrently reserved for Pyronear members. On request, you will be provided with\nAWS credentials to access our remote storage.\n\nPull all the data files tracked by DVC using this command:\n\n```sh\ndvc pull\n```\n\n### Scaffolding\n\nOne can use the default `./data` folder to store datasets and models to run\nevaluation on:\n\n- __Models__: One can use the `./data/models/` folder to store models to evaluate.\n- __Datasets__: One can use the `./data/datasets/` folder to store the datasets.\n- __Evaluation Results__: By default, the results of the evaluation runs are\nstored under `./data/evaluation/`.\n\nExample of files under `./data/`:\n\n```bash\n$ tree -L 3\n.\n├── datasets\n│   ├── gitkeep\n│   ├── wildfire_test\n│   │   ├── data.yaml\n│   │   ├── images\n│   │   ├── labels\n│   │   └── wildfire_test.csv\n│   └── wildfire_test_temporal\n│       ├── data.yaml\n│       ├── images\n│       ├── labels\n│       └── wildfire_test_temporal.csv\n├── evaluation\n│   ├── gitkeep\n│   └── runs\n│       └── run-20250522-1457-7552\n└── models\n    ├── artistic-alpaca_v1.1.0_fe129f2.onnx\n    ├── artistic-alpaca_v1.1.0_fe129f2.pt\n    └── gitkeep\n```\n\n## Usage\n\n### run_evaluation.py\n\nThis script runs the evaluation of the models on the provided test dataset.\n\n```bash\nuv run python ./scripts/run_evaluation.py \\\n  --dir-models ./data/models/ \\\n  --dir-dataset ./data/datasets/wildfire_test/ \\\n  --dir-temporal-dataset ./data/datasets/wildfire_test_temporal/ \\\n  --dir-save ./data/evaluation/runs/ \\\n  --device cuda \\\n  --loglevel info\n```\n\n## Evaluation Pipeline Design\n\nThe evaluation pipeline is composed of two steps: data preparation and metrics\ncomputation, respectively managed by the `EvaluationDataset` and\n`EvaluationPipeline` classes.\n\n### EvaluationDataset\n\nThe `EvaluationDataset` class helps creating a custom dataset object suited for\nmetric computation.\n\nThe object is instanciated from an existing image folder or a hugging face\nrepo. A dataset ID can be passed as input, by default the id will be computed\nfrom the current date and a custom hash of the dataset.\nWhen instanciating from a local folder, the following rules must be follow to\nensure a proper functioning of the class:\n\n- The root folder must contain one subfolder named `images` and one named\n`labels`\n- The `images` folder must contain the images files, named with the following\nconvention : `*_Y-m-dTH-M-S.jpg`, for example:\n`seq_44_sdis-07_brison-200_2024-02-16T16-38-22.jpg`\n- `labels` folder must contain a label `.txt` file in the YOLOv8 TXT format for\neach image with the coordinates of the groundtruth bounding box\n\n```txt\ndataset\n├── images\n│   ├── image1.jpg\n│   └── image2.jpg\n│   └── image2.jpg\n├── labels\n│   ├── image1.txt\n│   └── image2.txt\n    └── image2.txt\n```\n\n```python\ndatapath = \"path/to/dataset\"\ndataset_ID = \"dataset_v0\"\ndataset = EvaluationDataset(datapath, dataset_ID=dataset_ID)\n```\n\n`dir-dataset` is used to evaluate the model, while `dir-temporal-dataset` is used to evaluate the engine on sequences of images.\n\n### EvaluationPipeline\n\nThe EvaluationPipeline class helps launching the evaluation on a given dataset.\nThe evaluation is launched as follows:\n\n```python\nevaluation = EvaluationPipeline(dataset=dataset)\nevaluation.run()\nevaluation.save_metrics()\n```\n\nThe complete evaluation is composed of two part : `ModelEvaluator`, which\nprovides metrics on the model performance alone, and `EngineEvaluator` which\nprovides metrics on the whole detection pipeline in the PyroEngine.\n\nThe object can be instanciated with the following parameters as input:\n\n- `self.dataset` : `EvaluationDataset` object\n- `self.config` : config dictionary as described below\n- `self.run_id` : ID of the run, will be generated if not specified\n- `self.use_existing_predictions` : if True, we check for existing model predicitons in the predicition folder, each prediction is saved in a json file named after the model hash. Model hash is also saved in a hashfile next to the weight file.\n\n`config` is a dictionnary that describes the run configuration, if not in the\ndictionnary, the parameters will take the default values from the Engine and Classifier classes in pyro-engine.\n\n```json\n{\n    \"model_path\" : \"path/to/model.pt\",\n    \"model\" : {\n        \"iou\" : 0,\n        \"conf\" : 0.15,\n        \"imgsz\" : 1024,\n    },\n    \"engine\" : {\n        \"conf_thresh\" : 0.15,\n        \"max_bbox_size\" : 0.4,\n        \"nb_consecutive_frames\" : 8,\n    },\n    \"eval\" : [\"model\", \"engine\"]\n}\n```\n\nWith the following keys:\n\n- __nb_consecutive_frames__ (int): Number of consecutive frames taken into accoun in the Engine\n- __conf_thresh__ (float in [0.,1.]): Confidence threshold used in the Engine, below which detections are filtered out\n- __conf__ (float in [0.,1.]): Confidence threshold used in the Classifier, below which detections are filtered out\n- __max_bbox_size__ (float in [0., 1.]): Bbox size above which detections are filtered out\n- __iou__ (float in [0., 1.]): IoU threshold to compute matches between detected bboxes\n- __eval__ (array of strs): Parts of the evaluation pipeline\n\n### Launcher configuration\n\nThe evaluation can be launched on several configuration at once. `launcher.py`\nis used to configure the runs:\n\n```python\nconfigs = [\n        {\n            \"model_path\" : \"path/to/model_1.pt\",\n            \"engine\": {\n                \"conf_thresh\" : 0.1,\n            },\n        },\n        {\n            \"model_path\" : \"path/to/model_2.onnx\",\n            \"engine\": {\n                \"max_bbox_size\" : 0.12,\n            },\n            \"eval\" : [\"engine\"],\n        },\n        {\n            \"model_path\" : \"path/to/model_3.pt\",\n            \"model\" : {\n                \"iou\" : 0,\n            },\n            \"eval\" : [\"engine\"],\n        },\n    ]\n\n    for config in configs:\n        evaluation = EvaluationPipeline(dataset=dataset, config=config, device=\"mps\")\n        evaluation.run()\n        evaluation.save_metrics()\n```\n\n### Results\n\nMetrics are saved in the `results` folder, in a subdirectory named as the\nrun_ID. The data is stored in a json file with the following content.\n\nThe file contains:\n\n- __model_metrics__ : result of ModelEvaluator\n- __engine_metrics__ : result of EngineEvaluator\n- __config__ : run configuration\n- __dataset__ : dataset information\n\n## Useful definitions\n\n### EvaluationDataset()\n\n`dataset = EvaluationDataset(datapath)`:\n- `dataset.sequences`: list of image Sequence within the dataset. \n- `dataset.hash`: hash of the dataset\n- `dataset.dataframe`: pandas DataFrame describing the dataset\n\n### Sequence()\n\n`Sequence` : object that represents a sequence of images.\n- `sequence.images`: list of CustomImage objects, corresponding to image belonging to a single sequence\n- `sequence.id`: name of the sequence (name of the first image without extension)\n- `sequence.sequence_start`: timestamp of the first image of the sequence\n\n### CustomImage()\n\n`CustomImage`: object describing an image\n- `image.path`: file path\n- `image.sequence_id`: name of the sequence the image belongs to\n- `image.timedelta`: time elapsed between the start of the sequence and this image\n- `image.boxes`: ground truth coordinates\n- `image.prediction` : placeholder to store a prediction\n- `image.timestamp`: capture date of the image\n- `image.hash`: image hash\n- `image.label`: boolean label, True if wildfire present False otherwise\n- `image.name`: image name \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyronear%2Fpyro-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpyronear%2Fpyro-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyronear%2Fpyro-eval/lists"}