{"id":20058639,"url":"https://github.com/roboflow/model-leaderboard","last_synced_at":"2025-04-09T13:06:56.034Z","repository":{"id":255905006,"uuid":"851499721","full_name":"roboflow/model-leaderboard","owner":"roboflow","description":"Which model is the best at object detection? Which is best for small or large objects? We compare the results in a handy leaderboard.","archived":false,"fork":false,"pushed_at":"2025-03-24T17:32:47.000Z","size":416,"stargazers_count":68,"open_issues_count":4,"forks_count":8,"subscribers_count":7,"default_branch":"develop","last_synced_at":"2025-04-02T08:57:08.304Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://leaderboard.roboflow.com/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/roboflow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-03T07:57:26.000Z","updated_at":"2025-03-24T17:32:51.000Z","dependencies_parsed_at":"2024-11-11T18:29:34.778Z","dependency_job_id":"c050a268-8d38-4b4b-852d-236d34319d20","html_url":"https://github.com/roboflow/model-leaderboard","commit_stats":{"total_commits":116,"total_committers":5,"mean_commits":23.2,"dds":0.6379310344827587,"last_synced_commit":"113cfa1ba407291412933a8204786ed4f3a3e23b"},"previous_names":["roboflow/model-leaderboard"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fmodel-leaderboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fmodel-leaderboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fmodel-leaderboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fmodel-leaderboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/roboflow","download_url":"https://codeload.github.com/roboflow/model-leaderboard/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248045231,"owners_count":21038553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T13:02:40.223Z","updated_at":"2025-04-09T13:06:56.007Z","avatar_url":"https://github.com/roboflow.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Model Leaderboard Website\n\nThis project is a **model leaderboard** website for evaluating a variety of models on the COCO dataset. Models are compared compares models across a variety of metrics, computed with [supervision](https://github.com/roboflow/supervision).\n\n## Project Structure\n\n```text\nmodel-leaderboard/\n│\n├── data/                  # Directory for storing datasets (e.g., COCO)\n├── models/                # Model-specific directories\n│   ├── object_detection/   # Houses individual model folders\n│   │   ├── yolov8n/        # Example of a model folder\n│   │   ├── yolov9t/        # Example of another model folder\n│   │   └── yolov10m/       # And so on...\n│   └── utils.py            # Shared utility code (minimal cross-model dependencies)\n│\n├── index.html              # Main page for the static side\n├── static/                 # Static files, serving as the backend for the site\n│   └── ...\n├── download_data.py        # Script for downloading the ground truth dataset\n├── build_static_site.py    # Must be run to aggregate results for the static site\n└── requirements.txt        # Dependencies for data download and leaderboard front end\n```\n\nEach model is expected to have:\n\n```text\nmodel_name/\n│\n├── requirements.txt        # Dependencies for running the model\n├── run.py                  # Script that runs model on dataset in /data, compares with ground truth\n├── results.json            # Model results, created with run.py\n├── (any other scripts)\n└── (optional) README.md    # Links to original model page \u0026 instructions on how to produce results.py\n```\n\nEach model is expected to be run in a separate python virtual environment, with its own set of dependencies.\n\nBefore we automate the models to be run regularly, the naming standards are relaxed.\nThe only requirements is to store results for `results.json` in the model directory. For consistency, we advice to keep the scripts in `run.py`.\n\n### Key Files\n\n1. **`download_data.py`**: Downloads the dataset (currently configured for COCO) and places it into the `data/` directory.\n2. **`build_static_site.py`**: Aggregates the results of the models to be shown on the GitHub Pages site.\n3. **`run_overnight.sh`**: An early version of a script to run the entire process, generating model results and comparing to downloaded data. We hacked it together for the first iteration of the leaderboard. Requires `uv`.\n4. **`gradio_app.py`**: The initial version of the leaderboard UI. Displays model results in a gradio page.\n\n5. **Model-Specific Folders**\"\n\n   - Each object detection model is housed in its own folder under `models/object_detection/`. These folders include `run.py`, which generates evaluation results in the form of a `results.json` file.\n\n6. **`utils.py`**: Contains shared utility code across models.\n\n## Getting Started\n\n### 1. Download the Dataset\n\nRun the `download_data.py` script to download the COCO dataset (or any supported dataset in the future):\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\npython download_data.py\ndeactivate\n```\n\n### 2. Run the model\n\nEach model folder contains its own script to run predictions and generate results.\n\n```bash\ncd models/object_detection/yolov8n\npython -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\npython run.py  # Or any other scripts\ndeactivate\n```\n\n### 3. Run All Models\n\nCurrently, you may use `run_overnight.sh` to run all models.\nThis is an early version of the script and is due to change.\n\n```bash\n./run_overnight.sh\n```\n\n### 4. Launch the Leaderboard\n\nOnce the results are generated, you can launch the Gradio app to visualize the leaderboard:\n\n```bash\nsource .venv/bin/activate\npython gradio_app.py\n```\n\n## Notes\n\n- The leaderboard is currently configured to work with the COCO dataset, running the benchmark on the test set.\n- Model dependencies are handled separately in each model folder, with each model having its own `requirements.txt` to avoid dependency conflicts.\n\n## Contributing\n\nFeel free to contribute by adding new models, improving the existing evaluation process, or expanding dataset support. To add a new model, simply create a new folder under `models/object_detection/` and follow the structure of existing models.\n\nMake sure that the model's `run.py` script generates a `results.json` file in the model directory. If there are more scripts, please provide a `README.md` in the model folder how to run the model. Optionally, you may modify the `run_overnight.sh` script to help us automate the benchmarking in the future.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froboflow%2Fmodel-leaderboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Froboflow%2Fmodel-leaderboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froboflow%2Fmodel-leaderboard/lists"}