https://github.com/flatland-association/flatland-benchmarks
Flatland Benchmarks (FAB) is an open-source web-based platform for running Benchmarks to foster Open Research
https://github.com/flatland-association/flatland-benchmarks
benchmarks flatland open-access open-community open-data open-peer-review open-source
Last synced: about 1 month ago
JSON representation
Flatland Benchmarks (FAB) is an open-source web-based platform for running Benchmarks to foster Open Research
- Host: GitHub
- URL: https://github.com/flatland-association/flatland-benchmarks
- Owner: flatland-association
- License: mit
- Created: 2024-11-18T09:25:26.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2026-04-29T14:04:39.000Z (about 2 months ago)
- Last Synced: 2026-04-29T15:33:40.065Z (about 2 months ago)
- Topics: benchmarks, flatland, open-access, open-community, open-data, open-peer-review, open-source
- Language: Python
- Homepage: https://benchmarks.flatland.cloud
- Size: 10.7 MB
- Stars: 3
- Watchers: 4
- Forks: 3
- Open Issues: 44
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: docs/CONTRIBUTING.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
Flatland Benchmarks
===================
[](https://github.com/flatland-association/flatland-benchmarks/actions/workflows/checks.yml)
Flatland Benchmarks (FAB) is an open-source web-based platform for running Benchmarks to foster Open Research aiming at:
| Goal | Description | How does FAB contribute? |
|----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Open access** and **open data** | research results and gained knowledge are accessible and can be used by anyone for applications or as the basis further research | the web-based platform is open to researchers, developers, data scientists and domain experts, enabling to access and interpret data, problem formulations and results |
| **Open source software** and **open peer review** | allow for the replication and reproducibility of research as well as for the reuse of data and software in other research projects, empowering anyone to contribute to and accelerating the pace of research | participants disclose their code and the leaderboard links to papers, code and discussion |
| **Open community** | enables collaborative formulation of problems and research questions, bringing in the expertise of various domains and fostering interdisciplinary approaches | platform and problem formulations are made open and the community can contribute |
It supports the following use cases:
* competitions: submissions comprise all results of a benchmark, a leaderboard shows the best submissions per benchmark
* long-running benchmarks: idem
* validation campaigns: test results can come from different experiments and are aggregated to overall benchmark results; the campaign features the best results for each benchmark.
Experiments are run in the following loops:
* closed-loop: the evaluation is carried out without human in the loop (always for competition and benchmarks, optionally in campaign setting)
* interactive-loop: the evaluation is carried out by a human Operator interacting with a UI, the results are uploaded automatically or/and manually
* offline-loop: the evaluation is done completely offline, the results are uploaded manually at the end of the experiment
## FAB Scope and Context
More details, see [ARCHITECTURE.md](docs/ARCHITECTURE.md).
## Building Block View
More details, see [ARCHITECTURE.md](docs/ARCHITECTURE.md).
## Data Model
More details, see [ARCHITECTURE.md](docs/ARCHITECTURE.md).
### Nomenclature
| Technical naming | `CAMPAIGN` setup | `COMPETITION` setup | `DEFAULT` setup |
|------------------|----------------------|-----------------------|-------------------|
| `suite` | Validation Campaign | Competition/Challenge | (Benchmark) Suite |
| `benchmark` | Evaluation Objective | Round | Benchmark |
| `test` | KPI | Test | Test |
| `scenario` | Scenario | Scenario | Scenario |
📦 TL; DR;
----------
To participate, or even organize your own benchmarks or competitions, you don't need to install anything, you just need to sign in an instance of the platform (e.g. this one). If you wish to configure your own instance of the platform, here are the instructions:
```shell
cd evaluation
docker compose -f docker-compose.yml --profile full up --wait
```
📑 Documentation
----------------
* [User's Guide](docs/USER_GUIDE.md) User documentation for benchmark participants and benchmark administrators.
* [Contributing](docs/CONTRIBUTING.md) Contributions are welcome! See how.
* [Architecture](docs/ARCHITECTURE.md) Get a technical overview of the platform and the design decisions.
* [Administrator's Guide](docs/ADMINISTRATION.md) Documentation for technical platform admins.
* [Developer's Guide](docs/DEVELOPMENT.md) Documentation for platform developers.
💬 Communication
----------------
* [Issue Tracker](https://github.com/flatland-association/flatland-benchmarks/issues/)