{"id":35263285,"url":"https://github.com/flatland-association/flatland-benchmarks","last_synced_at":"2026-05-02T21:05:51.049Z","repository":{"id":275960853,"uuid":"890265107","full_name":"flatland-association/flatland-benchmarks","owner":"flatland-association","description":"Flatland Benchmarks (FAB) is an open-source web-based platform for running Benchmarks to foster Open Research","archived":false,"fork":false,"pushed_at":"2026-04-29T14:04:39.000Z","size":11178,"stargazers_count":3,"open_issues_count":44,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-04-29T15:33:40.065Z","etag":null,"topics":["benchmarks","flatland","open-access","open-community","open-data","open-peer-review","open-source"],"latest_commit_sha":null,"homepage":"https://benchmarks.flatland.cloud","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flatland-association.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-18T09:25:26.000Z","updated_at":"2026-04-27T14:59:57.000Z","dependencies_parsed_at":"2025-02-19T16:33:50.429Z","dependency_job_id":"fd4303a2-7440-4b3b-9176-59e724974d01","html_url":"https://github.com/flatland-association/flatland-benchmarks","commit_stats":null,"previous_names":["flatland-association/flatland-benchmarks"],"tags_count":45,"template":false,"template_full_name":null,"purl":"pkg:github/flatland-association/flatland-benchmarks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flatland-association%2Fflatland-benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flatland-association%2Fflatland-benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flatland-association%2Fflatland-benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flatland-association%2Fflatland-benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flatland-association","download_url":"https://codeload.github.com/flatland-association/flatland-benchmarks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flatland-association%2Fflatland-benchmarks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32549418,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T19:18:06.202Z","status":"ssl_error","status_checked_at":"2026-05-02T19:16:21.335Z","response_time":132,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarks","flatland","open-access","open-community","open-data","open-peer-review","open-source"],"created_at":"2025-12-30T09:32:39.910Z","updated_at":"2026-05-02T21:05:51.044Z","avatar_url":"https://github.com/flatland-association.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Flatland Benchmarks\n===================\n[![Main](https://github.com/flatland-association/flatland-benchmarks/actions/workflows/checks.yml/badge.svg)](https://github.com/flatland-association/flatland-benchmarks/actions/workflows/checks.yml)\n\nFlatland Benchmarks (FAB) is an open-source web-based platform for running Benchmarks to foster Open Research aiming at:\n\n| Goal                                               | Description                                                                                                                                                                                                  | How does FAB contribute?                                                                                                                                               |\n|----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| **Open access** and **open data**                  | research results and gained knowledge are accessible and can be used by anyone for applications or as the basis further research                                                                             | the web-based platform is open to researchers, developers, data scientists and domain experts, enabling to access and interpret data, problem formulations and results |\n| **Open source software**  and **open peer review** | allow for the replication and reproducibility of research as well as for the reuse of data and software in other research projects, empowering anyone to contribute to and accelerating the pace of research | participants disclose their code and the leaderboard links to papers, code and discussion                                                                              |\n| **Open community**                                 | enables collaborative formulation of problems and research questions, bringing in the expertise of various domains and fostering interdisciplinary approaches                                                | platform and problem formulations are made open and the community can contribute                                                                                       |\n\nIt supports the following use cases:\n\n* competitions: submissions comprise all results of a benchmark, a leaderboard shows the best submissions per benchmark\n* long-running benchmarks: idem\n* validation campaigns: test results can come from different experiments and are aggregated to overall benchmark results; the campaign features the best results for each benchmark.\n\nExperiments are run in the following loops:\n\n* closed-loop: the evaluation is carried out without human in the loop (always for competition and benchmarks, optionally in campaign setting)\n* interactive-loop: the evaluation is carried out by a human Operator interacting with a UI, the results are uploaded automatically or/and manually\n* offline-loop: the evaluation is done completely offline, the results are uploaded manually at the end of the experiment\n\n## FAB Scope and Context\n\n![SystemContext.drawio.png](docs/img/architecture/SystemContext.drawio.png)\n\nMore details, see [ARCHITECTURE.md](docs/ARCHITECTURE.md).\n\n## Building Block View\n\n![Closed_and_Interactive_Loop_Eval.drawio.png](docs/img/architecture/Closed_and_Interactive_Loop_Eval.drawio.png)\n\nMore details, see [ARCHITECTURE.md](docs/ARCHITECTURE.md).\n\n## Data Model\n\n![DataModel.drawio.png](docs/img/architecture/DataModel.drawio.png)\n\nMore details, see [ARCHITECTURE.md](docs/ARCHITECTURE.md).\n\n### Nomenclature\n\n| Technical naming | `CAMPAIGN` setup     | `COMPETITION` setup   | `DEFAULT` setup  |\n|------------------|----------------------|-----------------------|-------------------|\n| `suite`          | Validation Campaign  | Competition/Challenge | (Benchmark) Suite |\n| `benchmark`      | Evaluation Objective | Round                 | Benchmark         |\n| `test`           | KPI                  | Test                  | Test              |\n| `scenario`       | Scenario             | Scenario              | Scenario          |\n\n📦 TL; DR;\n----------\nTo participate, or even organize your own benchmarks or competitions, you don't need to install anything, you just need to sign in an instance of the platform (e.g. this one). If you wish to configure your own instance of the platform, here are the instructions:\n\n```shell\ncd evaluation\ndocker compose  -f docker-compose.yml --profile full up --wait\n```\n\n📑 Documentation\n----------------\n\n* [User's Guide](docs/USER_GUIDE.md) User documentation for benchmark participants and benchmark administrators.\n* [Contributing](docs/CONTRIBUTING.md) Contributions are welcome! See how.\n* [Architecture](docs/ARCHITECTURE.md) Get a technical overview of the platform and the design decisions.\n* [Administrator's Guide](docs/ADMINISTRATION.md) Documentation for technical platform admins.\n* [Developer's Guide](docs/DEVELOPMENT.md) Documentation for platform developers.\n\n💬 Communication\n----------------\n\n* [Issue Tracker](https://github.com/flatland-association/flatland-benchmarks/issues/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflatland-association%2Fflatland-benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflatland-association%2Fflatland-benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflatland-association%2Fflatland-benchmarks/lists"}