{"id":19053571,"url":"https://github.com/open-compass/compassbench","last_synced_at":"2026-03-02T04:34:33.826Z","repository":{"id":249280707,"uuid":"831007078","full_name":"open-compass/CompassBench","owner":"open-compass","description":"Demo data of CompassBench","archived":false,"fork":false,"pushed_at":"2024-08-07T12:39:22.000Z","size":623,"stargazers_count":11,"open_issues_count":3,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-12T04:04:39.262Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/open-compass.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-19T12:55:02.000Z","updated_at":"2025-08-09T17:19:32.000Z","dependencies_parsed_at":"2024-11-08T23:32:32.407Z","dependency_job_id":"5c5504cb-b974-4bdf-b684-314ad3df8aa7","html_url":"https://github.com/open-compass/CompassBench","commit_stats":null,"previous_names":["open-compass/compassbench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/open-compass/CompassBench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FCompassBench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FCompassBench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FCompassBench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FCompassBench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/open-compass","download_url":"https://codeload.github.com/open-compass/CompassBench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FCompassBench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29992327,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-02T01:47:34.672Z","status":"online","status_checked_at":"2026-03-02T02:00:07.342Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T23:32:16.915Z","updated_at":"2026-03-02T04:34:33.804Z","avatar_url":"https://github.com/open-compass.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# CompassBench\n\nCompassBench is the self-built benchmark for the [CompassRank LLM Leaderboard](https://rank.opencompass.org.cn/leaderboard-llm/), we will provide the example data for the benchmark.\n\n## 202407 V1.3\n\nPlease check [CompassBench](https://opencompass.readthedocs.io/zh-cn/latest/advanced_guides/compassbench_intro.html) for more information of the benchmark.\n\n\n```\nv1_3_data\n    ├── code\n    │   ├── compass_bench_coding_cn_val.json\n    │   └── compass_bench_coding_en_val.json\n    ├── instruct\n    │   ├── compass_bench_instruct_cn_val.json\n    │   └── compass_bench_instruct_en_val.json\n    ├── knowledge\n    │   └── single_choice_cn.jsonl\n    ├── language\n    │   ├── compass_bench_language_cn_val.json\n    │   └── compass_bench_language_en_val.json\n    ├── math\n    │   └── single_choice_cn.jsonl\n    └── reasoning\n        ├── compass_bench_reasoning_cn_val.json\n        └── compass_bench_reasoning_en_val.json\n```\n\n- For subjective evaluation, please refer to [CompassBench Subjective Config](https://github.com/open-compass/opencompass/blob/main/configs/eval_compassbench_v1_3_subjective.py)\n- For objective evaluation, please refer to [CompassBench Objective Config](https://github.com/open-compass/opencompass/blob/main/configs/datasets/compassbench_v1_3/compassbench_v1_3_objective_gen_068af0.py)\n\nPerformance of the example data will be updated soon.\n\n### Evaluation \n\n- Please link the `v1_3_data` to `data/compassbench_v1_3` within the opencompass directory\n\n\n```bash\nexport HUGGINGFACE_HUB_CACHE=/path-to-hf_hub/\nexport HF_HUB_CACHE=/path-to-hf_hub/\nexport HF_EVALUATE_OFFLINE=1\nexport HF_DATASETS_OFFLINE=1\nexport TRANSFORMERS_OFFLINE=1\n# Objective Evaluation\n# We use `perf_4` as the final metric\npython run.py --models hf_internlm2_chat_1_8b --datasets compassbench_v1_3_objective_gen\n\n# Subjective Evaluation\npython run.py configs/eval_compassbench_v1_3_subjective.py\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-compass%2Fcompassbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopen-compass%2Fcompassbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-compass%2Fcompassbench/lists"}