{"id":21018711,"url":"https://github.com/tal-tech/chinese-k12-evaluation","last_synced_at":"2025-04-12T10:07:12.471Z","repository":{"id":186632694,"uuid":"675473811","full_name":"tal-tech/chinese-k12-evaluation","owner":"tal-tech","description":null,"archived":false,"fork":false,"pushed_at":"2024-03-21T10:42:34.000Z","size":34531,"stargazers_count":25,"open_issues_count":1,"forks_count":2,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-04-12T10:06:05.936Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tal-tech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-07T02:28:56.000Z","updated_at":"2025-04-09T04:04:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"4df8f18f-ccf6-4a86-905b-2f280ac13320","html_url":"https://github.com/tal-tech/chinese-k12-evaluation","commit_stats":null,"previous_names":["tal-tech/chinese-k12-evaluation"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tal-tech%2Fchinese-k12-evaluation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tal-tech%2Fchinese-k12-evaluation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tal-tech%2Fchinese-k12-evaluation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tal-tech%2Fchinese-k12-evaluation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tal-tech","download_url":"https://codeload.github.com/tal-tech/chinese-k12-evaluation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248550632,"owners_count":21122933,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T10:27:10.757Z","updated_at":"2025-04-12T10:07:12.448Z","avatar_url":"https://github.com/tal-tech.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## CK12: A Rounded K12 Knowledge Graph Based Benchmark for Chinese Holistic Cognition Evaluation\n\n## Overview\n\n我们利用了k12教育领域知识图谱构建了一个LLM的评估基准。\n该评测集包括500+个一级知识点和1900+个二级知识点，基本涵盖了K12教育领域的全面知识。\n本次评估的主要目的是全面评估LLM在中文语境下的高级理解能力和推理能力。\n我们的评估包括九个学科、五个不同的问题类型，共有41k个问题。\n我们使用了四种不同的测试模式来测试当前主流的LLM（zero-shot、few-shot、 zero-shot-cot、 few-shot-cot）。并对数学的cot结果进行了推理步骤的评测。\n\n![example.jpg](https://github.com/tal-tech/chinese-k12-evaluation/blob/main/images/overview.png)\n\n\n## News\n* **[2023.12.09]** CK12 has been accepted to AAAI 2024 🎉🎉🎉\n\n## Leaderboard\n我们的测试集在四种测试模式下，不同模型的表现\nsingle，multi， filling， juding， sorting分别代表单选，多选，填空，判断，排序；single*代表选项被打乱，single**代表增加了10个干扰选项\n![example.jpg](https://github.com/tal-tech/chinese-k12-evaluation/blob/main/images/result.png)\n\n### 数学推理子集步骤准确性评估\n![example.jpg](https://github.com/tal-tech/chinese-k12-evaluation/blob/main/images/math_results.png)\n\n\n## data\nhttps://drive.google.com/drive/folders/1pvvhI_y1o5olcxHGGWFqt3SDv3PRX0rd?usp=drive_link\n\n## run testing\n\n### 环境依赖：\n\ntorch 2.0.1\n\ntransformers 4.33.3\n\nvllm 0.1.3\n\n\n\n### huggingface inference\n\n原始数据是 data/original_data \n\n打乱选项的数据是 data/shuffle_options\n\n增加干扰选项的数据是 data/adding_10_distractor_options\n\n`bash code/release_code_aaai_ck12/inference_hf.sh data/original_data sava_name`\n\n### vllm inference\n\n`bash code/release_code_aaai_ck12/inference_vllm.sh data/original_data sava_name`\n\n### get metric\n\n`python code/release_code_aaai_ck12/metric.py`\n\n\n\n## 例子\n\n文科和理科的题目例子\n### 政治\n![example.jpg](https://github.com/tal-tech/chinese-k12-evaluation/blob/main/images/exp5.png)\n\n### 生物\n![example.jpg](https://github.com/tal-tech/chinese-k12-evaluation/blob/main/images/exp4.png)\n\n## Citation\n\n\nPlease cite our paper if you use our dataset.\n\n```\n@article{you2023ck12,\ntitle={CK12: A Rounded K12 Knowledge Graph Based Benchmark for Chinese Holistic Cognition Evaluation}, \nauthor={Weihao You, Pengcheng Wang, Changlong Li, Zhilong Ji and  Jinfeng Bai}\nyear={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftal-tech%2Fchinese-k12-evaluation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftal-tech%2Fchinese-k12-evaluation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftal-tech%2Fchinese-k12-evaluation/lists"}