{"id":13526165,"url":"https://github.com/haonan-li/CMMLU","last_synced_at":"2025-04-01T06:31:22.392Z","repository":{"id":175850229,"uuid":"654578300","full_name":"haonan-li/CMMLU","owner":"haonan-li","description":"CMMLU: Measuring massive multitask language understanding in Chinese","archived":false,"fork":false,"pushed_at":"2024-12-06T08:54:24.000Z","size":4314,"stargazers_count":750,"open_issues_count":0,"forks_count":60,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-27T19:12:12.956Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/haonan-li.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-16T12:42:42.000Z","updated_at":"2025-03-27T09:14:32.000Z","dependencies_parsed_at":"2023-12-19T06:38:20.877Z","dependency_job_id":"8cec1cce-cb2e-4808-bdac-6061bfb8707e","html_url":"https://github.com/haonan-li/CMMLU","commit_stats":null,"previous_names":["haonan-li/cmmlu"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haonan-li%2FCMMLU","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haonan-li%2FCMMLU/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haonan-li%2FCMMLU/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haonan-li%2FCMMLU/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/haonan-li","download_url":"https://codeload.github.com/haonan-li/CMMLU/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246596897,"owners_count":20802919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T06:01:26.003Z","updated_at":"2025-04-01T06:31:22.385Z","avatar_url":"https://github.com/haonan-li.png","language":"Python","funding_links":[],"categories":["Datasets-or-Benchmark","A01_文本生成_文本对话","📏 评测基准","Benchmark"],"sub_categories":["通用","大语言对话模型及数据","🧩 领域模型","Chinese"],"readme":"# CMMLU---中文多任务语言理解评估\n[![evaluation](https://img.shields.io/badge/OpenCompass-Support-royalblue.svg\n)](https://github.com/internLM/OpenCompass/) [![evaluation](https://img.shields.io/badge/lm--evaluation--harness-Support-blue\n)](https://github.com/EleutherAI/lm-evaluation-harness)\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"fig/banner_zh.jpg\" style=\"width: 100%;\" id=\"title-icon\"\u003e       \u003c/p\u003e\n\n\u003ch4 align=\"center\"\u003e\n    \u003cp\u003e\n        \u003cb\u003e简体中文\u003c/b\u003e |\n        \u003ca href=\"https://github.com/haonan-li/CMMLU/blob/master/README_EN.md\"\u003eEnglish\u003c/a\u003e \n    \u003cp\u003e\n\u003c/h4\u003e\n\n\u003cp align=\"center\" style=\"display: flex; flex-direction: row; justify-content: center; align-items: center\"\u003e\n📄 \u003ca href=\"https://arxiv.org/abs/2306.09212\" target=\"_blank\" style=\"margin-right: 15px; margin-left: 10px\"\u003e论文\u003c/a\u003e • \n🏆 \u003ca href=\"https://github.com/haonan-li/CMMLU/#排行榜\" target=\"_blank\"  style=\"margin-left: 10px\"\u003e排行榜\u003c/a\u003e •\n🤗 \u003ca href=\"https://huggingface.co/datasets/haonan-li/cmmlu\" target=\"_blank\" style=\"margin-left: 10px\"\u003e数据集\u003c/a\u003e \n\u003c/p\u003e\n\n\n## 简介\n\nCMMLU是一个综合性的中文评估基准，专门用于评估语言模型在中文语境下的知识和推理能力。CMMLU涵盖了从基础学科到高级专业水平的67个主题。它包括：需要计算和推理的自然科学，需要知识的人文科学和社会科学,以及需要生活常识的中国驾驶规则等。此外，CMMLU中的许多任务具有中国特定的答案，可能在其他地区或语言中并不普遍适用。因此是一个完全中国化的中文测试基准。\n\n注：如果有古汉语的评估需求，欢迎使用[ACLUE](https://github.com/isen-zhang/ACLUE).\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"fig/logo.jpg\" style=\"width: 85%;\" id=\"title-icon\"\u003e       \u003c/p\u003e\n\n## 排行榜\n\n\u003e **Note：**\n\u003e 自2023-12-16日起，对于未开放公测的API模型，我们将验证 1.模型是否有基本的指令跟随能力；2.是否存在数据污染，通过验证的模型会被更新在榜单。\n\n以下表格显示了模型在 five-shot 和 zero-shot 测试下的表现。\n\n\u003cdetails\u003e\n\u003csummary\u003eFive-shot\u003c/summary\u003e\n\n| 模型                 | STEM  | 人文学科 | 社会科学 | 其他  | 中国特定主题 | 平均分  |\n|---------------------|------|------------|----------------|-------|----------------|---------|\n| 开放测试的模型 |\n| [Lingzhi-72B-chat](https://huggingface.co/Lingzhi-AI/Lingzhi-72B-chat)       | **84.82** | 92.93 | **91.25** | 92.64 | **90.89** | **90.26** |\n| [Telechat2-35B](https://modelscope.cn/models/TeleAI/TeleChat2-35B-Nov) | 83.59 | 93.69 | 90.99 | **93.34** | 87.89 | 90.16 |\n| [Spark 4.0-2024-10-14](https://xinghuo.xfyun.cn/sparkapi)       | 84.75 | **93.53** | 90.64 | 91.03 | 90.09 | 90.07 |\n| [Qwen2-72B](https://huggingface.co/Qwen/Qwen2-72B-Instruct)       |   82.80   |   93.84   |   90.38   |   92.71   |   90.60   |   89.65   |\n| [Jiutian-大模型](https://jiutian.10086.cn/qdlake/qdh-web/#/model/detail/1241)  |   80.58   |   93.33   |   89.81   |   91.79   |   89.8   |   88.59   |\n| [Qwen1.5-110B](https://modelscope.cn/models/qwen/Qwen1.5-110B)    |   81.59   |   92.41   |   89.14   |   91.19   |   89.02   |   88.32   |\n| [JIUTIAN-57B](https://jiutian.10086.cn/portal/common-helpcenter#/document/320?platformCode=LLM_STUDIO)  |   79.79   |   91.99   |   88.57   |   90.27   |   88.02   |   87.39   |\n| [Qwen2.5-72B](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct)   |   80.35   |   88.41   |   85.96   |   86.06   |   88.91   |   85.67   |\n| Youyuanjian  |   76.34 |   88.38   |   84.74   |   86.57   |   83.89   |   83.72   |\n| [Qwen1.5-72B](https://modelscope.cn/models/qwen/Qwen1.5-72B)      |   76.83   |   88.37   |   84.15   |   86.06   |   83.77   |   83.54   |\n| [PCI-TransGPT](http://123.249.36.167/call-frontend/#/transGpt)    |   76.85   |   86.46   |   81.65   |   84.57   |   82.85   |   82.46   |\n| [Qwen1.5-32B](https://modelscope.cn/models/qwen/Qwen1.5-32B)      |   76.25   |   86.31   |   83.42   |   83.82   |   82.84   |   82.25   |\n| [ZhiLu-2-8B](https://huggingface.co/SYSU-MUCFC-FinTech-Research-Center/ZhiLu-2-8B-Instruct)   |   74.32  |   83.33   |   81.06   |   83.78   |   78.58   |   79.95   |\n| [BlueLM-7B](https://github.com/vivo-ai-lab/BlueLM)                |   61.36   |   79.83   |   77.80   |   78.89   |   76.74   |   74.27   |\n| [Qwen1.5-7B](https://github.com/QwenLM/Qwen1.5)                   |   63.64   |   76.42   |   74.69   |   75.91   |   73.43   |   72.50   |\n| [XuanYuan-70B](https://huggingface.co/Duxiaoman-DI/XuanYuan-70B)  |   60.74   |   77.79   |   75.47   |   70.81   |   70.92   |   71.10   |\n| [GPT4](https://openai.com/gpt4)                                   |   65.23   |   72.11   |   72.06   |   74.79   |   66.12   |   70.95   |\n| [Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct)  |   55.05   |   66.62   |   66.08   |   70.50   |   61.65   |   64.38   |\n| [XuanYuan-13B](https://github.com/Duxiaoman-DI/XuanYuan)          |   50.07   |\t66.32\t|   64.11   |   59.99\t|   60.55\t|   60.05   |\n| [Qwen-7B](https://github.com/QwenLM/Qwen-7B)                      |   48.39   |   63.77   |   61.22   |   62.14   |   58.73   |   58.66   |\n| [ZhiLu-13B](https://github.com/SYSU-MUCFC-FinTech-Research-Center/ZhiLu)   |   44.26   |   61.54   |   60.25   |   61.14   |   57.14   |   57.16   |\n| [ChatGPT](https://openai.com/chatgpt)                             |   47.81   |   55.68   |   56.50   |   62.66   |   50.69   |   55.51   |\n| [Baichuan-13B](https://github.com/baichuan-inc/Baichuan-13B)      |   42.38   |   61.61   |   60.44   |   59.26   |   56.62   |   55.82   |\n| [ChatGLM2-6B](https://huggingface.co/THUDM/chatglm2-6b)           |   42.55   |   50.98   |   50.99   |   50.80   |   48.37   |   48.80   |\n| [Baichuan-7B](https://github.com/baichuan-inc/baichuan-7B)        |   35.25   |   48.07   |   47.88   |   46.61   |   44.14   |   44.43   |\n| [Falcon-40B](https://huggingface.co/tiiuae/falcon-40b)            |   33.33   |   43.46   |   44.28   |   44.75   |   39.46   |   41.45   |\n| [LLaMA-65B](https://github.com/facebookresearch/llama)            |   34.47   |   40.24   |   41.55   |   42.88   |   37.00   |   39.80   |\n| [ChatGLM-6B](https://github.com/THUDM/GLM-130B)                   |   32.35   |   39.22   |   39.65   |   38.62   |   37.70   |   37.48   |\n| [BatGPT-15B](https://arxiv.org/abs/2307.00360)                    |   34.96   |   35.45   |   36.31   |   42.14   |   37.89   |   37.16   |\n| [BLOOMZ-7B](https://github.com/bigscience-workshop/xmtf)          |   30.56   |   39.10   |   38.59   |   40.32   |   37.15   |   37.04   |\n| [Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) |   30.10   |   39.38   |   32.93   |   48.05   |   37.17   |   36.85   |\n| [Chinese-LLaMA-13B](https://github.com/ymcui/Chinese-LLaMA-Alpaca)|   27.12   |   33.18   |   34.87   |   35.10   |   32.97   |   32.63   |\n| [Bactrian-LLaMA-13B](https://github.com/mbzuai-nlp/bactrian-x)    |   27.52   |   32.47   |   32.27   |   35.77   |   31.56   |   31.88   |\n| [MOSS-SFT-16B](https://github.com/OpenLMLab/MOSS)                 |   27.23   |   30.41   |   28.84   |   32.56   |   28.68   |   29.57   |\n| 未开放测试的模型 |\n| [BlueLM]()                                                        | **78.16** | **90.50** | **86.88** | **87.87** | **87.55** | **85.59** |\n| [Mind GPT]()                                                      |   76.76   |   87.09   |   83.74   |   84.70   |   81.82   |   82.84   |\n| [ZW-LM]()                                                         |   72.68   |   85.84   |   83.61   |   85.68   |   82.71   |   81.73   |\n| [QuarkLLM](https://www.quark.cn/)                                 |   70.97   |   85.20   |   82.88   |   82.71   |   81.12   |   80.27   |\n| [Galaxy](https://www.zuoyebang.com/)                              |   69.61   |   74.95   |   78.54   |   77.93   |   73.99   |   74.03   |\n| [KwaiYii-66B](https://github.com/kwai/KwaiYii)                    |   56.70   |   79.43   |   72.84   |   74.43   |   71.11   |   71.12   |\n| [FanFan-1.5B]()                                                   |   59.84   |   70.86   |   70.72   |   72.19   |   69.73   |   66.50   |\n| [Mengzi-7B](https://www.langboat.com/)                            |   49.59   |   75.27   |   71.36   |   70.52   |   69.23   |   66.41   |\n| [KwaiYii-13B](https://github.com/kwai/KwaiYii)                    |   46.54   |   69.22   |   64.49   |   65.09   |   63.10   |   61.73   |\n| [MiLM-6B](https://github.com/XiaoMi/MiLM-6B/)                     |   46.85   |   61.12   |   61.68   |   58.84   |   59.39   |   57.17   |\n| [MiLM-1.3B](https://github.com/XiaoMi/MiLM-6B/)                   |   35.59   |   49.58   |   49.03   |   47.56   |   48.17   |   45.39   |\n| Random                                                            |   25.00   |   25.00   |   25.00   |   25.00   |   25.00   |   25.00   |\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eZero-shot\u003c/summary\u003e\n\n| 模型                 | STEM  | 人文学科 | 社会科学 | 其他  | 中国特定主题 | 平均分  |\n|---------------------|------|------------|----------------|-------|----------------|---------|\n| 开放测试的模型 |\n| [Spark 4.0-2024-10-14](https://xinghuo.xfyun.cn/sparkapi)       | **87.36** | **93.97** | 90.03 | 92.71 | 90.4 | **90.97** |\n| [Telechat2-35B](https://modelscope.cn/models/TeleAI/TeleChat2-35B-Nov) | 84.01 | 93.16 | **91.96** | **93.34** | 88.39 | 90.49 |\n| [Lingzhi-72B-chat](https://huggingface.co/Lingzhi-AI/Lingzhi-72B-chat) | 84.85 | 92.99 | 90.75 | 92.47 | **90.68** | 90.07 |\n| [Qwen1.5-110B](https://modelscope.cn/models/qwen/Qwen1.5-110B)    |   80.84   |   91.51   |   89.01   |   89.99   |   88.64   |   87.64   |\n| [Qwen2-72B](https://huggingface.co/Qwen/Qwen2-72B-Instruct)       |   80.92   |   90.90   |   87.93   |   91.23   |   87.24   |   87.47   |\n| [Qwen2.5-72B](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct)   |   80.67   |   87.00   |   84.66  |    87.35   |   83.21  |    84.70   |\n| [PCI-TransGPT](http://123.249.36.167/call-frontend/#/transGpt)    |   76.69   |   86.26   |   81.71   |   84.47   |   83.13   |   82.44   |\n| [Qwen1.5-72B](https://modelscope.cn/models/qwen/Qwen1.5-72B)      |   75.07   |   86.15   |   83.06   |   83.84   |   82.78   |   81.81   |\n| [Qwen1.5-32B](https://modelscope.cn/models/qwen/Qwen1.5-32B)      |   74.82   |   85.13   |   82.49   |   84.34   |   82.47   |   81.47   |\n| Youyuanjian     |   73.34   |   85.43   |   82.37  |   84.67   |   81.21   |   81.19   |\n| [ZhiLu-2-8B](https://huggingface.co/SYSU-MUCFC-FinTech-Research-Center/ZhiLu-2-8B-Instruct)   |   74.32  |   83.33   |   81.06   |   83.78   |   78.58   |   79.95   |\n| [BlueLM-7B](https://github.com/vivo-ai-lab/BlueLM)                |   62.08   |   81.29   |   79.38   |   79.56   |   77.69   |   75.40   |\n| [Qwen1.5-7B](https://github.com/QwenLM/Qwen1.5)                   |   62.87   |   74.90   |   72.65   |   74.64   |   71.94   |   71.05   |\n| [XuanYuan-70B](https://huggingface.co/Duxiaoman-DI/XuanYuan-70B)  |   61.21   |   76.25   |   74.44   |   70.67   |   69.35   |   70.59   |\n| [Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct)  |   61.60   |   71.44   |   69.42   |   74.72   |   63.79   |   69.01   |\n| [GPT4](https://openai.com/gpt4)                                   |   63.16   |   69.19   |   70.26   |   73.16   |   63.47   |   68.90   |\n| [Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) |   57.02   |   67.87   |   68.67   |   73.95   |   62.96   |   66.74   |\n| [XuanYuan-13B](https://github.com/Duxiaoman-DI/XuanYuan)          |   50.22   |\t67.55\t|   63.85\t|   61.17   |   61.50\t|   60.51   |\n| [Qwen-7B](https://github.com/QwenLM/Qwen-7B)                      |   46.33   |   62.54   |   60.48   |   61.72   |   58.77   |   57.57   |\n| [ZhiLu-13B](https://github.com/SYSU-MUCFC-FinTech-Research-Center/ZhiLu)  |   43.53   |   61.60   |   61.40   |   60.15   |   58.97   |   57.14   |\n| [ChatGPT](https://openai.com/chatgpt)                             |   44.80   |   53.61   |   54.22   |   59.95   |   49.74   |   53.22   |\n| [Baichuan-13B](https://github.com/baichuan-inc/Baichuan-13B)      |   42.04   |   60.49   |   59.55   |   56.60   |   55.72   |   54.63   |\n| [ChatGLM2-6B](https://huggingface.co/THUDM/chatglm2-6b)           |   41.28   |   52.85   |   53.37   |   52.24   |   50.58   |   49.95   |\n| [BLOOMZ-7B](https://github.com/bigscience-workshop/xmtf)          |   33.03   |   45.74   |   45.74   |   46.25   |   41.58   |   42.80   |\n| [Baichuan-7B](https://github.com/baichuan-inc/baichuan-7B)        |   32.79   |   44.43   |   46.78   |   44.79   |   43.11   |   42.33   |\n| [ChatGLM-6B](https://github.com/THUDM/GLM-130B)                   |   32.22   |   42.91   |   44.81   |   42.60   |   41.93   |   40.79   |\n| [BatGPT-15B](https://arxiv.org/abs/2307.00360)                    |   33.72   |   36.53   |   38.07   |   46.94   |   38.32   |   38.51   |\n| [Falcon-40B](https://huggingface.co/tiiuae/falcon-40b)            |   31.11   |   41.30   |   40.87   |   40.61   |   36.05   |   38.50   |\n| [LLaMA-65B](https://github.com/facebookresearch/llama)            |   31.09   |   34.45   |   36.05   |   37.94   |   32.89   |   34.88   |\n| [Bactrian-LLaMA-13B](https://github.com/mbzuai-nlp/bactrian-x)    |   26.46   |   29.36   |   31.81   |   31.55   |   29.17   |   30.06   |\n| [Chinese-LLaMA-13B](https://github.com/ymcui/Chinese-LLaMA-Alpaca)|   26.76   |   26.57   |   27.42   |   28.33   |   26.73   |   27.34   |\n| [MOSS-SFT-16B](https://github.com/OpenLMLab/MOSS)                 |   25.68   |   26.35   |   27.21   |   27.92   |   26.70   |   26.88   |\n| 未开放测试的模型 |\n| [BlueLM]()                                                        | **76.36** | **90.34** | **86.23** | **86.94** | **86.84** | **84.68** |\n| [DiMind]()                                                        | **70.92** | **86.66** | **86.04** | **86.60** | **81.49** | **82.73** |\n| [云天天书]()                                                       |   73.03   |   83.78   |   82.30   |   84.04   |   81.37   |   80.62   |\n| [Mind GPT]()                                                      |   71.20   |   83.95   |   80.59   |   82.11   |   78.90   |   79.20   |\n| [QuarkLLM](https://www.quark.cn/)                                 |   67.23   |   81.69   |   79.47   |   80.74   |   77.00   |   77.08   |\n| [Galaxy](https://www.zuoyebang.com/)                              |   69.38   |   75.33   |   78.27   |   78.19   |   73.25   |   73.85   |\n| [ZW-LM]()                                                         |   63.93   |   77.95   |   76.28   |   72.99   |   72.94   |   72.74   |\n| [KwaiYii-66B](https://github.com/kwai/KwaiYii)                    |   55.20   |   77.10   |   71.74   |   73.30   |   71.27   |   69.96   |\n| [Mengzi-7B](https://www.langboat.com/)                            |   49.49   |   75.84   |   72.32   |   70.87   |   70.00   |   66.88   |\n| [KwaiYii-13B](https://github.com/kwai/KwaiYii)                    |   46.82   |   69.35   |   63.42   |   64.02   |   63.26   |   61.22   |\n| [FanFan-1.5B]()                                                   |   54.02   |   64.53   |   63.22   |   67.09   |   62.57   |   61.03   |\n| [MiLM-6B](https://github.com/XiaoMi/MiLM-6B/)                     |   48.88   |   63.49   |   66.20   |   62.14   |   62.07   |   60.37   |\n| [MiLM-1.3B](https://github.com/XiaoMi/MiLM-6B/)                   |   40.51   |   54.82   |   54.15   |   53.99   |   52.26   |   50.79   |\n| Random                                                            |   25.00   |   25.00   |   25.00   |   25.00   |   25.00   |   25.00   |\n\u003c/details\u003e\n\n\n## 如何提交测试结果\n\n* 对于开源或开放API的模型，可直接提交拉取请求（可以选择同时在`src`目录下更新测试代码）。\n* 如模型未开放公测，请发送测试代码到haonan.li@librai.tech，同时将测试结果更新在榜单，并提交拉取请求。我们会在验证结果的真实性之后更新榜单。\n\n## 数据\n我们根据每个主题在[data](data)目录中提供了开发和测试数据集。您也可以通过[Hugging Face](https://huggingface.co/datasets/haonan-li/cmmlu)获取我们的数据。\n\n#### 快速使用\n\n我们的数据集已经添加到 [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) 和 [OpenCompass](https://github.com/InternLM/opencompass) 中，您可以通过这些开源平台快速测试。\n\n#### 数据格式\n数据集中的每个问题都是一个多项选择题，有4个选项，只有一个选项是正确答案。数据以逗号分隔的.csv文件形式存在。示例：\n\n```\n    同一物种的两类细胞各产生一种分泌蛋白，组成这两种蛋白质的各种氨基酸含量相同，但排列顺序不同。其原因是参与这两种蛋白质合成的,tRNA种类不同,同一密码子所决定的氨基酸不同,mRNA碱基序列不同,核糖体成分不同,C\n```\n\n#### 提示\n我们在`src/mp_utils`目录中提供了预处理代码。其中包括我们用于生成直接回答提示和思路链 (COT) 提示的方法。\n\n以下是添加直接回答提示后的数据示例：\n\n```\n    以下是关于(高中生物)的单项选择题，请直接给出正确答案的选项。\n    题目：同一物种的两类细胞各产生一种分泌蛋白，组成这两种蛋白质的各种氨基酸含量相同，但排列顺序不同。其原因是参与这两种蛋白质合成的：\n    A. tRNA种类不同\n    B. 同一密码子所决定的氨基酸不同\n    C. mRNA碱基序列不同\n    D. 核糖体成分不同\n    答案是：C\n\n    ... [其他例子] \n\n    题目：某种植物病毒V是通过稻飞虱吸食水稻汁液在水稻间传播的。稻田中青蛙数量的增加可减少该病毒在水稻间的传播。下列叙述正确的是：\n   \n    A. 青蛙与稻飞虱是捕食关系\n    B. 水稻和病毒V是互利共生关系\n    C. 病毒V与青蛙是寄生关系\n    D. 水稻与青蛙是竞争关系\n    答案是： \n```\n\n对于思路链提示，我们将提示从“请直接给出正确答案的选项”修改为“逐步分析并选出正确答案”。\n\n#### 评估\n我们使用的每个模型的评估代码位于[src](src)中，运行它们的代码列在[script](script)目录中。\n\n\n## 引用\n\n```\n@misc{li2023cmmlu,\n      title={CMMLU: Measuring massive multitask language understanding in Chinese}, \n      author={Haonan Li and Yixuan Zhang and Fajri Koto and Yifei Yang and Hai Zhao and Yeyun Gong and Nan Duan and Timothy Baldwin},\n      year={2023},\n      eprint={2306.09212},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n```\n## 许可证\n\nCMMLU数据集采用\n[Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-nc-sa/4.0/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaonan-li%2FCMMLU","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhaonan-li%2FCMMLU","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaonan-li%2FCMMLU/lists"}