{"id":13754093,"url":"https://github.com/llmeval/llmeval-1","last_synced_at":"2025-05-09T22:30:56.099Z","repository":{"id":179084379,"uuid":"653386689","full_name":"llmeval/llmeval-1","owner":"llmeval","description":"中文大语言模型评测第一期 ","archived":false,"fork":false,"pushed_at":"2023-10-23T07:56:38.000Z","size":7286,"stargazers_count":107,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-16T06:31:36.502Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/llmeval.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-14T00:49:15.000Z","updated_at":"2024-10-21T08:54:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"c7a2f4fc-4d30-41b5-b393-8e3962e033e6","html_url":"https://github.com/llmeval/llmeval-1","commit_stats":null,"previous_names":["llmeval/llmeval-1"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llmeval%2Fllmeval-1","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llmeval%2Fllmeval-1/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llmeval%2Fllmeval-1/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llmeval%2Fllmeval-1/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/llmeval","download_url":"https://codeload.github.com/llmeval/llmeval-1/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253335353,"owners_count":21892656,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T09:01:39.986Z","updated_at":"2025-05-09T22:30:53.726Z","avatar_url":"https://github.com/llmeval.png","language":null,"funding_links":[],"categories":["A01_文本生成_文本对话"],"sub_categories":["大语言对话模型及数据"],"readme":"# LLMEVAL-1 中文大语言模型评测第一期\n\n\u003cdiv align=center\u003e\u003cimg src=\"llmeval-logo.png\"/\u003e\u003c/div\u003e\n\n## 项目简介\n\n自2022年以来，大量不同类型的大模型评测方法相继涌现。然而，为了方便进行自动化评测，目前的评测方法主要采用选择题或者依赖GPT-4进行评估。虽然选择题能够方便进行自动化处理，但其无法有效评估大模型最为关键的生成能力，仅在一定程度上反映模型的知识覆盖范围。尽管GPT-4的自动评测模型可以对文本生成能力进行评估，但我们仍缺乏大规模数据对比分析，无法确定其结果与人工评测之间的实际差距。\nLMEVAL系列评测旨在系统研究大模型评价方法，并试图回答以下几个关键问题：\n\n问题一：应该从哪些方面评测大模型？\n\n问题二：应该用什么方法评测大模型？\n\n问题三：应该使用什么方法进行排序？\n\n为此，本次评测涵盖了17个大类、453个问题，包括事实性问答、阅读理解、框架生成、段落重写、摘要、数学解题、推理、诗歌生成、编程等各个领域。\n在这些问题上，为大模型的回答设置了5个评分项，分别是：正确性、流畅性、信息量、逻辑性和无害性。\n在构造了评测目标的基础上，有多种方法可以对模型进行评测。包括分项评测、众包对比评测、公众对比评测、GPT 4自动分项评测、GPT 4 对比评测等方式。\n\n## 排行榜\n| 模型名称            | 分项测评       |        |             |                   |              |              |                | 对比测评      |             |\n|:-------------------|:-------------|:-------|:-------------|:------------------|:-------------|:-------------|:---------------|:-------------|:------------|\n|                    | 综合          | 排名   | 准确率     | 信息量   | 流畅性      | 逻辑性   | 无害性   | 对比打分     | 排名   |\n| GPT4.0             | 2.833(2.917) | 1(1)   | 2.709(2.803) | 2.817(2.882)      | 2.870(3.000) | 2.832(2.901) | 2.937(3.000)   | 0.701(0.894) | 1(1)        |\n| GPT3.5             | 2.789(2.878) | 2(2)   | 2.616(2.717) | 2.742(2.807)      | 2.850(3.000) | 2.785(2.868) | 2.954(2.998)   | 0.643(0.818) | 2(2)        |\n| Xunfei-Xinhuo      | 2.639(2.724) | 3(4)   | 2.391(2.427) | 2.523(2.564)      | 2.745(2.987) | 2.633(2.646) | 2.904(2.996)   | 0.550(0.623) | 5(3)        |\n| Chat-Baichuan-7B   | 2.633(2.821) | 4(3)   | 2.401(2.651) | 2.453(2.709)      | 2.720(2.998) | 2.627(2.753) | 2.964(2.993)   | 0.603(0.621) | 3(4)        |\n| ChatGLM-6B         | 2.597(2.644) | 5(7)   | 2.323(2.312) | 2.504(2.442)      | 2.703(2.956) | 2.555(2.518) | 2.899(2.989)   | 0.579(0.547) | 4(5)        |\n| Chinese-LLAMA-7B   | 2.571(2.723) | 6(5)   | 2.293(2.431) | 2.394(2.549)      | 2.696(2.993) | 2.553(2.646) | 2.919(2.998)   | 0.506(0.457) | 6(7)        |\n| Ali-Tongyiqianwen  | 2.523(2.646) | 7(6)   | 2.203(2.309) | 2.339(2.403)      | 2.670(2.983) | 2.530(2.542) | 2.875(2.991)   | 0.491(0.507) | 7(6)        |\n| NewBing            | 2.464(2.622) | 8(8)   | 2.127(2.263) | 2.144(2.320)      | 2.607(2.996) | 2.550(2.531) | 2.892(2.998)   | 0.415(0.425) | 9(8)        |\n| MOSS-16B           | 2.337(2.518) | 9(10)  | 1.994(2.109) | 2.054(2.173)      | 2.498(2.974) | 2.288(2.370) | 2.849(2.965)   | 0.377(0.272) | 11(10)      |\n| Linly-ChatFlow-13B | 2.312(2.534) | 11(9)  | 1.966(2.158) | 2.067(2.257)      | 2.408(2.928) | 2.288(2.351) | 2.830(2.976)   | 0.398(0.339) | 10(9)       |\n| ChatYuan-Large     | 2.312(2.411) | 11(12) | 1.933(1.961) | 2.095(2.056)      | 2.458(2.851) | 2.247(2.195) | 2.826(2.994)   | 0.426(0.245) | 8(12)       |\n| MOSS-w-Plugin-16B  | 2.310(2.506) | 12(11) | 1.966(2.101) | 1.964(2.150)      | 2.499(2.963) | 2.285(2.352) | 2.834(2.967)   | 0.352(0.254) | 12(11)      |\n\n*注：括号内数值为GPT-4自动测评的打分及排名\n\n## NEWS\n### 2023.06.19 - 测试结果发布\n\n在过去的一个月中，共有2186位用户参与评测，提交了总计24.3万个评测结果。此外，我们还利用GPT 4 API进行了5.75万次自动评测。本次评测涵盖了17个大类、453个问题，包括事实性问答、阅读理解、框架生成、段落重写、摘要、数学解题、推理、诗歌生成、编程等各个领域。\nhttps://mp.weixin.qq.com/s/FeAH_30IkXHNfywKXoog1w\n\n### 2023.05.24 - 中期进展公布\n\n5月24日，大众匿名评测标注已达到预定目标的约30%，专业评测已完成总量的40%。LLMEVAL自推出以来，就受到了多方的广泛关注，为了能够满足大家的对该项目的期待，我们整理发布了初步评测结果。目前参与评测人员超过2000人，投票数量接近5.8万次。\nhttps://mp.weixin.qq.com/s/frAi29mgQE482hpgztHcPA\n\n### 2023.05.01 - 测试发布\n由于现代大模型主要侧重于文本生成能力，传统的BLUE值、ROUGE、精确率、召回率等评测指标已经无法充分反映出系统的实际效能。对于ChatGPT等类似的系统和模型，我们也迫切需要一种更专业、更大规模的评测方法和评测集合。因此，我们决定从认知心理学的角度出发，以人类处理信息、思考和解决问题的能力为基准，构建了一套包含17类，438条任务的全新评测问题集。我们采用众包与专业相结合的方式，匿名化系统名称，并通过对比评测的方式构建了LLMEVAL评测集，期待你的积极参与。\nhttps://mp.weixin.qq.com/s/0ZxX58FZxX6R_YY2sTEPig\n\n## Citation\n\nIf you find this repository useful, please cite our project:\n```\n@misc{LLMEval1,\n  author = {Ming Zhang and Yue Zhang and Shichun Liu and Haipeng Yuan and Junzhe Wang and Tao Gui and Qi Zhang and Xuanjing Huang},\n  title = {{LLMEval-1}},\n  year = {2023},\n  month = {June},\n  url = {https://github.com/llmeval/llmeval-1},\n}\n```\n\n## 联系我们\n\nLLMEval-3项目已经向公众开放，欢迎参与我们的评测。http://llmeval.com/\n\n机构评测需要进行认证，注册完账户以后，请联系管理员认证并申请评测权限。\n\n如无特殊情况，在评测完成之后，相关结果都会添加在排行榜上。\n\nEmail: mingzhang23@m.fudan.edu.cn\n\nWechat: zanyingluan\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllmeval%2Fllmeval-1","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fllmeval%2Fllmeval-1","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllmeval%2Fllmeval-1/lists"}