{"id":27948308,"url":"https://github.com/pku-alignment/eval-anything","last_synced_at":"2025-05-07T14:57:30.601Z","repository":{"id":276413505,"uuid":"929222473","full_name":"PKU-Alignment/eval-anything","owner":"PKU-Alignment","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-31T09:11:49.000Z","size":276,"stargazers_count":11,"open_issues_count":4,"forks_count":7,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-07T14:57:26.871Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PKU-Alignment.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-08T03:34:20.000Z","updated_at":"2025-04-21T07:37:50.000Z","dependencies_parsed_at":"2025-02-08T04:23:04.232Z","dependency_job_id":"3fee8e4e-85bb-4042-b0ce-7a9c312e4064","html_url":"https://github.com/PKU-Alignment/eval-anything","commit_stats":null,"previous_names":["kass123777/eval-anything","pku-alignment/eval-anything"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-Alignment%2Feval-anything","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-Alignment%2Feval-anything/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-Alignment%2Feval-anything/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-Alignment%2Feval-anything/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PKU-Alignment","download_url":"https://codeload.github.com/PKU-Alignment/eval-anything/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252902624,"owners_count":21822257,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T14:57:29.929Z","updated_at":"2025-05-07T14:57:30.585Z","avatar_url":"https://github.com/PKU-Alignment.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# eval-anything\n\n## 代码结构\n```\neval-anything/\n├── eval-anything/\n│   ├── benchmarks/                     # 各benchmark的评测代码\n│   │   ├── text_image_to_text/         # 以模态作为一级分类\n│   │   │   └── mmmu/                   # benchmark子文件夹\n│   │   │       ├── mmmu.yaml           # 存储评测集的基本信息，包括数据集路径、size、评测方式、key值...\n│   │   │       └── eval.py             # 评测代码，继承自eval-anything/pipeline下的基类\n│   │   └── text_to_text/\n│   │       └── gsm8k/\n│   │           ├── gsm8k.yaml\n│   │           └── eval.py\n│   ├── configs/                        # 评测参数文件夹\n│   │   └── evaluate.yaml               # 包含了推理参数（top-k, temperature...）、评测参数(cot, few-shot...)\n│   ├── dataloader/                     # 数据集加载与预处理\n│   │   ├── base_dataloader.py          # dataloader基类，继承使用\n│   │   ├── mm_dataloader.py            # 多模态dataloader，支持多模态数据的本地和在线读取，支持数据预处理\n│   │   └── t2t_dataloader.py           # 纯文本dataloader，支持数据加载和预处理\n│   ├── models/                         # 各模型推理的代码，包含在线api、hf、vllm以及多模态生成模型各自的生成代码\n│   │   ├── base_model.py\n│   │   ├── api.py                      # 在线api访问\n│   │   ├── hf_lm.py                    # 基于transformers的t2t推理\n│   │   ├── hf_mm.py                    # 基于transformers的多模态推理\n│   │   ├── vllm_lm.py                  # 基于vllm的t2t推理\n│   │   └── vllm_mm.py                  # 基于vllm的多模态推理推理\n│   ├── pipeline/                       # 完整评测pipeline代码\n│   │   ├── base_task.py                # pipeline基类，一般直接使用\n│   │   ├── base_benchmark.py           # benchmark基类，继承使用\n│   │   ├── mm_gen_benchmark.py         # 多模态生成任务benchmark\n│   │   ├── mm_und_benchmark.py         # 多模态理解任务benchmark\n│   │   └── t2t_benchmark.py            # t2t任务benchmark\n│   ├── evaluate_tools/                 # 评测过程中常见的工具包，例如计算各种指标、pattern匹配等\n│   │   ├── base_tools.py               # 评测指标和评测工具的基类\n│   │   ├── metrics.py                  # 各种评测指标的计算 \n│   │   └── t2t_tools.py                # 评测工具\n│   └── utils/                          # 项目中使用到的工具包\n│       ├── cache_manager.py            # 推理过程的cache机制\n│       ├── data_type.py                # 定义数据基类，用于存储模型输入数据和评测结果数据\n│       ├── register.py                 # 注册器类\n│       ├── logger.py                   # 评测logger，用于输出评测结果以及写入本地log\n│       ├── utils.py                    # 框架开发时使用的通用工具\n│       └── template.py                 # chat_template\n├── scripts/                        # 评测启动脚本文件夹\n│   └── run.sh                      # 评测启动脚本\n├── setup.py\n└── README.md\n```\n\n## Installation\n\n```bash\nconda create -n eval-anything python==3.11\npip install -e .\n```\n\n## 开发指南\n- [ ] 从旧框架中剥离出dataloader, inferencer进行适当复用，但是需要根据多模态特性进行更完善的基类开发\n    - [x] T2T\n    - [ ] 多模态\n    - [ ] **$New$** 在dataloader中支持多轮对话数据的拼接\n- [x] 复用旧框架中的data_type, logger, template\n- [ ] **$New$** 设计更加完善的cache机制，保存中间推理和评测结果\n    - [x] T2T\n    - [ ] 多模态\n- [x] **$New$** 参考lm_eval重新开发pipeline基类，后续的benchmark适配（eval.py）完全以面向对象的形式进行开发\n- [x] **$New$** 开发evaluator.py（面向对象），实现常用评测指标的计算。例如win rate, pass rate, F1-Score, ASR, SD, word error rate (WER) (↓), BLEU(↑),  automatic speech recognition (ASR), perplexity\n- [x] 对传参体系进行重新设计\n    - [x] **$New$** 参考lm_eval，构建数据集的yaml文件\n    - [x] 重新设计传递评测config的yaml文件\n- [ ] 整理不在hf上的数据集，统一键值\n- [ ] 在eval-anything/models底下进行各种多模态生成模型的推理单独适配（非紧急，但必要）\n\n## 期望的Features\n\n- 多节点自动化评测，给机器、任务、模型的list，自动分配机器进行评测\n- 优化传参体系\n    - 以yaml为一次评测的传参接口，在启动评测时既能支持bash传参，也能通过yaml传参\n    - 支持同时指定多个yaml，实现每个benchmark细粒度的调参\n    - 保存evaluation结果时，还同时能保存评测参数（yaml+命令行参数），可以参考align-anything\n- 若有多个benchmark串行评测，不重复load模型；同一个benchmark不同task评测时，不重复infer\n- 对于多模态评测，有一个直观展示评测结果（过程）的方式，类似gradio\n- docker评测","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpku-alignment%2Feval-anything","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpku-alignment%2Feval-anything","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpku-alignment%2Feval-anything/lists"}