{"id":31034643,"url":"https://github.com/freedomintelligence/mevalkit","last_synced_at":"2025-09-14T02:46:40.055Z","repository":{"id":293013031,"uuid":"939811497","full_name":"FreedomIntelligence/MEvalKit","owner":"FreedomIntelligence","description":"An Evaluator Kit for different kinds of evaluation datasets.","archived":false,"fork":false,"pushed_at":"2025-09-12T08:15:30.000Z","size":79575,"stargazers_count":4,"open_issues_count":0,"forks_count":3,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-09-12T10:20:52.394Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-02-27T06:30:37.000Z","updated_at":"2025-09-12T08:15:35.000Z","dependencies_parsed_at":"2025-05-13T09:26:42.757Z","dependency_job_id":"d0b13142-a875-41c4-99f9-30bf68c92ee1","html_url":"https://github.com/FreedomIntelligence/MEvalKit","commit_stats":null,"previous_names":["freedomintelligence/mevalkit"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FreedomIntelligence/MEvalKit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMEvalKit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMEvalKit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMEvalKit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMEvalKit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/MEvalKit/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMEvalKit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275054971,"owners_count":25397576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-14T02:00:10.474Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-14T02:46:38.856Z","updated_at":"2025-09-14T02:46:40.043Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MEvalKit - 多模态大语言模型评测平台\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)\n[![Python](https://img.shields.io/badge/Python-3.8+-blue.svg)](https://www.python.org/)\n[![Flask](https://img.shields.io/badge/Flask-2.0+-green.svg)](https://flask.palletsprojects.com/)\n[![Docker](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://www.docker.com/)\n\n**MEvalKit** 是一个功能强大的多模态大语言模型评测平台，支持纯文本、多模态和LLMJudge型评测集，提供Web界面和命令行两种使用方式，让模型评测变得简单高效。\n\n## 🌟 项目特色\n\n- **🎯 一键评测**：仅用一句命令即可完成对已有或自定义评测集的评测\n- **🔧 多种模型支持**：通过标准OpenAI接口支持GPT系列、Qwen系列等多种纯文本及多模态模型\n- **📊 丰富评测集**：支持MMLU、MMStar、MT-Bench、CMB等多种评测集\n- **🌐 Web界面**：提供直观的Web管理界面，支持任务创建、进度监控、结果查看\n- **🐳 Docker部署**：提供完整的Docker部署方案，快速搭建评测环境\n- **🔄 断点续评**：支持评测任务的中断和恢复，避免重复计算\n- **📈 排行榜系统**：自动生成模型性能排行榜，便于比较分析\n- **🔒 数据安全**：支持加密数据库存储，保护评测数据安全\n\n## 📋 目录\n\n- [快速开始](#快速开始)\n- [支持的评测集](#支持的评测集)\n- [使用方式](#使用方式)\n  - [命令行使用](#命令行使用)\n  - [Web界面使用](#web界面使用)\n- [部署指南](#部署指南)\n  - [Docker快速部署](#docker快速部署)\n  - [手动安装部署](#手动安装部署)\n- [评测模式](#评测模式)\n- [API文档](#api文档)\n- [自定义评测集](#自定义评测集)\n- [项目结构](#项目结构)\n- [贡献指南](#贡献指南)\n- [许可证](#许可证)\n\n## 🚀 快速开始\n\n### 使用Docker（推荐）\n\n```bash\n# 克隆项目\ngit clone https://github.com/FreedomIntelligence/MEvalKit.git\ncd MEvalKit\n\n# 一键部署\n./deploy.sh\n\n# 访问Web界面\n# http://localhost:5000\n```\n\n### 手动安装\n\n```bash\n# 克隆项目\ngit clone https://github.com/FreedomIntelligence/MEvalKit.git\ncd MEvalKit\n\n# 安装依赖\npip install -r requirements.txt\n\n# 配置环境变量\ncp env.example .env\n# 编辑.env文件，设置API密钥等\n\n# 启动Web服务\npython app.py\n\n# 或使用命令行评测\npython run.py --dataset MMLU --model_name gpt-4o --evaluation_mode automatic\n```\n\n## 📊 支持的评测集\n\n| 评测集名称 | 类型 | 描述 | 数据源 |\n|-----------|------|------|--------|\n| **MMLU** | 纯文本 | 大规模多任务语言理解评测集 | [HuggingFace](https://huggingface.co/datasets/cais/mmlu) |\n| **GPQA** | 纯文本 | 研究生水平物理问答评测集 | [GitHub](https://github.com/idavidrein/gpqa) |\n| **CMB** | 纯文本 | 中文医学基准评测集 | [GitHub](https://github.com/FreedomIntelligence/CMB) |\n| **MMStar** | 多模态 | 多模态科学问答评测集 | [HuggingFace](https://huggingface.co/datasets/Lin-Chen/MMStar) |\n| **MT-Bench** | LLMJudge | 多轮对话评测集 | [GitHub](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge) |\n\n## 💻 使用方式\n\n### 命令行使用\n\n#### 自动模式（推荐）\n\n```bash\n# 基本用法\npython run.py --dataset MMLU --model_name gpt-4o --evaluation_mode automatic\n\n# 完整参数示例\npython run.py \\\n  --evaluation_mode automatic \\\n  --dataset MMLU \\\n  --model_name gpt-4o \\\n  --api_base \"https://api.openai.com/v1\" \\\n  --model_key \"your-api-key\" \\\n  --question_limitation 100 \\\n  --user_id \"test_user\"\n```\n\n#### 手动模式\n\n```bash\n# 使用预生成的响应文件进行评测\npython run.py \\\n  --evaluation_mode manual \\\n  --dataset MMLU \\\n  --model_name gpt-4o \\\n  --response_url \"https://example.com/responses.json\" \\\n  --question_limitation 100\n```\n\n### Web界面使用\n\n1. **访问主页**：打开浏览器访问 `http://localhost:5000`\n2. **创建评测任务**：点击\"开始新评测\"，填写评测参数\n3. **监控进度**：在任务详情页面查看实时进度\n4. **查看结果**：评测完成后查看详细结果和排行榜\n\n## 🐳 部署指南\n\n### Docker快速部署\n\n#### 一键部署（推荐）\n\n```bash\n# 首次部署（构建镜像并启动服务）\n./deploy.sh\n\n# 只构建镜像，不启动服务\n./deploy.sh --build-only\n\n# 只启动服务，不构建镜像\n./deploy.sh --start-only\n\n# 重新构建镜像并启动服务\n./deploy.sh --rebuild\n```\n\n#### 手动Docker部署\n\n```bash\n# 构建镜像\ndocker build -t mevalkit:latest .\n\n# 配置环境变量\ncp env.example .env\n# 编辑.env文件\n\n# 启动服务\ndocker-compose up -d\n\n# 查看日志\ndocker-compose logs -f\n```\n\n### 手动安装部署\n\n```bash\n# 1. 安装Python依赖\npip install -r requirements.txt\n\n# 2. 配置环境变量\ncp env.example .env\n# 编辑.env文件，设置必要的API密钥\n\n# 3. 启动Web服务\npython app.py\n\n# 4. 访问应用\n# Web界面: http://localhost:5000\n# API文档: http://localhost:5000/apidocs/\n```\n\n## 🔄 评测模式\n\n### 自动模式（Automatic）\n\n- **特点**：实时调用模型API进行评测\n- **适用场景**：有模型API访问权限的情况\n- **优势**：实时性好，支持大规模评测\n- **参数**：需要提供API密钥和接口地址\n\n### 手动模式（Manual）\n\n- **特点**：使用预生成的模型响应文件进行评测\n- **适用场景**：模型响应已预先生成或API访问受限\n- **优势**：成本低，可重复使用响应数据\n- **参数**：需要提供响应数据URL\n\n## 📚 API文档\n\n### 主要接口\n\n| 接口 | 方法 | 描述 |\n|------|------|------|\n| `/` | GET | 主页（总排行榜） |\n| `/new-evaluation` | GET | 创建评测任务页面 |\n| `/run-evaluation` | POST | 运行评测任务 |\n| `/task-status/\u003ctask_id\u003e` | GET | 获取任务状态 |\n| `/task-detail/\u003ctask_id\u003e` | GET | 查看任务详情 |\n| `/results` | GET | 查看所有结果 |\n\n### Swagger文档\n\n访问 `http://localhost:5000/apidocs/` 查看完整的API文档。\n\n## 🔧 自定义评测集\n\n### 配置文件结构\n\n评测集配置位于 `dataset_info/` 目录下：\n\n- `text_dataset.json` - 纯文本评测集配置\n- `image_dataset.json` - 多模态评测集配置  \n- `LLMJudge_dataset.json` - LLMJudge型评测集配置\n\n### 配置示例\n\n```json\n{\n  \"MyDataset\": {\n    \"language\": \"zh\",\n    \"question\": {\n      \"loading_way\": \"huggingface\",\n      \"path\": \"your-dataset-path\",\n      \"subset_name\": \"default\",\n      \"split_name\": \"test\",\n      \"key\": \"question\"\n    },\n    \"choices\": {\n      \"loading_way\": \"huggingface\",\n      \"path\": \"your-dataset-path\",\n      \"key\": [\"A\", \"B\", \"C\", \"D\"]\n    },\n    \"answer\": {\n      \"loading_way\": \"huggingface\",\n      \"path\": \"your-dataset-path\",\n      \"key\": \"answer\",\n      \"answer_type\": \"choice\"\n    }\n  }\n}\n```\n\n## 📁 项目结构\n\n```\nMEvalKit/\n├── app.py                 # Flask Web应用主文件\n├── run.py                 # 命令行评测入口\n├── requirements.txt       # Python依赖\n├── Dockerfile            # Docker镜像配置\n├── docker-compose.yml    # Docker Compose配置\n├── deploy.sh             # 一键部署脚本\n├── evaluation/           # 评测核心模块\n│   ├── TextMCQ_eval.py   # 文本多选题评测\n│   ├── ImageMCQ_eval.py  # 图像多选题评测\n│   └── LLMJudge_eval.py  # LLMJudge评测\n├── src/                  # 工具模块\n│   ├── utils/           # 工具函数\n│   ├── database/        # 数据库模块\n│   ├── api/            # API接口模块\n│   └── dataset/        # 数据集处理模块\n├── dataset_info/        # 数据集配置\n│   ├── text_dataset.json\n│   ├── image_dataset.json\n│   └── LLMJudge_dataset.json\n├── templates/           # Web模板\n├── results/            # 评测结果存储\n├── data/              # 数据集缓存\n└── logs/              # 日志文件\n```\n\n## 🤝 贡献指南\n\n我们欢迎社区贡献！请遵循以下步骤：\n\n1. Fork 本仓库\n2. 创建特性分支 (`git checkout -b feature/AmazingFeature`)\n3. 提交更改 (`git commit -m 'Add some AmazingFeature'`)\n4. 推送到分支 (`git push origin feature/AmazingFeature`)\n5. 开启 Pull Request\n\n### 贡献类型\n\n- 🐛 Bug修复\n- ✨ 新功能开发\n- 📚 文档改进\n- 🧪 测试用例\n- 🔧 性能优化\n\n## 📄 许可证\n\n本项目采用 [Apache License 2.0](LICENSE) 许可证。\n\n## 🙏 致谢\n\n本项目受益于以下开源项目：\n\n- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) - 多模态评测框架\n- [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) - 大语言模型训练框架\n\n## 📞 联系我们\n\n- 项目主页：https://github.com/FreedomIntelligence/MEvalKit\n- 问题反馈：https://github.com/FreedomIntelligence/MEvalKit/issues\n- 邮箱：1481345518@qq.com\n\n---\n\n**MEvalKit** - 让模型评测变得简单高效！ 🚀","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fmevalkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fmevalkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fmevalkit/lists"}