{"id":19053570,"url":"https://github.com/open-compass/compassjudger","last_synced_at":"2025-11-12T03:02:51.289Z","repository":{"id":258777060,"uuid":"860467149","full_name":"open-compass/CompassJudger","owner":"open-compass","description":"The All-in-one Judge Models introduced by Opencompass","archived":false,"fork":false,"pushed_at":"2025-07-15T02:52:08.000Z","size":153,"stargazers_count":97,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-07-15T07:02:05.901Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/open-compass.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-20T13:43:15.000Z","updated_at":"2025-07-15T06:33:29.000Z","dependencies_parsed_at":"2024-10-25T12:39:40.836Z","dependency_job_id":"72b983ff-a879-4b97-92ab-98b6cbfa74ae","html_url":"https://github.com/open-compass/CompassJudger","commit_stats":null,"previous_names":["open-compass/compassjudger"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/open-compass/CompassJudger","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FCompassJudger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FCompassJudger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FCompassJudger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FCompassJudger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/open-compass","download_url":"https://codeload.github.com/open-compass/CompassJudger/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FCompassJudger/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":283966764,"owners_count":26924587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-12T02:00:06.336Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T23:32:16.703Z","updated_at":"2025-11-12T03:02:51.284Z","avatar_url":"https://github.com/open-compass.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards\n\n\n\u003cp align=\"center\"\u003e\n \u003ca href=\"https://github.com/open-compass/opencompass\"\u003e\n \u003cimg\n src=\"https://img.shields.io/badge/OpenCompass-GitHub-0A66C2?logo=github\u0026logoColor=white\"\n alt=\"OpenCompass GitHub\"\n /\u003e\n \u003c/a\u003e\n \u003ca href=\"https://arxiv.org/abs/2507.09104\"\u003e\n \u003cimg\n src=\"https://img.shields.io/badge/CompassJudger--2-Paper-red?logo=arxiv\u0026logoColor=red\"\n alt=\"CompassJudger-2 Paper on arXiv\"\n /\u003e\n \u003c/a\u003e\n \u003ca href=\"https://huggingface.co/opencompass/CompassJudger-2-7B-Instruct\"\u003e\n \u003cimg \n src=\"https://img.shields.io/badge/CompassJudger--2-Model-yellow?logo=huggingface\u0026logoColor=yellow\" \n alt=\"CompassJudger-2 Model\"\n /\u003e\n \u003c/a\u003e\n \u003ca href=\"https://discord.gg/WgjnuBmDAd\"\u003e\n \u003cimg\n src=\"https://img.shields.io/badge/OpenCompass-Discord-5865F2?logo=discord\u0026logoColor=purple\"\n alt=\"OpenCompass Discord\"\n /\u003e\n \u003c/a\u003e\n\u003c/p\u003e\n\n\u003e Taolin Zhang*, Maosong Cao*, Alexander Lam, Songyang Zhang, Kai Chen\n\u003e\n\u003e Recently, the role of LLM-as-judge in evaluating large language models has gained prominence. However, current judge models suffer from narrow specialization and limited robustness, undermining their capacity for comprehensive evaluations. In this work, we present \\textit{CompassJudger-2}, a novel generalist judge model that overcomes these limitations via a task-driven, multi-domain data curation strategy. Central to our approach is supervising judgment tasks with verifiable rewards, guiding intrinsic critical reasoning through rejection sampling to foster robust, generalizable judgment capabilities. We introduce a refined learning objective with margin policy gradient loss to enhance performance. Empirically, CompassJudger-2 achieves superior results across multiple judge and reward benchmarks, and our 7B model demonstrates competitive judgment accuracy with significantly larger models like DeepSeek-V3 and Qwen3-235B-A22B. Additionally, we propose JudgerBenchV2, a comprehensive benchmark evaluating cross-domain judgment accuracy and rank consistency to standardize judge model evaluation. These contributions advance robust, scalable LLM judgment and establish new performance and evaluation standards.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"./assets/framework.png\" width=\"95%\"\u003e\u003c/p\u003e\n\n## πŸ“’ News\n\n- **July 11, 2025:** Our paper, **CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards**, is now available.\n\n## πŸ€— Models\n\n| Model Name | Size | Base Model | Download | Notes |\n| :--------------------------------- | :--: | :------------------- | :----------------------------------------------------------: | :-------------------------------------------- |\n| πŸ‘‰ **CompassJudger-1-7B-Instruct** | 7B | Qwen2.5-7B-Instruct | πŸ€— [Model](https://huggingface.co/opencompass/CompassJudger-1-7B-Instruct) | CompassJudger-1 7B Model. |\n| πŸ‘‰ **CompassJudger-2-32B-Instruct** | 32B | Qwen2.5-32B-Instruct | πŸ€— [Model](https://huggingface.co/opencompass/CompassJudger-1-32B-Instruct) | CompassJudger-1 32B Model. |\n| πŸ‘‰ **CompassJudger-2-7B-Instruct** | 7B | Qwen2.5-7B-Instruct | πŸ€— [Model](https://huggingface.co/opencompass/CompassJudger-2-7B-Instruct) | Fine-tuned for generalist judge capabilities. |\n| πŸ‘‰ **CompassJudger-2-32B-Instruct** | 32B | Qwen2.5-32B-Instruct | πŸ€— [Model](https://huggingface.co/opencompass/CompassJudger-2-32B-Instruct) | A larger, more powerful judge model. |\n\n## πŸ”₯ Quick Start\n\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_path = \"opencompass/CompassJudger-2-7B-Instruct\"\n\nmodel = AutoModelForCausalLM.from_pretrained(\n model_name,\n torch_dtype=\"auto\",\n device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Example: Pair-wise Comparison\nprompt = \"\"\"\nPlease act as an impartial judge to evaluate the responses provided by two AI assistants to the user question below. Your evaluation should focus on the following criteria: helpfulness, relevance, accuracy, depth, creativity, and level of detail.\n\n- Do not let the order of presentation, response length, or assistant names influence your judgment.\n- Base your decision solely on how well each response addresses the user’s question and adheres to the instructions.\n\nYour final reply must be structured in the following format:\n{\n \"Choice\": \"[Model A or Model B]\"\n}\n\nUser Question: {question}\n\nModel A's Response: {answerA}\n\nModel B's Response: {answerB}\n\nNow it's your turn. Please provide selection result as required:\n\"\"\"\n\nmessages = [\n {\"role\": \"user\", \"content\": prompt}\n]\n\ntext = tokenizer.apply_chat_template(\n messages,\n tokenize=False,\n add_generation_prompt=True\n)\nmodel_inputs = tokenizer([text], return_tensors=\"pt\").to(model.device)\n\ngenerated_ids = model.generate(\n **model_inputs,\n max_new_tokens=2048\n)\ngenerated_ids = [\n output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)\n]\n\nresponse = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\nprint(response)\n```\n\n## πŸ“Š JudgerBenchV2\n\n### Evaluate on JudgerBenchV2\nWe provide scripts for evaluating JudgerBenchV2 within the OpenCompass framework. Please refer to the [OpenCompass](https://github.com/open-compass/opencompass) repository.\n\nModify the `models` in `examples/eval_judgerbenchv2.py` to your judge model, then run:\n```bash\ngit clone https://github.com/open-compass/opencompass opencompass\ncd opencompass\npip install -e .\npython run.py examples/eval_judgerbenchv2.py --mode all --reuse latest\n```\n\n## πŸ“Š Benchmarks\n\n### Judge Model Benchmarks.\nCompassJudger-2 achieves state-of-the-art performance on major judge benchmarks across both 7B and 32B+ variants.\n\n| Model | JudgerBenchV2 | JudgeBench | RMB | RewardBench | Average |\n| :------------------------------- | :-----------: | :--------: | :-------: | :---------: | :-------: |\n| **General Models** | | | | | |\n| Qwen2.5-7B-Instruct | 57.14 | 23.23 | 69.03 | 79.69 | 57.27 |\n| Llama3.1-8B-Instruct | 57.64 | 33.23 | 66.01 | 73.64 | 57.63 |\n| Qwen2.5-32B-Instruct | 62.97 | 59.84 | 74.99 | 85.61 | 70.85 |\n| DeepSeek-V3-0324 | 64.43 | 59.68 | 78.16 | 85.17 | 71.86 |\n| Qwen3-235B-A22B | 61.40 | 65.97 | 75.59 | 84.68 | 71.91 |\n| **7B Judge Models** | | | | | |\n| CompassJudger-1-7B-Instruct | 57.96 | 46.00 | 38.18 | 80.74 | 55.72 |\n| Con-J-7B-Instruct | 52.35 | 38.06 | 71.50 | 87.10 | 62.25 |\n| RISE-Judge-Qwen2.5-7B | 46.12 | 40.48 | 72.64 | 88.20 | 61.61 |\n| **CompassJudger-2-7B-Instruct** | **60.52** | **63.06** | **73.90** | **90.96** | **72.11** |\n| **32B+ Judge Models** | | | | | |\n| CompassJudger-1-32B-Instruct | 60.33 | 62.29 | 77.63 | 86.17 | 71.61 |\n| RISE-Judge-Qwen2.5-32B | 56.42 | 63.87 | 73.70 | 92.70 | 71.67 |\n| **CompassJudger-2-32B-Instruct** | **62.21** | **65.48** | **72.98** | **92.62** | **73.32** |\n\n\n## ✍️ Citation\n\n```bibtex\n@article{zhang2025compassjudger,\n title={CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards},\n author={Zhang, Taolin and Cao, Maosong and Lam, Alexander and Zhang, Songyang and Chen, Kai},\n journal={arXiv preprint arXiv:2507.09104},\n year={2025}\n}\n@article{cao2024compass,\n title={CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution},\n author={Maosong Cao, Alexander Lam, Haodong Duan, Hongwei Liu, Songyang Zhang, Kai Chen},\n journal={arXiv preprint arXiv:2410.16256},\n year={2024}\n}\n```\n\n\n## πŸ“œ License\nCompassJudger-2 is licensed under the Apache 2.0.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-compass%2Fcompassjudger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopen-compass%2Fcompassjudger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-compass%2Fcompassjudger/lists"}