{"id":19053575,"url":"https://github.com/open-compass/code-evaluator","last_synced_at":"2026-04-02T18:37:41.363Z","repository":{"id":187652567,"uuid":"676443113","full_name":"open-compass/code-evaluator","owner":"open-compass","description":"A multi-language code evaluation tool.","archived":false,"fork":false,"pushed_at":"2024-01-26T04:12:27.000Z","size":599,"stargazers_count":24,"open_issues_count":1,"forks_count":10,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-10T20:56:14.931Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/open-compass.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-09T07:59:40.000Z","updated_at":"2025-06-29T12:01:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"c8d6acaf-343f-4105-8b6b-d6bde5195df4","html_url":"https://github.com/open-compass/code-evaluator","commit_stats":null,"previous_names":["ezra-yu/code-evaluator","open-compass/code-evaluator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/open-compass/code-evaluator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2Fcode-evaluator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2Fcode-evaluator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2Fcode-evaluator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2Fcode-evaluator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/open-compass","download_url":"https://codeload.github.com/open-compass/code-evaluator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2Fcode-evaluator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31313054,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T12:59:32.332Z","status":"ssl_error","status_checked_at":"2026-04-02T12:54:48.875Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T23:32:23.621Z","updated_at":"2026-04-02T18:37:41.340Z","avatar_url":"https://github.com/open-compass.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Code Evaluator\n\n[中文](./README_CN.md)\n\nA multi-language code evaluation tool.\n\n**Why create this repo.**\n\n1. **Environment integration**: When evaluating code, various environments need to be pre-installed, such as JDK for Java, Node for JavaScript, various versions of numpy and torch in DS1000, etc. This project pre-installs all these environments in a Docker image.\n\n2. **Easy to evaluate**: We can start the service through this repo with very simple steps, and then submit the results to the desired service. There's no need to go into the Docker container and execute the script, which can be quite cumbersome.\n\n## 📄 Table of Contents\n\n- [Support Datasets](#-support-datasets-)\n- [Evaluation Environments](#-evaluation-environments-)\n- [How to use](#-how-to-use-)\n- [Acknowledgement](#-acknowledgement-)\n\n## 📖 Support Datasets\u0026Language\n\n### Humanevalx\n\nHumanEval-X is a benchmark for evaluating the multilingual ability of code generative models. It consists of 820 high-quality human-crafted data samples (each with test cases) in **Python**, **C++**, **Java**, **JavaScript**, and **Go**, and can be used for various tasks, such as code generation and translation.\n\n[paper link](https://arxiv.org/abs/2303.17568)   \u0026nbsp;  [Github repo](https://github.com/THUDM/CodeGeeX2)  \u0026nbsp;  [Huggingface](https://huggingface.co/datasets/THUDM/humaneval-x)\n\n## 🛠️ Evaluation Environments\n\nThe generated code for evaluation requires compilation and execution in multiple languages. The versions of the programming languages we depend on, as well as the packages used are as follows:\n\n| Dependencies | Version |\n| ------- | -------- |\n| Python  | 3.8.12   |\n| JDK     | 18.0.2.1 |\n| Node.js | 16.14.0  |\n| js-md5  | 0.7.3    |\n| C++     | 11       |\n| g++     | 7.5.0    |\n| Boost   | 1.71.0   |\n| OpenSSL | 3.0.0    |\n| go      | 1.18.4   |\n\n## 👨‍🏫 How to use\n\n### 1. Launch a service\n\nMake sure you have install docker, and then build a image and run a service of container.\n\nbuild Docker Image:\n\nChoose your dataset: `humanevalx` or `ds1000`\n\n```shell\ngit clone https://github.com/open-compass/code-evaluator.git\nsudo docker build -t code-eval-{your-dataset}:latest -f docker/{your-dataset}/Dockerfile .\n```\n\nAfter getting the image, use the following command to create the container:\n\n```shell\n# Output Log Format\nsudo docker run -it -p 5000:5000 code-eval:latest python server.py\n\n# Running programs in the background\n# sudo docker run -itd -p 5000:5000 code-eval:latest python server.py\n\n# use differnet port\n# sudo docker run -itd -p 5001:5001 code-eval:latest python server.py --port 5001\n```\n\nMake sure you can reach the service by checking the fllowing commands(If you run service in the loaclhost, just skip this.):\n\n```shell\nping your_service_ip_address\ntelnet your_service_ip_address your_service_port\n```\n\n### 2. Prepare submit result files\n\n\n### humanevalx\nWe give sample formats for different datasets in the [examples](./examples/) folder.\n\nLet's take huamanevalx as an example，which submits results in the following format：\n\n```text\n{\"task_id\": \"../..\", \"generation: \"...\"}\n{\"task_id\": \"../..\", \"generation: \"...\"}\n...\n```\n\n### ds1000\n\nSkip this step, use prediction by opencompass directly.\n\n### 3. Submit service request\n\nUse curl to submit your request\n\n```shell\ncurl -X POST -F 'file=@{result_absolute_path}' -F 'dataset={dataset/language}' {your_service_ip_address}:{your_service_port}/evaluate\n```\n\nsuch as evaluate 'humanevalx/python' in 'localhost:5000':\n\n```shell\ncurl -X POST -F 'file=@./examples/humanevalx/python.json' -F 'dataset=humanevalx/python' localhost:5000/evaluate\n```\n\nYou will get the fllowing result：\n\n```text\n\"{\\\"pass@1\\\": 37.19512195121951}\"% \n```\n\nsuch as evaluate 'ds1000_Numpy' in 'localhost:5000':\n\n```shell\ncurl -X POST -F 'file=@./internlm-chat-7b-hf-v11/ds1000_Numpy.json' localhost:5000/evaluate\n```\n\nYou will get the fllowing result：\n\n```text\n\"{\\\"accuracy\\\": xx}\"%\n```\n\n## 🤝 Acknowledgements\n\nSome code in this project is cited and modified from [CodeGeeX2](https://github.com/THUDM/CodeGeeX2). Thanks for [THUDM Team](https://github.com/THUDM).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-compass%2Fcode-evaluator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopen-compass%2Fcode-evaluator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-compass%2Fcode-evaluator/lists"}