{"id":19932136,"url":"https://github.com/amazon-science/cceval","last_synced_at":"2025-07-24T15:02:13.589Z","repository":{"id":200887120,"uuid":"705470018","full_name":"amazon-science/cceval","owner":"amazon-science","description":"CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion (NeurIPS 2023)","archived":false,"fork":false,"pushed_at":"2024-07-29T15:11:21.000Z","size":73393,"stargazers_count":136,"open_issues_count":4,"forks_count":20,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-03T09:21:22.305Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://crosscodeeval.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amazon-science.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-16T04:23:03.000Z","updated_at":"2025-03-21T00:26:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"33ceda45-2db2-4e7c-bee8-d60f08d64f84","html_url":"https://github.com/amazon-science/cceval","commit_stats":null,"previous_names":["amazon-science/cceval"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/amazon-science/cceval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fcceval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fcceval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fcceval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fcceval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amazon-science","download_url":"https://codeload.github.com/amazon-science/cceval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fcceval/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266861201,"owners_count":23996557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T23:09:11.667Z","updated_at":"2025-07-24T15:02:13.526Z","avatar_url":"https://github.com/amazon-science.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion\n\nThis repository contains the data and inference code of the NeurIPS 2023  (Datasets and Benchmarks track)\npaper \"[CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion](https://arxiv.org/abs/2310.11248).\"\n\n## Requirements\n\n- Uncompress the CrossCodeEval data via `tar -xvJf data/crosscodeeval_data.tar.xz -C data/`\n    - The data contains {baseline, retrieval, retrieval w/ ref.} setting x {bm25, UniXCoder, OpenAI Ada} retriever.\n    - **Please email us if you need the raw data.**\n- Install dependencies via `pip install -r requirements.txt`\n- Build tree sitter via `bash scripts/build_treesitter.sh`\n\n\n## Evaluation on CrossCodeEval\nOur evaluation consists of two steps: generation and metrics calculation.\n\n\n### Generation\n\n#### Publicly Available Models\nFor publicly available models like StarCoder, DeepSeek-Coder, etc., we recommended using [vLLM](https://github.com/vllm-project/vllm) for fast and distributed inference on CrossCodeEval. \n\n```bash\nexport gpus=2\nexport model=bigcode/starcoder2-3b\nexport language=python\nexport task=line_completion_rg1_unixcoder_cosine_sim\nexport output_dir=./tmp/crosscodeeval_testrun/\npython scripts/vllm_inference.py \\\n  --tp $gpus \\\n  --task $task \\\n  --language $language \\\n  --model $model \\\n  --output_dir $output_dir \\\n  --use_crossfile_context \n```\nFor additional args, e.g., cross-file context length and sampling top_p, please see `python vllm_inference.py --help`.\n\n\u003cdetails\u003e\u003csummary\u003e If you prefer non-vLLM script \u003ci\u003e:: click to expand ::\u003c/i\u003e\u003c/summary\u003e\n\u003cdiv\u003e\n\nFirst, configure `accelerate` via `accelerate config` if you haven't. A reference configuration is available at `cceval_config.yaml`\n\nThe following command demonstrates how to run greedy eval using codegen-350M on python with cross-file context.\n\n```bash\nexport model_type=codelm_cfc # or codelm for no cross-file context eval\nexport model_name=Salesforce/codegen-350M-mono\nexport language=python\nexport ts_lib=./build/${language}-lang-parser.so\nexport dtype=bf16 # or fp16\nexport prompt_file=./data/crosscodeeval_data/${language}/line_completion_rg1_unixcoder_cosine_sim.jsonl # or other options in the dir, which corresponds to different retrieval methods and/or retrieval settings\nexport max_seq_length=2048\nexport cfc_seq_length=512 \nexport batch_size=16 # reduce for larger models\nexport output_dir=./tmp/crosscodeeval_testrun/\n\naccelerate launch eval.py \\\n        --model_type $model_type \\\n        --model_name_or_path $model_name \\\n        --cfc_seq_length $cfc_seq_length \\\n        --prompt_file $prompt_file \\\n        --gen_length 50 \\\n        --max_seq_length $max_seq_length \\\n        --batch_size $batch_size \\\n        --output_dir $output_dir \\\n        --dtype $dtype \\\n        --num_return_sequences 1 \\\n        --overwrite_cache True \\\n        --ts_lib $ts_lib \\\n        --language $language\n```\n\nYou may run sampling via the following (additional) args:\n\n```bash\n        --do_sample \\\n        --top_p 0.95 \\\n        --temperature 0.2 \\\n        --num_return_sequences 5 \\\n```\n\n\n\u003c/div\u003e\n\u003c/details\u003e\n\n#### OpenAI models\nOpenAI models are accessible through an API. You may use the following script:\n```bash\nexport model=gpt-3.5-turbo-0125 \nexport language=python\nexport task=line_completion_rg1_unixcoder_cosine_sim\nexport output_dir=./tmp/crosscodeeval_openai_testrun/\npython scripts/openai_inference.py \\\n  --task $task \\\n  --language $language \\\n  --model $model \\\n  --output_dir $output_dir \\\n  --use_crossfile_context \n\n```\n\n\n### Metrics Calculation\nAfter obtaining the generation, we can calculate the final metrics\n```bash\nexport language=python\nexport ts_lib=./build/${language}-lang-parser.so; \nexport task=line_completion_oracle_unixcoder_cosine_sim\nexport prompt_file=./data/${language}/${task}.jsonl \nexport output_dir=./tmp/crosscodeeval_testrun/;  \npython scripts/eval.py \\\n  --prompt_file $prompt_file \\\n  --output_dir $output_dir \\\n  --ts_lib $ts_lib \\\n  --language $language \\\n  --only_compute_metric\n```\n\n\n\n\n\n\n\n## Citation\n\n```\n\n@inproceedings{ding2023crosscodeeval,\n    title={CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion}, \n    author={Yangruibo Ding and Zijian Wang and Wasi Uddin Ahmad and Hantian Ding and Ming Tan and Nihal Jain and Murali Krishna Ramanathan and Ramesh Nallapati and Parminder Bhatia and Dan Roth and Bing Xiang},\n    year={2023},\n    booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},\n    url={https://arxiv.org/pdf/2310.11248.pdf}\n}\n```\n## Questions\nPlease feel free to email us (email addresses in the [paper](https://arxiv.org/pdf/2310.11248.pdf)). You may also submit an issue in this repo.\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis project is licensed under the Apache-2.0 License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Fcceval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famazon-science%2Fcceval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Fcceval/lists"}