{"id":13773706,"url":"https://github.com/NVlabs/verilog-eval","last_synced_at":"2025-05-11T06:30:32.978Z","repository":{"id":194928199,"uuid":"682202757","full_name":"NVlabs/verilog-eval","owner":"NVlabs","description":"Verilog evaluation benchmark for large language model","archived":false,"fork":false,"pushed_at":"2024-08-21T19:59:45.000Z","size":426,"stargazers_count":147,"open_issues_count":6,"forks_count":16,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-08-21T22:16:47.358Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"SystemVerilog","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-23T17:00:58.000Z","updated_at":"2024-08-21T21:03:34.000Z","dependencies_parsed_at":"2024-08-21T22:03:42.771Z","dependency_job_id":null,"html_url":"https://github.com/NVlabs/verilog-eval","commit_stats":null,"previous_names":["nvlabs/verilog-eval"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fverilog-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fverilog-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fverilog-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fverilog-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVlabs","download_url":"https://codeload.github.com/NVlabs/verilog-eval/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225017824,"owners_count":17407832,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T17:01:19.232Z","updated_at":"2024-11-17T08:31:28.173Z","avatar_url":"https://github.com/NVlabs.png","language":"SystemVerilog","funding_links":[],"categories":["Benchmarks","Python"],"sub_categories":[],"readme":"# VerilogEval Overview\n\nThis is an evaluation harness for the VerilogEval problem solving dataset originally described in the paper \"[VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://arxiv.org/abs/2309.07544),\" published in 2023. In August 2024, this repository was revised to cover specification-to-RTL tasks in addition to the original code completion task, add in-context learning examples to prompts, and categorize common iverilog failures. Please see the related apaper \"[Revisiting VerilogEval: Newer LLMs, In-Context Learning, and Specification-to-RTL Tasks](https://arxiv.org/abs/2408.11053),\" published in 2024.\n\n**If you would like to benchmark against the original VerilogEval 1.0 harness, please checkout Git branch \"release/1.0.0\" which has been kept to preserve this original benchmark. Otherwise, the main branch can be used for the improved harness.**\n\n### VerilogEvalV2 with Reframed Prompts and New Scripts\n\nThis repo contains the original VerilogEval dataset with reframed prompts\nand new scripts. The original VerilogEval prompts explicitly included the\nVerilog module interface, while in this version we specify the module\ninterface more abstractly. The new scripts manage the dataset as plain\ntext files (instead of a large JSONL file), include generation and\nanalysis scripts, and include a Makefile to drive the workflow. The\ngeneration script includes support for easily changing the LLM model,\nincluding/excluding in-context learning rules and in-context learning\nexamples. The analysis script includes support for categorizing common\niverilog errors and outputing the results in both plain text and CSV\nfiles.\n\nMachineEval is not supported in VerilogEvalV2, only the Human Eval problem statements. Pass@10 is no longer being reported either, instead Pass@1 with number of samples n=1 (temperature=0, top_p=0.01) and n=20 (temperature=0.85, top_p=0.95) for low and high and temperature results, respectively.\n\n### Setup Linux Environment\n\nIn order to use PyHDL-Eval you will need to install iverilog, verilator,\nand python3 along with several Python packages. These are the versions\nwhich were used for this project:\n\n - iverilog (v12)\n - python3 (v3.11.0)\n\n**Please note that iverilog v13 (development release) is not supported.**\n\nTo install Python 3.11:\n```\n$ conda create -n codex python=3.11\n$ conda activate codex\n```\n\nInstall [ICARUS Verilog](https://github.com/steveicarus/iverilog):\n```\n$ git clone https://github.com/steveicarus/iverilog.git \u0026\u0026 cd iverilog \\\n        \u0026\u0026 git checkout v12-branch \\\n        \u0026\u0026 sh ./autoconf.sh \u0026\u0026 ./configure \u0026\u0026 make -j4\\\n        \u0026\u0026 make install\n```\n\nYou will also need the following Python packages:\n\n```\n % pip install langchain langchain-openai langchain-nvidia-ai-endpoints\n```\n\nWe plan to provide a Dockerfile and backwards compatibility mode with a prebuilt jsonl soon.\n\n### Usage \n\nThe evalution harness is run using make and various evaluation parameters can be set as below:\n\n```\nmkdir -p build/\n../configure  --with-task=$task --with-model=$model --with-examples=$shots --with-samples=$samples --with-temperature=$temperature --with-top-p=$top_p\nmake\n```\n\nEvaluation can be sped up by providing the `-j` flag to make, such as `-j4` to run 4 worker processes.\n\nAvailable tasks are `code-complete-iccad2023` and `spec-to-rtl` with each referencing their corresponding `dataset_$task` directory containig the problems. Problem themselves are identical between the two datasets and only the task format changes.\n\nValid models are listed at the top of `scripts/sv-generate`. The number of in-context learning examples can be between 0-4, and given with `--with-examples`. Samples to collect per problem are given by `--with-samples`. Finally, model temperature and top_p can be set to --with-temperature and --with-top-p, respectively.\n\nThese parameters can be easily swept with a shell script, to create separate build directories for each evaluation harness configuration target. \n\n## Citation\n\nFor this VerilogEval v2, please cite the following paper:\n\n```\n@misc{pinckney2024revisitingverilogevalnewerllms,\n      title={Revisiting VerilogEval: Newer LLMs, In-Context Learning, and Specification-to-RTL Tasks}, \n      author={Nathaniel Pinckney and Christopher Batten and Mingjie Liu and Haoxing Ren and Brucek Khailany},\n      year={2024},\n      eprint={2408.11053},\n      archivePrefix={arXiv},\n      primaryClass={cs.SE},\n      url={https://arxiv.org/abs/2408.11053}, \n}\n```\n\nFor the original VerilogEval v1, please use:\n\n```\n@inproceedings{liu2023verilogeval,\n  title={{VerilogEval:} Evaluating Large Language Models for Verilog Code Generation},\n  author={Liu, Mingjie and Pinckney, Nathaniel and Khailany, Brucek and Ren, Haoxing},\n  booktitle={2023 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)}, \n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNVlabs%2Fverilog-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNVlabs%2Fverilog-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNVlabs%2Fverilog-eval/lists"}