{"id":39546433,"url":"https://github.com/fsoft-ai4code/code-llm-evaluator","last_synced_at":"2026-01-18T06:46:02.869Z","repository":{"id":239796248,"uuid":"778873110","full_name":"FSoft-AI4Code/code-llm-evaluator","owner":"FSoft-AI4Code","description":"🏎️ Fast Code LLMs evaluator","archived":false,"fork":false,"pushed_at":"2024-05-14T16:55:35.000Z","size":40,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-05T13:17:45.136Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://code-llm-evaluator.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FSoft-AI4Code.png","metadata":{"files":{"readme":"README.rst","changelog":"HISTORY.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.bib","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-28T15:21:31.000Z","updated_at":"2025-02-21T12:37:38.000Z","dependencies_parsed_at":"2024-05-15T12:21:22.877Z","dependency_job_id":"1f34b0a3-3d94-45e1-97ed-006a384cbb72","html_url":"https://github.com/FSoft-AI4Code/code-llm-evaluator","commit_stats":null,"previous_names":["fsoft-ai4code/code-llm-evaluator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FSoft-AI4Code/code-llm-evaluator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FSoft-AI4Code%2Fcode-llm-evaluator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FSoft-AI4Code%2Fcode-llm-evaluator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FSoft-AI4Code%2Fcode-llm-evaluator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FSoft-AI4Code%2Fcode-llm-evaluator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FSoft-AI4Code","download_url":"https://codeload.github.com/FSoft-AI4Code/code-llm-evaluator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FSoft-AI4Code%2Fcode-llm-evaluator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28531998,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-18T06:46:02.763Z","updated_at":"2026-01-18T06:46:02.861Z","avatar_url":"https://github.com/FSoft-AI4Code.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"=================\nCodeLLM Evaluator\n=================\nEasy to evaluate with fast inference settings CodeLLMs\n\nOverview\n========\n`CodeLLM Evaluator` provide the ability for fast and efficiently evaluation \non code generation task. Inspired by `lm-evaluation-harness \u003chttps://github.com/EleutherAI/lm-evaluation-harness\u003e`_ and `bigcode-eval-harness \u003chttps://github.com/bigcode-project/bigcode-evaluation-harness\u003e`_,\nwe designed our framework for multiple use-case, easy to add new metrics and customized task.\n\n**Features:**\n\n* Implemented HumanEval, MBPP benchmarks for Coding LLMs.\n* Support for models loaded via `transformers \u003chttps://github.com/huggingface/transformers\u003e`_, `DeepSpeed \u003chttps://github.com/microsoft/DeepSpeed\u003e`_.\n* Support for evaluation on adapters (e.g. LoRA) supported in HuggingFace's `PEFT \u003chttps://github.com/huggingface/peft\u003e`_ library.\n* Support for inference with distributed native transformers or fast inference with `VLLMs \u003chttps://github.com/vllm-project/vllm\u003e`_ backend.\n* Easy support for custom prompts, task and metrics.\n\nSetup\n=====\nInstall `code-eval` package from the github repository via `pip`:\n\n.. code-block:: console\n\n    $ git clone https://github.com/FSoft-AI4Code/code-llm-evaluator.git\n    $ cd code-llm-evaluator\n    $ pip install -e .\n\nQuick-start\n===========\n\nTo evaluate a supported task in python, you can load our :py:func:`code_eval.Evaluator` to generate\nand compute evaluate metrics on the run.\n\n.. code-block:: python\n\n    from code_eval import Evaluator\n    from code_eval.task import HumanEval\n\n    task = HumanEval()\n    evaluator = Evaluator(task=task)\n\n    output = evaluator.generate(num_return_sequences=3,\n                                batch_size=16,\n                                temperature=0.9)\n    result = evaluator.evaluate(output)\n\n\nCLI Usage\n=========\n\nInference with Transformers\n---------------------------\n\nLoad model and generate answer using native transformers (``tf``), pass model local path or\nHuggingFace Hub name. We select transformers as default backend, but you can pass ``backend=\"tf\"`` to specify it:\n\n.. code-block:: console\n\n    $ code-eval --model_name microsoft/phi-1 \\\n        --task humaneval \\\n        --batch_size 8 \\\n        --backend hf \\\n\n.. tip:: \n    Load LoRA adapters by add ``--peft_model`` argument. The ``--model_name`` must point\n    to full model architecture.\n\n    .. code-block:: console\n\n        $ code-eval --model_name microsoft/phi-1 \\\n            --peft_model \u003cadapters-name\u003e \\\n            --task humaneval \\\n            --batch_size 8 \\\n            --backend hf \\\n\n\nInference with vLLM engine\n--------------------------\n\nWe recommend using vLLM engine for fast inference. \nvLLM supported tensor parallel, data parallel or combination of both.\nReference to vLLM documenation for more detail. \n\nTo use ``code-eval`` with vLLM engine, please refer to vLLM engine documents to `instal it \u003chttps://docs.vllm.ai/en/latest/getting_started/installation.html\u003e`_.\n\n.. note:: \n    \n    You can install vLLM using pip:\n\n    .. code-block:: console\n\n        $ pip install vllm\n\n\nWith model supported by vLLM (See more: `vLLM supported model \u003chttps://docs.vllm.ai/en/latest/models/supported_models.html\u003e`_) \nrun:\n\n.. code-block:: console\n\n    $ code-eval --model_name microsoft/phi-1 \\\n        --task humaneval \\\n        --batch_size 8 \\\n        --backend vllm\n\n.. tip::\n\n    You can use LoRA with similar syntax.\n\n    .. code-block:: console\n\n        $ code-eval --model_name microsoft/phi-1 \\\n            --peft_model \u003cadapters-name\u003e \\\n            --task humaneval \\\n            --batch_size 8 \\\n            --backend vllm \\\n\nCite as\n=======\n\n.. code-block:: \n\n    @misc{code-eval,\n        author       = {Dung Nguyen Manh},\n        title        = {A framework for easily evaluation code generation model},\n        month        = 3,\n        year         = 2024,\n        publisher    = {github},\n        version      = {v0.0.1},\n        url          = {https://github.com/FSoft-AI4Code/code-llm-evaluator}\n    }\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffsoft-ai4code%2Fcode-llm-evaluator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffsoft-ai4code%2Fcode-llm-evaluator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffsoft-ai4code%2Fcode-llm-evaluator/lists"}