{"id":13627993,"url":"https://github.com/lm-sys/vicuna-blog-eval","last_synced_at":"2025-04-17T00:32:58.669Z","repository":{"id":177197911,"uuid":"660051210","full_name":"lm-sys/vicuna-blog-eval","owner":"lm-sys","description":"The code and data for the GPT-4 based benchmark in the vicuna blog post","archived":true,"fork":false,"pushed_at":"2023-08-02T02:32:40.000Z","size":712,"stargazers_count":32,"open_issues_count":0,"forks_count":8,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-01T22:41:44.226Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lm-sys.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-06-29T06:22:19.000Z","updated_at":"2024-08-01T22:41:44.227Z","dependencies_parsed_at":null,"dependency_job_id":"f4135543-e21f-4087-8d69-4a8d122fcbe4","html_url":"https://github.com/lm-sys/vicuna-blog-eval","commit_stats":null,"previous_names":["lm-sys/vicuna-blog-eval"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lm-sys%2Fvicuna-blog-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lm-sys%2Fvicuna-blog-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lm-sys%2Fvicuna-blog-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lm-sys%2Fvicuna-blog-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lm-sys","download_url":"https://codeload.github.com/lm-sys/vicuna-blog-eval/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223735222,"owners_count":17194065,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T22:00:42.370Z","updated_at":"2024-11-08T18:31:14.503Z","avatar_url":"https://github.com/lm-sys.png","language":"Python","funding_links":[],"categories":["Evaluation Datasets"],"sub_categories":["General"],"readme":"This repo is an archive of the code and data used in the [vicuna blog post](https://lmsys.org/blog/2023-03-30-vicuna/).\n\n**This repo is deprecated and we recommend using our new question set and evaluation pipeline at [fastchat.llm_judge](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge).**\n\nWe do not recommend using this repo because its questions are relatively easy and it does not address limitations of GPT-4 based evaluation such as [position bias](https://arxiv.org/abs/2306.05685).\n\n---------------\n\n## Evaluation\n\nOur AI-enhanced evaluation pipeline is based on GPT-4. This section provides a high-level summary of the pipeline. For detailed instructions, please refer to the [evaluation](eval) documentation.\n\n### Pipeline Steps\n\n1. Generate answers from different models: Use `qa_baseline_gpt35.py` for ChatGPT, or specify the model checkpoint and run `get_model_answer.py` for Vicuna and other models.\n\n2. Generate reviews with GPT-4: Use GPT-4 to generate reviews automatically. This step can also be performed manually if the GPT-4 API is not available to you.\n\n3. Generate visualization data: Run `generate_webpage_data_from_table.py` to generate data for a static website, which allows you to visualize the evaluation data.\n\n4. Visualize the data: Serve a static website under the `webpage` directory. You can use `python3 -m http.server` to serve the website locally.\n\n### Data Format and Contribution\n\nWe use a data format encoded with JSON Lines for evaluation. The format includes information on models, prompts, reviewers, questions, answers, and reviews.\n\nYou can customize the evaluation process or contribute to our project by accessing the relevant [data](eval/table/).\n\nFor detailed instructions, please refer to the [evaluation](eval) documentation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flm-sys%2Fvicuna-blog-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flm-sys%2Fvicuna-blog-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flm-sys%2Fvicuna-blog-eval/lists"}