{"id":20723564,"url":"https://github.com/networks-learning/prediction-powered-ranking","last_synced_at":"2025-04-23T17:27:31.623Z","repository":{"id":224740963,"uuid":"764093544","full_name":"Networks-Learning/prediction-powered-ranking","owner":"Networks-Learning","description":"Code for \"Prediction-Powered Ranking of Large Language Models\", NeurIPS 2024.","archived":false,"fork":false,"pushed_at":"2024-10-28T10:35:13.000Z","size":4973,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T01:51:45.655Z","etag":null,"topics":["llm-eval","llm-evaluation","llm-evaluation-framework","prediction-powered-inference","rank-sets","ranking-algorithm"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Networks-Learning.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-02-27T13:21:18.000Z","updated_at":"2024-12-17T06:24:13.000Z","dependencies_parsed_at":"2024-03-15T16:27:37.172Z","dependency_job_id":"d1ed989a-af0f-42d9-a22f-738a7a6be5cb","html_url":"https://github.com/Networks-Learning/prediction-powered-ranking","commit_stats":null,"previous_names":["networks-learning/prediction-powered-ranking"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Networks-Learning%2Fprediction-powered-ranking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Networks-Learning%2Fprediction-powered-ranking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Networks-Learning%2Fprediction-powered-ranking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Networks-Learning%2Fprediction-powered-ranking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Networks-Learning","download_url":"https://codeload.github.com/Networks-Learning/prediction-powered-ranking/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250479613,"owners_count":21437395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm-eval","llm-evaluation","llm-evaluation-framework","prediction-powered-inference","rank-sets","ranking-algorithm"],"created_at":"2024-11-17T04:09:07.507Z","updated_at":"2025-04-23T17:27:31.606Z","avatar_url":"https://github.com/Networks-Learning.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prediction-Powered Ranking\n\nThis repository contains the code for the paper [Prediction-Powered Ranking of Large Language Models](https://arxiv.org/abs/2402.17826) published in NeurIPS (2024) by Ivi Chatzi, Eleni Straitouri, Suhas Thejaswi and Manuel Gomez-Rodriguez.\n\n__Contents__:\n- [Dependencies](#dependencies)\n- [Usage](#usage)\n- [Repository structure](#repository-structure)\n- [Contact \u0026 attribution](#contact--attribution)\n\n## Dependencies\n\nAll the code is written in Python 3.11.2\\\nIn order to create a virtual environment and install the project dependencies you can run the following commands:\n\n```commandline\npython3 -m venv env\nsource env/bin/activate\npip install -r requirements.txt\n```\n\nIn addition to the above dependencies, to run the notebooks and produce the figures it is necessary to install the additional requirements from [notebooks/requirements.txt](notebooks/requirements.txt).\n\n## Usage\nTo reproduce the main experiments of the paper (Section 5 and Appendix D), run:\n```\n./scripts/llm-ranking.sh\n```\n\nTo reproduce the synthetic experiments in Appendix E of the paper, run:\n```\n./scripts/synthetic.sh\n```\n\nTo create the figures, run the notebooks in [notebooks](notebooks/).\n\n\u003c!-- \n\n### Plots\nFirst run the experiments via [llm-ranking.py](scripts/llm-ranking.py) using [config.json](config.json).\n\nThen, install the plot code requirements:\n```commandline\npip install -r plots/plot_requirements.txt\n```\nThen, run:\n```commandline\npython3 plots/create_plots.py\n```\n\nFigures 3, 4, 9 and 10 are stored in folder [plots/ranksets](plots/ranksets).\\\nFigures 1, 2, 6, 7 and 8 are stored in folder [plots/intersect_size](plots/intersect_size). --\u003e\n\n## Repository structure\n\n```\n├── data\n│   ├── human.json\n│   ├── gpt-4-0125-preview.json\n│   ├── claude-3-opus-20240229.json\n│   └── gpt-3.5-turbo.json\n├── figures\n├── notebooks\n├── outputs\n├── scripts\n│   ├── llm-ranking.sh\n│   ├── synthetic.sh\n│   └── config.json\n└── src\n    ├── data_process.py\n    ├── estimate.py\n    ├── llm-ranking.py\n    ├── plot_utils.py\n    ├── ranksets.py\n    ├── run_experiments.py\n    └── synthetic.py\n\n```\n\nThe folder [data](data/) contains the datasets used for our experimentation:\n- [human.json](data/human.json): pairwise comparisons by humans.\n- [gpt-4-0125-preview.json](data/gpt-4-0125-preview.json): pairwise comparisons by GPT 4.\n- [claude-3-opus-20240229.json](data/claude-3-opus-20240229.json): pairwise comparisons by Claude 3.\n- [gpt-3.5-turbo.json](data/gpt-3.5-turbo.json): pairwise comparisons by GPT 3.5.\n\nThe folder [figures](figures/) contains all the figures presented in the paper.\n\nThe folder [notebooks](notebooks/) contains python notebooks that generate all the figures included in the paper. \n\nThe folder [outputs](outputs/) contains the output files produced by the experiments' scripts.\n\nThe folder [scripts](scripts/) contains bash scripts used to run all the experiments presented in the paper:\n- [llm-ranking.sh](scripts/llm-ranking.sh): runs the main experiments of the paper, in Section 5 and Appendix D.\n- [synthetic.sh](scripts/synthetic.sh): runs the synthetic experiments in Appendix E of the paper.\n- [config.json](scripts/config.json): configuration file with parameters for the experiments ran by [llm-ranking.sh](scripts/llm-ranking.sh).\n\nThe folder [src](src/) contains all the code necessary to reproduce the results in the paper. Specifically:\n- [data_process.py](src/data_process.py): inputs and subsamples from datasets.\n- [estimate.py](src/estimate.py): implements Algorithms 1,3,4 from the paper to compute $\\hat{\\theta}$ and $\\widehat{\\Sigma}$.\n- [llm-ranking.py](src/llm-ranking.py): reads config file and runs the experiments in Section 5 and Appendix D of the paper.\n- [plot_utils.py](src/plot_utils.py): contains auxiliary functions for plotting.\n- [ranksets.py](src/ranksets.py): implements Algorithm 2 from the paper to construct rank-sets.\n- [run_experiments.py](src/run_experiments.py): runs experiments for all input parameters.\n- [synthetic.py](src/synthetic.py): generates synthetic data and runs the synthetic experiments in Appendix E of the paper.\n\n\n## Contact \u0026 attribution\n\nIn case you have questions about the code, you identify potential bugs or you would like us to include additional functionalities, feel free to open an issue or contact [Ivi Chatzi](mailto:ichatzi@mpi-sws.org).\n\nIf you use parts of the code in this repository for your own research purposes, please consider citing:\n\n    @inproceedings{chatzi2024prediction,\n      title={Prediction-Powered Ranking of Large Language Models},\n      author={Ivi Chatzi and Eleni Straitouri and Suhas Thejaswi and Manuel Gomez Rodriguez},\n      year={2024},\n      booktitle = {Advances in Neural Information Processing Systems},\n      publisher = {Curran Associates, Inc.}\n      }\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetworks-learning%2Fprediction-powered-ranking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnetworks-learning%2Fprediction-powered-ranking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetworks-learning%2Fprediction-powered-ranking/lists"}