{"id":20779457,"url":"https://github.com/freedomintelligence/llmarena","last_synced_at":"2025-06-16T04:33:56.802Z","repository":{"id":236625774,"uuid":"632049232","full_name":"FreedomIntelligence/LLMArena","owner":"FreedomIntelligence","description":"LLMArena is an open-source platform designed to facilitate AI competitions between large language models.","archived":false,"fork":false,"pushed_at":"2023-08-08T01:55:24.000Z","size":9,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-23T18:11:14.374Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-24T15:49:30.000Z","updated_at":"2025-03-23T05:57:52.000Z","dependencies_parsed_at":"2024-04-28T04:31:03.416Z","dependency_job_id":"4f8b3229-0ec4-4fab-b929-3358dcdd99c2","html_url":"https://github.com/FreedomIntelligence/LLMArena","commit_stats":null,"previous_names":["freedomintelligence/llmarena"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FLLMArena","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FLLMArena/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FLLMArena/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FLLMArena/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/LLMArena/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248064263,"owners_count":21041854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T13:28:02.209Z","updated_at":"2025-04-09T15:52:10.216Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLMArena\n\n-------\nLLMArena is an open-source platform designed to facilitate AI competitions between large language models. The name LLMArena stands for \"Large Language Model Arena\", and it reflects our mission to create a level playing field where different language models can compete and showcase their capabilities.\n\nWe created LLMArena because we believe that AI competitions are a great way to advance state of art in natural language processing. By pitting different models against each other and evaluating their performance using standardized metrics, we can identify strengths and weaknesses in each model, and use that knowledge to improve them.\n\nIn LLMArena, we use GPT-4/3.5 as the referee to evaluate the performance of two competing models. GPT-4 is a state-of-the-art language model developed by OpenAI, and it's well-suited for the task of evaluating the quality of generated text.\n\nTo participate in an LLMArena competition, a language model needs to implement the Competitor interface, which consists of a single method called answer_question(). This method takes a natural language question as input and returns a natural language answer.\n\nLLMArena competitions are managed by the Arena class, which reads a set of questions from a JSON file, calls the answer_question() method of each competitor for each question, and passes the answers to the referee for evaluation. The referee then determines the winner based on the scores assigned to each model.\n\nWe hope that LLMArena will become a popular platform for evaluating and comparing different language models, and that it will help drive advances in the field of natural language processing.\n\n---------\n# Requirements\n\n- Python 3.x \n- retrying==1.3.3 \n- statistics==1.0.3\n\nUsage\nTo use this project, follow these steps:\n\nInstall the required packages by running the following command:\n\n```commandline\npip install -r requirements.txt\n```\nCreate a new competition by running the following command:\n\n```python\npython demo_main.py\n```\n\nThis will start a competition between two competitors and print the final results.\n\n(Optional) If you want to customize the competition, you can modify the demo_main.py file to change the competitors, the referee, or the questions.\n\n**Competitor**\n\nA competitor is a class that inherits from the Competitor base class. Each competitor must implement the answer_question() method, which takes a question as input and returns an answer.\n\n**Referee**\n\nThe referee is a class that inherits from the Referee base class. The referee reads the answers of the two competitors, evaluates them, and determines the winner.\n\n**Arena**\n\nThe arena is a class that manages the competition between the two competitors. It reads the questions from a file, calls the answer_question() method of each competitor, and passes the answers to the referee for evaluation.\n\n**Question File**\n\nThe question file is a JSONL file that contains a list of questions. Each question should have an instruction field, which contains the question text.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fllmarena","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fllmarena","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fllmarena/lists"}