{"id":13932118,"url":"https://github.com/OpenGenerativeAI/llm-colosseum","last_synced_at":"2025-07-19T16:30:56.507Z","repository":{"id":229488243,"uuid":"776301107","full_name":"OpenGenerativeAI/llm-colosseum","owner":"OpenGenerativeAI","description":"Benchmark LLMs by fighting in Street Fighter 3! The new way to evaluate the quality of an LLM","archived":false,"fork":false,"pushed_at":"2024-11-20T11:12:52.000Z","size":48383,"stargazers_count":1333,"open_issues_count":15,"forks_count":161,"subscribers_count":20,"default_branch":"main","last_synced_at":"2024-11-24T20:50:28.412Z","etag":null,"topics":["benchmark","genai","llm","streetfighterai"],"latest_commit_sha":null,"homepage":"https://huggingface.co/spaces/junior-labs/llm-colosseum","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenGenerativeAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-23T05:52:20.000Z","updated_at":"2024-11-23T21:31:29.000Z","dependencies_parsed_at":"2024-04-15T11:27:48.134Z","dependency_job_id":"8e0f7e18-782a-4219-9052-9164cf5ca89d","html_url":"https://github.com/OpenGenerativeAI/llm-colosseum","commit_stats":null,"previous_names":["opengenerativeai/llm-colosseum"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGenerativeAI%2Fllm-colosseum","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGenerativeAI%2Fllm-colosseum/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGenerativeAI%2Fllm-colosseum/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGenerativeAI%2Fllm-colosseum/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenGenerativeAI","download_url":"https://codeload.github.com/OpenGenerativeAI/llm-colosseum/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226635584,"owners_count":17662050,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","genai","llm","streetfighterai"],"created_at":"2024-08-07T21:00:49.328Z","updated_at":"2025-07-19T16:30:56.498Z","avatar_url":"https://github.com/OpenGenerativeAI.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook","Benchmarks","2024/03","Benchmark","Content","NLP","Repos"],"sub_categories":["Open Models","Action Games","3. Pretraining"],"readme":"# Evaluate LLMs in real time with Street Fighter III\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./logo.png\" alt=\"colosseum-logo\" width=\"30%\"  style=\"border-radius: 50%; padding-bottom: 20px\"/\u003e\n\u003c/div\u003e\n\nMake LLM fight each other in real time in Street Fighter III.\n\nWhich LLM will be the best fighter ?\n\n**Demo:** [Try it in your browser right here!](https://llm-colosseum.phospho.ai)\n\n## Our criterias 🔥\n\nThey need to be:\n\n- **Fast**: It is a real time game, fast decisions are key\n- **Smart**: A good fighter thinks 50 moves ahead\n- **Out of the box thinking**: Outsmart your opponent with unexpected moves\n- **Adaptable**: Learn from your mistakes and adapt your strategy\n- **Resilient**: Keep your RPS high for an entire game\n\n## Let the fight begin 🥷\n\n### 1 VS 1: Mistral 7B vs Mistral 7B\n\nhttps://github.com/OpenGenerativeAI/llm-colosseum/assets/19614572/79b58e26-7902-4687-af5d-0e1e845ecaf8\n\n### 1 VS 1 X 6 : Mistral 7B vs Mistral 7B\n\nhttps://github.com/OpenGenerativeAI/llm-colosseum/assets/19614572/5d3d386b-150a-48a5-8f68-7e2954ec18db\n\n## A new kind of benchmark ?\n\nStreet Fighter III assesses the ability of LLMs to understand their environment and take actions based on a specific context.\nAs opposed to RL models, which blindly take actions based on the reward function, LLMs are fully aware of the context and act accordingly.\n\n# Results\n\nOur experimentations (546 fights so far) led to the following leaderboard.\nEach LLM has an ELO score based on its results.\n\n## Ranking\n\n[Huggingface ranking](https://huggingface.co/spaces/junior-labs/llm-colosseum)\n\n### ELO ranking\n\n| Rank | Model                                                              |  Rating |\n| ---: | :----------------------------------------------------------------- | ------: |\n|    1 | 🥇openai:gpt-4o:text                                               |  1912.5 |\n|    2 | 🥈**openai:gpt-4o-mini:vision**                                    | 1835.27 |\n|    3 | 🥉openai:gpt-4o-mini:text                                          | 1670.89 |\n|    4 | **openai:gpt-4o:vision**                                           | 1656.93 |\n|    5 | **mistral:pixtral-large-latest:vision**                            | 1654.61 |\n|    6 | **mistral:pixtral-12b-2409:vision**                                | 1590.77 |\n|    7 | mistral:pixtral-12b-2409:text                                      | 1569.03 |\n|    8 | together:meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo:text       | 1441.45 |\n|    9 | **anthropic:claude-3-haiku-20240307:vision**                       | 1364.87 |\n|   10 | mistral:pixtral-large-latest:text                                  | 1356.32 |\n|   11 | anthropic:claude-3-haiku-20240307:text                             |  1333.6 |\n|   12 | **anthropic:claude-3-sonnet-20240229:vision**                      | 1314.61 |\n|   13 | **together:meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo:vision** | 1269.84 |\n|   14 | anthropic:claude-3-sonnet-20240229:text                            | 1029.31 |\n\n*Note: In our experiments, Claude 3 Sonnet got a low score due to many refusal to fight and large API latencies.*\n\n### Win rate matrix\n\n![Win rate matrix](notebooks/result_matrix.png)\n\n# Explanation\n\nEach player can be controlled by a text generating model or a multimodal model. We call them through API endpoints. Learn more about models:\n- [Text generating models](https://huggingface.co/docs/transformers/en/llm_tutorial)\n- [**Multimodal LLM** models](https://huggingface.co/blog/vlms)\n\n### TextRobot\n\nWe send to the LLM a text description of the screen. The LLM decide on the next moves its character will make. The next moves depends on its previous moves, the moves of its opponents, its power and health bars.\n\n- Agent based\n- Multithreading\n- Real time\n\n  ![fight3 drawio](https://github.com/OpenGenerativeAI/llm-colosseum/assets/78322686/3a212601-f54c-490d-aeb9-6f7c2401ebe6)\n\n### VisionRobot\n\nWe send to the LLM a screenshot of the current state of the game precising which character he is controlling. His decision is only based on this visual information.\n\n# Installation\n\n- Follow instructions in https://docs.diambra.ai/#installation\n- Download the ROM and put it in `~/.diambra/roms` (no need to dezip the content)\n- (Optional) Create and activate a [new python venv](https://docs.python.org/3/library/venv.html)\n- Install dependencies with `make install` or `pip install -r requirements.txt`\n- Create a `.env` file and fill it with the content like in the `.env.example` file\n- Run with `make run`\n\n## Running with Docker\n\nYou can also run the application using Docker.\n\n### Building the Docker Image\n\nTo build the Docker image, use the following command:\n\n```bash\ndocker build -t diambra-app .\n```\n\n### Running the Docker Container\n\nTo run the Docker container, use the following command:\n\n```bash\ndocker run --name diambra-container -v ~/.diambra/roms:/app/roms diambra-app\n```\n\n- If you encounter a conflict with an existing container name, you can remove the existing container with:\n\n```bash\ndocker rm diambra-container\n```\n\n### Running with Docker Compose on Ollama locally\n\nTo start the services, use the following command:\n\n```bash\ndocker-compose up\n```\n\n### Stopping the Services\n\nTo stop the services, use:\n\n```bash\ndocker-compose down\n```\n\n## Test mode\n\nTo disable the LLM calls, set `DISABLE_LLM` to `True` in the `.env` file.\nIt will choose the actions randomly.\n\n## Logging\n\nChange the logging level in the `script.py` file.\n\n## Local model\n\nYou can run the arena with local models using [Ollama](https://ollama.com/).\n\n1. Make sure you have ollama installed, running, and with a model downloaded (run `ollama serve mistral` in the terminal for example)\n\n2. Run `make local` to start the fight.\n\nBy default, it runs mistral against mistral. To use other models, you need to change the parameter model in `local.py`.\n\n```python\nfrom eval.game import Game, Player1, Player2\n\ndef main():\n    # Environment Settings\n\n    game = Game(\n        render=True,\n        save_game=True,\n        player_1=Player1(\n            nickname=\"Baby\",\n            model=\"ollama:mistral\",\n            robot_type=\"text\",  # vision or text\n            temperature=0.7,\n        ),\n        player_2=Player2(\n            nickname=\"Daddy\",\n            model=\"ollama:mistral\",\n            robot_type=\"text\",\n            temperature=0.7,\n        ),\n    )\n\n    game.run()\n    return 0\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\nThe convention we use is `model_provider:model_name`. If you want to use another local model than Mistral, you can do `ollama:some_other_model`\n\n## How to make my own LLM model play? Can I improve the prompts?\n\nThe LLM is called in `\u003cText||Vision\u003eRobot.call_llm()` method of the `agent/robot.py` file.\n\n#### TextRobot method:\n\n```python\n    def call_llm(\n        self,\n        max_tokens: int = 50,\n        top_p: float = 1.0,\n    ) -\u003e Generator[ChatResponse, None, None]:\n        \"\"\"\n        Make an API call to the language model.\n\n        Edit this method to change the behavior of the robot!\n        \"\"\"\n\n        # Generate the prompts\n        move_list = \"- \" + \"\\n - \".join([move for move in META_INSTRUCTIONS])\n        system_prompt = f\"\"\"You are the best and most aggressive Street Fighter III 3rd strike player in the world.\nYour character is {self.character}. Your goal is to beat the other opponent. You respond with a bullet point list of moves.\n{self.context_prompt()}\nThe moves you can use are:\n{move_list}\n----\nReply with a bullet point list of moves. The format should be: `- \u003cname of the move\u003e` separated by a new line.\nExample if the opponent is close:\n- Move closer\n- Medium Punch\n\nExample if the opponent is far:\n- Fireball\n- Move closer\"\"\"\n\n        start_time = time.time()\n\n        client = get_client(self.model, temperature=self.temperature)\n\n        messages = [\n            ChatMessage(role=\"system\", content=system_prompt),\n            ChatMessage(role=\"user\", content=\"Your next moves are:\"),\n        ]\n        resp = client.stream_chat(messages)\n\n        logger.debug(f\"LLM call to {self.model}: {system_prompt}\")\n        logger.debug(f\"LLM call to {self.model}: {time.time() - start_time}s\")\n\n        return resp\n```\n\n#### VisionRobot method:\n\n```python\ndef call_llm(\n        self,\n        max_tokens: int = 50,\n        top_p: float = 1.0,\n    ) -\u003e Generator[CompletionResponse, None, None]:\n        \"\"\"\n        Make an API call to the language model.\n\n        Edit this method to change the behavior of the robot!\n        \"\"\"\n\n        # Generate the prompts\n        move_list = \"- \" + \"\\n - \".join([move for move in META_INSTRUCTIONS])\n        system_prompt = f\"\"\"You are the best and most aggressive Street Fighter III 3rd strike player in the world.\nYour character is {self.character}. Your goal is to beat the other opponent. You respond with a bullet point list of moves.\n\nThe current state of the game is given in the following image.\n\nThe moves you can use are:\n{move_list}\n----\nReply with a bullet point list of 3 moves. The format should be: `- \u003cname of the move\u003e` separated by a new line.\nExample if the opponent is close:\n- Move closer\n- Medium Punch\n\nExample if the opponent is far:\n- Fireball\n- Move closer\"\"\"\n\n        start_time = time.time()\n\n        client = get_client_multimodal(\n            self.model, temperature=self.temperature\n        )  # MultiModalLLM\n\n        resp = client.stream_complete(\n            prompt=system_prompt, image_documents=[self.last_image_to_image_node()]\n        )\n\n        logger.debug(f\"LLM call to {self.model}: {system_prompt}\")\n        logger.debug(f\"LLM call to {self.model}: {time.time() - start_time}s\")\n\n        return resp\n```\n\nYou can personnalise your prompt in these functions.\n\n### Submit your model\n\nCreate a new class herited from Robot that has the changes you want to make and open a PR.\n\nWe'll do our best to add it to the ranking!\n\n# Credits\n\nMade with ❤️ by the OpenGenerativeAI team from [phospho](https://phospho.ai) (@oulianov @Pierre-LouisBJT @Platinn) and [Quivr](https://www.quivr.app) (@StanGirard) during Mistral Hackathon 2024 in San Francisco\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenGenerativeAI%2Fllm-colosseum","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOpenGenerativeAI%2Fllm-colosseum","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenGenerativeAI%2Fllm-colosseum/lists"}