{"id":23932662,"url":"https://github.com/balrog-ai/BALROG","last_synced_at":"2025-09-11T15:32:04.025Z","repository":{"id":263904503,"uuid":"891746957","full_name":"balrog-ai/BALROG","owner":"balrog-ai","description":"Benchmarking Agentic LLM and VLM Reasoning On Games","archived":false,"fork":false,"pushed_at":"2025-01-03T12:33:58.000Z","size":1555,"stargazers_count":83,"open_issues_count":3,"forks_count":12,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-03T13:28:08.176Z","etag":null,"topics":["agentic","benchmark","games","llm","reasoning","vlm"],"latest_commit_sha":null,"homepage":"https://balrogai.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/balrog-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-20T22:08:12.000Z","updated_at":"2025-01-03T12:34:01.000Z","dependencies_parsed_at":"2024-11-20T23:25:34.574Z","dependency_job_id":"c1582634-0548-47c5-bb11-a3cc4344f676","html_url":"https://github.com/balrog-ai/BALROG","commit_stats":null,"previous_names":["balrog-ai/balrog"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balrog-ai%2FBALROG","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balrog-ai%2FBALROG/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balrog-ai%2FBALROG/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balrog-ai%2FBALROG/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/balrog-ai","download_url":"https://codeload.github.com/balrog-ai/BALROG/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232657377,"owners_count":18556830,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic","benchmark","games","llm","reasoning","vlm"],"created_at":"2025-01-06T00:29:16.121Z","updated_at":"2025-09-11T15:32:04.012Z","avatar_url":"https://github.com/balrog-ai.png","language":"Python","funding_links":[],"categories":["Building","Tools"],"sub_categories":["Benchmarks","LLM Evaluations and Benchmarks"],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://balrogai.com\"\u003e\n    \u003cimg src=\"docs/imgs/balrog_banner.png\" width=\"50%\" alt=\"BALROG Agent\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n# BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games\n\nBALROG is a novel benchmark evaluating agentic LLM and VLM capabilities on long-horizon interactive tasks using reinforcement learning environments. Check out how current models fare on our [leaderboard](https://balrogai.com). You can read more about BALROG in our [paper](https://arxiv.org/abs/2411.13543).\n\n## Features\n\n- Comprehensive evaluation of agentic abilities\n- Support for both language and vision-language models\n- Integration with popular AI APIs and local deployment\n- Easy integration for custom agents, new environments and new models\n\n## Installation\n\nWe advise using conda for the installation\n\n```bash\nconda create -n balrog python=3.10 -y\nconda activate balrog\n\ngit clone https://github.com/balrog-ai/BALROG.git\ncd BALROG\npip install -e .\nbalrog-post-install\n```\n\nOn Mac make sure you have `wget` installed for the `balrog-post-install`\n\n## Docker\n\nWe have provided some docker images. Please see the [relevant README](docker/README.md).\n\n## ⚡️ Evaluate using vLLM locally\n\nWe support running LLMs/VLMs locally using [vLLM](https://github.com/vllm-project/vllm). You can spin up a vLLM client and evaluate your agent on BALROG in the following way:\n\n```bash\npip install vllm numpy==1.23\nvllm serve meta-llama/Llama-3.2-1B-Instruct --port 8080\n\npython eval.py \\\n  agent.type=naive \\\n  agent.max_image_history=0 \\\n  agent.max_text_history=16 \\\n  eval.num_workers=32 \\\n  client.client_name=vllm \\\n  client.model_id=meta-llama/Llama-3.2-1B-Instruct \\\n  client.base_url=http://0.0.0.0:8080/v1\n```\n\nOn Mac you might have to first export the following to suppress some fork() errors:\n\n```\nexport OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES\n```\n\nCheck out [vLLM](https://github.com/vllm-project/vllm) for more options on how to serve your models fast and efficiently.\n\n## 🛜 Evaluate using API\n\nWe support how of the box clients for OpenAI, Anthropic and Google Gemini APIs. If you want to evaluate an agent using one of these APIs, you first have to set up your API key in one of two ways:\n\nYou can either directly export it:\n\n```bash\nexport OPENAI_API_KEY=\u003cKEY\u003e\nexport ANTHROPIC_API_KEY=\u003cKEY\u003e\nexport GEMINI_API_KEY=\u003cKEY\u003e\n```\n\nOr you can modify the `SECRETS` file, adding your api keys.\n\nYou can then run the evaluation with:\n\n```bash\npython eval.py \\\n  agent.type=naive \\\n  agent.max_image_history=0 \\\n  agent.max_text_history=16 \\\n  eval.num_workers=16 \\\n  client.client_name=openai \\\n  client.model_id=gpt-4o-mini-2024-07-18\n```\n\n## Documentation\n\n- [Evaluation Guide](https://github.com/balrog-ai/BALROG/blob/main/docs/evaluation.md) - Detailed instructions for various evaluation scenarios\n- [Agent Development](https://github.com/balrog-ai/BALROG/blob/main/docs/agents.md) - Tutorial on creating custom agents\n- [Few Shot Learning](https://github.com/balrog-ai/BALROG/blob/main/docs/few_shot_learning.md) - Instructions on how to run Few Shot Learning\n\nWe welcome contributions! Please see our [Contributing Guidelines](https://github.com/balrog-ai/BALROG/blob/main/docs/contribution.md) for details.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Citation\n\nIf you use BALROG in any of your work, please cite:\n\n```\n@article{paglieri2024balrog,\n  title={Benchmarking Agentic LLM and VLM Reasoning On Games},\n  author={Paglieri, Davide and Cupia{\\l}, Bart{\\l}omiej and Coward, Sam and Piterbarg, Ulyana and Wo{\\l}czyk, Maciej and Khan, Akbir and Pignatelli, Eduardo and Kuci{\\'n}ski, {\\L}ukasz and Pinto, Lerrel and Fergus, Rob and Foerster, Jakob Nicolaus and Parker-Holder, Jack and Rockt{\\\"a}schel, Tim},\n  journal={arXiv preprint arXiv:2411.13543},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbalrog-ai%2FBALROG","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbalrog-ai%2FBALROG","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbalrog-ai%2FBALROG/lists"}