{"id":23726594,"url":"https://github.com/camel-ai/crab","last_synced_at":"2025-05-15T15:01:56.522Z","repository":{"id":239584336,"uuid":"799215963","full_name":"camel-ai/crab","owner":"camel-ai","description":"🦀️ CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents. https://crab.camel-ai.org/","archived":false,"fork":false,"pushed_at":"2025-05-05T09:02:47.000Z","size":8269,"stargazers_count":338,"open_issues_count":34,"forks_count":49,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-10T10:43:56.964Z","etag":null,"topics":["gui-automation","language-model-agent","large-language-models","multi-agent-systems","visual-language-models"],"latest_commit_sha":null,"homepage":"https://crab.camel-ai.org/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/camel-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"licenses/LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-11T13:38:52.000Z","updated_at":"2025-05-07T05:03:18.000Z","dependencies_parsed_at":"2024-05-13T13:32:47.356Z","dependency_job_id":"4443f96b-a1fa-4928-907b-c1289b8ff1b7","html_url":"https://github.com/camel-ai/crab","commit_stats":null,"previous_names":["camel-ai/crab"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/camel-ai%2Fcrab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/camel-ai%2Fcrab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/camel-ai%2Fcrab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/camel-ai%2Fcrab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/camel-ai","download_url":"https://codeload.github.com/camel-ai/crab/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253404957,"owners_count":21903109,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gui-automation","language-model-agent","large-language-models","multi-agent-systems","visual-language-models"],"created_at":"2024-12-31T00:32:23.168Z","updated_at":"2025-05-15T15:01:56.408Z","avatar_url":"https://github.com/camel-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🦀 CRAB: Cross-platform Agent Benchmark for Multimodal Embodied Language Model Agents\r\n\r\n[![arXiv][arxiv-image]][arxiv-url]\r\n[![Slack][slack-image]][slack-url]\r\n[![Discord][discord-image]][discord-url]\r\n[![Wechat][wechat-image]][wechat-url]\r\n[![Twitter][twitter-image]][twitter-url]\r\n\r\n\u003cp align=\"center\"\u003e\r\n  \u003ca href=\"https://camel-ai.github.io/crab/\"\u003eDocumentation\u003c/a\u003e |\r\n  \u003ca href=\"https://crab.camel-ai.org/\"\u003eWebsite \u0026 Demos\u003c/a\u003e |\r\n  \u003ca href=\"https://www.camel-ai.org/post/crab\"\u003eBlog\u003c/a\u003e |\r\n  \u003ca href=\"https://dandansamax.github.io/posts/crab-paper/\"\u003eChinese Blog\u003c/a\u003e |\r\n  \u003ca href=\"https://www.camel-ai.org/\"\u003eCAMEL-AI\u003c/a\u003e\r\n\u003c/p\u003e\r\n\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg src='https://raw.githubusercontent.com/camel-ai/crab/main/assets/CRAB_logo1.png' width=800\u003e\r\n\u003c/p\u003e\r\n\r\n## Overview\r\n\r\nCRAB is a framework for building LLM agent benchmark environments in a Python-centric way.\r\n\r\n#### Key Features\r\n\r\n🌐 Cross-platform and Multi-environment\r\n* Create build agent environments that support various deployment options including in-memory, Docker-hosted, virtual machines, or distributed physical machines, provided they are accessible via Python functions.\r\n* Let the agent access all the environments in the same time through a unified interface.\r\n\r\n⚙ ️Easy-to-use Configuration\r\n* Add a new action by simply adding a `@action` decorator on a Python function.\r\n* Define the environment by integrating several actions together.\r\n\r\n📐 Novel Benchmarking Suite\r\n* Define tasks and the corresponding evaluators in an intuitive Python-native way.\r\n* Introduce a novel graph evaluator method providing fine-grained metrics.\r\n\r\n## Installation\r\n\r\n#### Prerequisites\r\n\r\n- Python 3.10 or newer\r\n\r\n```bash\r\npip install crab-framework[client]\r\n```\r\n\r\n## Experiment on CRAB-Benchmark-v0\r\n\r\nAll datasets and experiment code are in [crab-benchmark-v0](./crab-benchmark-v0/) directory. Please carefully read the [benchmark tutorial](./crab-benchmark-v0/README.md) before using our benchmark.\r\n\r\n## Examples\r\n\r\n#### Run template environment with openai agent\r\n\r\n```bash\r\nexport OPENAI_API_KEY=\u003cyour api key\u003e\r\npython examples/single_env.py\r\npython examples/multi_env.py\r\n```\r\n\r\n## Demo Video\r\n\r\n[![demo_video](https://i.ytimg.com/vi_webp/PNqrHNQlU6I/maxresdefault.webp)](https://www.youtube.com/watch?v=PNqrHNQlU6I\u0026ab_channel=CamelAI)\r\n\r\n## Cite\r\nPlease cite [our paper](https://arxiv.org/abs/2407.01511) if you use anything related in your work:\r\n```\r\n@misc{xu2024crab,\r\n      title={CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents}, \r\n      author={Tianqi Xu and Linyao Chen and Dai-Jie Wu and Yanjun Chen and Zecheng Zhang and Xiang Yao and Zhiqiang Xie and Yongchao Chen and Shilong Liu and Bochen Qian and Philip Torr and Bernard Ghanem and Guohao Li},\r\n      year={2024},\r\n      eprint={2407.01511},\r\n      archivePrefix={arXiv},\r\n      primaryClass={cs.AI},\r\n      url={https://arxiv.org/abs/2407.01511}, \r\n}\r\n```\r\n\r\n## Community\r\nJoin us ([*Discord*](https://discord.camel-ai.org/) or [*WeChat*](https://ghli.org/camel/wechat.png)) in pushing the boundaries of finding the scaling laws of agents. \r\n\r\n- **WeChat Community:** Scan the QR code below to join our WeChat community.\r\n\r\n  \u003cdiv align=\"center\"\u003e\r\n    \u003cimg src=\"assets/wechatgroup.jpeg\" alt=\"WeChat QR Code\"\u003e\r\n  \u003c/div\u003e\r\n\r\n\r\n\u003cbr\u003e\r\n\r\n[slack-url]: https://join.slack.com/t/camel-kwr1314/shared_invite/zt-1vy8u9lbo-ZQmhIAyWSEfSwLCl2r2eKA\r\n[slack-image]: https://img.shields.io/badge/Slack-CAMEL--AI-blueviolet?logo=slack\r\n[discord-url]: https://discord.gg/CNcNpquyDc\r\n[discord-image]: https://img.shields.io/badge/Discord-CAMEL--AI-7289da?logo=discord\u0026logoColor=white\u0026color=7289da\r\n[wechat-url]: https://ghli.org/camel/wechat.png\r\n[wechat-image]: https://img.shields.io/badge/WeChat-CamelAIOrg-brightgreen?logo=wechat\u0026logoColor=white\r\n[twitter-url]: https://twitter.com/CamelAIOrg\r\n[twitter-image]: https://img.shields.io/twitter/follow/CamelAIOrg?style=social\u0026color=brightgreen\u0026logo=twitter\r\n[arxiv-image]: https://img.shields.io/badge/arXiv-2407.01511-b31b1b.svg\r\n[arxiv-url]: https://arxiv.org/abs/2407.01511\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcamel-ai%2Fcrab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcamel-ai%2Fcrab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcamel-ai%2Fcrab/lists"}