{"id":20779433,"url":"https://github.com/freedomintelligence/smurfs","last_synced_at":"2025-08-13T03:06:22.599Z","repository":{"id":244310787,"uuid":"774446459","full_name":"FreedomIntelligence/Smurfs","owner":"FreedomIntelligence","description":"Smurfs: Leveraging Multiple Proficiency Agents with Context-Efficiency for Tool Planning","archived":false,"fork":false,"pushed_at":"2025-06-24T05:40:31.000Z","size":36644,"stargazers_count":14,"open_issues_count":0,"forks_count":0,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-06-24T06:34:39.640Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-03-19T15:07:50.000Z","updated_at":"2025-06-24T05:40:36.000Z","dependencies_parsed_at":"2024-09-08T06:35:17.248Z","dependency_job_id":"bac25d87-ae17-4023-9ae9-a37d98593395","html_url":"https://github.com/FreedomIntelligence/Smurfs","commit_stats":null,"previous_names":["freedomintelligence/smurfs"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FreedomIntelligence/Smurfs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FSmurfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FSmurfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FSmurfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FSmurfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/Smurfs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FSmurfs/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270172581,"owners_count":24539386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-13T02:00:09.904Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T13:27:54.897Z","updated_at":"2025-08-13T03:06:22.577Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\r\n\u003ch1\u003eSmurfs\u003cbr\u003e\u003ca href=https://yoursmiles.org/h-smurf.php\u003e\u003cimg src=https://yoursmiles.org/hsmile/smurf/h3602.gif\u003e\u003c/a\u003e\u003ca href=https://yoursmiles.org/h-smurf.php\u003e\u003cimg src=https://yoursmiles.org/hsmile/smurf/h3607.gif\u003e\u003c/a\u003e\u003ca href=https://yoursmiles.org/h-smurf.php\u003e\u003cimg src=https://yoursmiles.org/hsmile/smurf/h3623.gif\u003e\u003c/a\u003e\u003ca href=https://yoursmiles.org/h-smurf.php\u003e\u003cimg src=https://yoursmiles.org/hsmile/smurf/h3625.gif\u003e\u003c/a\u003e\u003c/h1\u003e\r\n\u003c/div\u003e\r\n\r\n\u003cp align=\"center\"\u003e\r\n\u003cimg src=\"assets/logo.webp\" width=\"512\"\u003e\r\n\u003c/p\u003e\r\n\r\n🤖This project aims to construct a synergistic multi-agent system that can handle complex multi-tool instructions without necessitating extra training. This MAS system is called Smurfs, just like the beloved cartoon characters of the same name, symbolize unity and resourcefulness, and are good at\r\nusing tools to overcome any challenge they encounter.\r\n\r\n## ✨ What's New\r\n+  [2025.04.29] Smurfs is accepted by NAACL 2025! Final version of paper can be seen [here](https://aclanthology.org/2025.naacl-long.169.pdf)\r\n+  [2024.09.08] You can try Smurfs on huggingface space [here](https://huggingface.co/spaces/szjiozi/Smurfs)\r\n+  [2024.07.05] CLI and GUI inference have been supported.\r\n+  [2024.06.25] HotpotQA evaluation has been supported.\r\n+  [2024.06.25] We release the new version of our paper at [here](http://arxiv.org/abs/2405.05955)\r\n+  [2024.05.23] We release Smurfs, a multi-agent framework that gives LLM access to external tools to solve complex tasks efficiently.\r\n   + The code and data are available at [Smurfs](#).\r\n\r\n## 🗓 Coming Soon\r\n- [x] Code release of our [paper](http://arxiv.org/abs/2405.05955)\r\n- [x] Support customized API inference\r\n- [x] Support CLI inference\r\n- [x] Support GUI inference\r\n- [ ] More tools are coming\r\n\r\n✨Here is an overview of the Smurfs framework.\r\n\r\n\u003cbr\u003e\r\n\u003cdiv align=\"center\"\u003e\r\n\u003cimg src=\"assets/overview.png\" width=\"800px\"\u003e\r\n\u003c/div\u003e\r\n\u003cbr\u003e\r\n\r\n✨✨Here is a demo of using Smurfs\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n\r\n\r\n\r\nhttps://github.com/FreedomIntelligence/Smurfs/assets/99324175/2edd6d2e-e7f1-4e8e-a78e-56c613d2ba13\r\n\r\n\r\n\r\n\u003c/div\u003e\r\n\r\n✨✨You can also try it using our huggingface space [here](https://huggingface.co/spaces/szjiozi/Smurfs)\r\n\r\n## 🚀 Inference\r\n- CLI Inference:\r\n\r\nAdd tool function to Smurfs/tools/tool_env.py and add all available tool function to tool_env variable, for example:\r\n```python\r\nclass HotpotToolEnv: ...\r\n\r\nHPEnv = HotpotToolEnv()\r\n\r\ntool_env = {\r\n    \"BingSearch\": HPEnv.BingSearch,\r\n    \"Retrieve\": HPEnv.Retrieve,\r\n    \"Lookup\": HPEnv.Lookup\r\n    }\r\n```\r\nThen add the tool description to a json file, for example:\r\n```json\r\n[\r\n    {\r\n        \"api_name\": \"BingSearch\",\r\n        \"api_description\": \"BingSearch can search for rich external knowledge on the Internet based on keywords, which can compensate for knowledge fallacy and knowledge outdated.\",\r\n        \"required_parameters\": [\r\n            {\r\n                \"name\": \"query\",\r\n                \"type\": \"string\",\r\n                \"description\": \"query used to search on the Internet. Should be specific and precise with your query to increase the chances of getting relevant results.\",\r\n                \"default\": \"\"\r\n            }\r\n        ],\r\n        \"optional_parameters\": []\r\n    },\r\n   ... \r\n]\r\n```\r\nthen run\r\n```bash\r\npython Smurfs/deploy/cli_inference.py\r\n```\r\nand type in the input query.\r\n\r\n- GUI Inference:\r\nFollow the same steps as in CLI inference to prepare the tools, then run\r\n```python\r\npython Smurfs/deploy/gradio_inference.py\r\n```\r\n  \r\n## 📚 Data\r\nYou need to first get the StableToolBench dataset and server cache by following the instructions in their [repo](https://github.com/THUNLP-MT/StableToolBench.git), and deploy the API server to perform the experiment.\r\n\r\nThe reproduction data of smurfs can be found at [reproduction_data](https://github.com/FreedomIntelligence/Smurfs/tree/main/reproduction_data). You can use these data to reproduce our experiment result.\r\n\r\n## 🧐 Experiment\r\n- Launch vLLM server:\r\nUsing the script in Smurfs/script/vllm_server.sh to launch a vLLM server of the model that you want to use in the experiment. Suppose you use Mistral-7B-Instruct-v0.2 to do the experiment, you use 4 GPUs to launch the vLLM server and the model is saved at /home/Mistral-7B-Instruct-v0.2, the script looks like:\r\n```bash\r\nmodel_path=\"/home/Mistral-7B-Instruct-v0.2\"\r\nmodel_name=\"Mistral-7B-Instruct-v0.2\"\r\ntensor_parallel_size=4\r\n\r\ncd $model_path\r\ncd ..\r\npython -m vllm.entrypoints.openai.api_server --model $model_name --dtype=half --tensor-parallel-size $tensor_parallel_size\r\n```\r\n\r\nNoted that some models do not have chat template in their tokenizer config file like vicuna, you need to download their chat template from the internet (for example [here](https://github.com/chujiezheng/chat_templates.git)) and use the script below:\r\n```bash\r\nmodel_name=\"Your/Model/Name\"\r\ntensor_parallel_size=4\r\nchat_template_path=\"Your/Template/Path\"\r\n\r\ncd $model_path\r\ncd ..\r\npython -m vllm.entrypoints.openai.api_server --model $model_name --dtype=half --tensor-parallel-size $tensor_parallel_size --chat-template $chat_template_path\r\n```\r\n\r\nThe vLLM server can provide easy, fast, and cheap LLM serving for most popular open-source models. Using it can significantly increase the experiment speed. For more information of vLLM, see [vLLM](https://github.com/vllm-project/vllm.git)\r\n  \r\n- Inference:\r\nTo use the toolbench apis with the toolbench server, you need to first get your toolbench_key (More information can be seen [here](https://github.com/OpenBMB/ToolBench.git)) and pass it through `toolbench_key`. Suppose you save the toolbench data in the directory toolbench_data/data/, the script looks like:\r\n```bash\r\nexport toolbench_key=\"Your_key\"\r\n\r\nmodel_name=\"Mistral-7B-Instruct-v0.2\"\r\nmethod_name=\"smurfs\"\r\ntest_query_id_path=\"toolbench_data/data/test_query_ids\"\r\nquery_file_dir=\"toolbench_data/data/test_instruction\"\r\ntool_env_dir=\"toolbench_data/data/toolenv/tools\"\r\n\r\n\r\npython Smurfs/inference/inference.py \\\r\n    --model_name $model_name \\\r\n    --toolbench_key $toolbench_key \\\r\n    --method_name $method_name \\\r\n    --test_query_id_path $test_query_id_path \\\r\n    --query_file_dir $query_file_dir \\\r\n    --tool_env_dir $tool_env_dir\r\n```\r\nIf you want to do inference with customized RapidAPI account, pass your rapidapi key through rapidapi_key and specify the `use_rapidapi_key` argument in the script:\r\n```bash\r\nexport rapidapi_key=\"Your_key\"\r\n\r\nmodel_name=\"Mistral-7B-Instruct-v0.2\"\r\nmethod_name=\"smurfs\"\r\ntest_query_id_path=\"toolbench_data/data/test_query_ids\"\r\nquery_file_dir=\"toolbench_data/data/test_instruction\"\r\ntool_env_dir=\"toolbench_data/data/toolenv/tools\"\r\n\r\n\r\npython Smurfs/inference/inference.py \\\r\n    --model_name $model_name \\\r\n    --toolbench_key $toolbench_key \\\r\n    --method_name $method_name \\\r\n    --test_query_id_path $test_query_id_path \\\r\n    --query_file_dir $query_file_dir \\\r\n    --tool_env_dir $tool_env_dir \\\r\n    --use_rapidapi_key\r\n```\r\n- Post Process:\r\nThe output of your experiment will be saved at Smurfs/data/your_method_name/. You need to post process it using the following script so that the tooleval from toolbench can evaluate its pass rate and win rate:\r\n```bash\r\ntest_sets=(\"G2_category\" \"G2_instruction\" \"G3_instruction\")\r\ninput_dir=\"data/smurfs\"\r\nexample_dir=\"reproduction_data/mistral_smurfs\"\r\n\r\npython Smurfs/data/post_process.py \\\r\n    --input_dir $input_dir \\\r\n    --test_sets \"${test_sets[@]}\" \\\r\n    --example_dir $example_dir\r\n```\r\n- Evaluation:\r\nFor Evaluation process, download tooleval from [tooleval](https://github.com/OpenBMB/ToolBench/tree/master/toolbench/tooleval) and use the post-processd data as the CONVERTED_ANSWER to do the evaluation following [tooleval](https://github.com/OpenBMB/ToolBench/tree/master/toolbench/tooleval).\r\n\r\n## 📊 Experiment Result\r\n\r\nIn our main experiments on StableToolBench, Smurfs can improve the ability of the base model to handle complex multi-tool instructions that match or even exceed that of capabilities of GPT4-DFSDT. Below are the main results. The win rate for each model is compared with ChatGPT-ReACT.\r\n\r\n**Pass Rate:**\r\n| Backbone | Method | I1-Inst. | I1-Cat. | I1-Tool. | I2-Cat. | I2-Inst. | I3-Inst. | Average |\r\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\r\n| GPT-3.5 Turbo | ReACT | 41.6±1.2 | 48.4±0.5 | 52.5±0.5 | 52.2±1.0 | 31.6±1.2 | 39.9±2.0 | 44.4±1.1 |\r\n| GPT-3.5 Turbo | DFSDT | 54.1±1.0 | 60.1±0.0 | 59.9±1.7 | 60.9±0.9 | 52.8±3.7 | 44.3±4.8 | 55.4±2.0 |\r\n| GPT-3.5 Turbo | **Smurfs** | 60.3±1.5 | 67.0±1.0 | 60.3±1.3 | 54.3±0.4 | 42.6±1.6 | 60.1±1.0 | 57.4±1.1 |\r\n| Mistral-7B | ReACT | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |\r\n| Mistral-7B | DFSDT | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |\r\n| Mistral-7B | **Smurfs** | **76.3±0.8** | **86.7±1.2** | **81.0±1.9** | **70.4±2.7** | **63.8±2.4** | **85.2±0.7** | **77.2±1.6** |\r\n| GPT-4 Turbo | ReACT | 41.1±1.5 | 53.2±1.3 | 42.2±1.1 | 50.0±0.7 | 38.7±0.8 | 37.7±1.3 | 43.8±1.1 |\r\n| GPT-4 Turbo | DFSDT | 52.7±1.4 | 58.2±0.9 | 59.7±1.2 | 59.3±0.7 | 52.2±2.3 | 61.5±1.8 | 57.3±1.4 |\r\n| GPT-4 Turbo | **Smurfs** | 59.3±1.4 | 73.3±1.3 | 67.4±0.7 | 66.7±1.9 | 55.5±1.4 | 70.5±0.0 | 65.5±1.1 |\r\n\r\n**Win Rate:**\r\n| Backbone | Method | I1-Inst. | I1-Cat. | I1-Tool. | I2-Cat. | I2-Inst. | I3-Inst. | Average |\r\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\r\n| GPT-3.5 Turbo | ReACT | / | / | / | / | / | / | / |\r\n| GPT-3.5 Turbo | DFSDT | 64.4 | 61.4 | 53.8 | 62.9 | 66.0 | 54.1 | 60.4 |\r\n| GPT-3.5 Turbo | **Smurfs** | 65.0 | 69.9 | 54.4 | 63.7 | 64.2 | 57.4 | 62.4 |\r\n| Mistral-7B | ReACT | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |\r\n| Mistral-7B | DFSDT | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |\r\n| Mistral-7B | **Smurfs** | 63.8 | 62.7 | 58.2 | 54.0 | 67.0 | 57.4 | 60.5 |\r\n| GPT-4 Turbo | ReACT | 60.1 | 62.1 | 48.1 | 57.3 | 65.1 | 47.5 | 56.7 |\r\n| GPT-4 Turbo | DFSDT | 69.9 | 66.0 | 58.2 | 62.1 | **67.9** | 65.6 | 65.0 |\r\n| GPT-4 Turbo | **Smurfs** | **71.2** | **72.5** | **69.6** | **73.4** | 66.0 | **72.1** | **70.8** |\r\n\r\n## Citation\r\n```\r\n@misc{chen2024smurfs,\r\n      title={Smurfs: Leveraging Multiple Proficiency Agents with Context-Efficiency for Tool Planning}, \r\n      author={Junzhi Chen and Juhao Liang and Benyou Wang},\r\n      year={2024},\r\n      eprint={2405.05955},\r\n      archivePrefix={arXiv},\r\n      primaryClass={cs.CL}\r\n}\r\n```\r\nWe are from the School of Data Science, the Chinese University of Hong Kong, Shenzhen (CUHKSZ) and the Shenzhen Rsearch Institute of Big Data (SRIBD).\r\n\r\n## Acknowledgement\r\nWe are aware that our works are inspired by the following works, including but not limited to\r\n- [Toolbench](https://github.com/OpenBMB/ToolBench.git)\r\n- [StableToolBench](https://github.com/THUNLP-MT/StableToolBench.git)\r\n- [Least to most prompt](https://github.com/RUCAIBox/LLMBox.git)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fsmurfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fsmurfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fsmurfs/lists"}