{"id":13754123,"url":"https://github.com/tangqiaoyu/ToolAlpaca","last_synced_at":"2025-05-09T22:31:01.676Z","repository":{"id":175225476,"uuid":"653428642","full_name":"tangqiaoyu/ToolAlpaca","owner":"tangqiaoyu","description":"ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases","archived":false,"fork":false,"pushed_at":"2024-07-04T07:40:02.000Z","size":3918,"stargazers_count":271,"open_issues_count":7,"forks_count":32,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-08-03T09:06:53.900Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tangqiaoyu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-14T03:21:40.000Z","updated_at":"2024-08-03T03:11:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"9e57932b-788e-42c0-bf68-7ce11702ccfe","html_url":"https://github.com/tangqiaoyu/ToolAlpaca","commit_stats":null,"previous_names":["tangqiaoyu/toolalpaca"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tangqiaoyu%2FToolAlpaca","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tangqiaoyu%2FToolAlpaca/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tangqiaoyu%2FToolAlpaca/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tangqiaoyu%2FToolAlpaca/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tangqiaoyu","download_url":"https://codeload.github.com/tangqiaoyu/ToolAlpaca/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224884612,"owners_count":17386121,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T09:01:41.079Z","updated_at":"2024-11-16T06:31:40.209Z","avatar_url":"https://github.com/tangqiaoyu.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话"],"sub_categories":["大语言对话模型及数据"],"readme":"# ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases\r\n\r\n[![arXiv](https://img.shields.io/badge/arXiv-2306.05301-\u003cCOLOR\u003e.svg?style=flat-square)](https://arxiv.org/abs/2306.05301)\r\n[![](https://img.shields.io/badge/huggingface-ToolAlpaca_7B-blue)](https://huggingface.co/TangQiaoYu/ToolAlpaca-7B)\r\n[![](https://img.shields.io/badge/huggingface-ToolAlpaca_13B-blue)](https://huggingface.co/TangQiaoYu/ToolAlpaca-13B)\r\n\r\n\u003cdiv align=center\u003e\u003cimg src=\"./figures/ToolAlpaca.png\" width=\"400px\" /\u003e\u003c/div\u003e\r\n\r\n`ToolAlpaca` is a framework designed for learning generalized tool-use abilities in compact language models with minimal human supervision. It addresses the challenge of tool learning by generating a tool-use corpus via a multi-agent simulation environment, providing 3.9k tool-use instances from more than 400 tools. \r\n\r\n\u003cdiv align=center\u003e\u003cimg src=\"./figures/pipeline.png\" width=\"500px\" /\u003e\u003c/div\u003e\r\n\r\n\r\n## Data\r\n\r\nDataset list:\r\n\r\n- train_data.json: training data with 400+ APIs\r\n- eval_simulated.json: evaluation data with 10 simulated APIs\r\n- eval_real.json: evaluation data with 11 real APIs, some APIs require authentication.\r\n\r\nData format:\r\n\r\n```json\r\n{\r\n  \"Name\": \"name, from public-apis\",\r\n  \"Description\": \"description, from public-apis\",\r\n  \"Category\": \"category, from public-apis\",\r\n  \"Introduction\": \"introduction, generated by LLM\",\r\n  \"Functions\": \"NLDocumentation in paper v1, generated by LLM\",\r\n  \"Documentation\": \"str(json), OpenAPI Specification documentation, generated by LLM\",\r\n  \"NLDocumentation\": \"natural language documentation, similar to Functions, converted from Documentation\",\r\n  \"Function_Description\": \"each functions description in NLDocumentation\",\r\n  \"Function_Projection\": \"function to HTTP request method\",\r\n  \"Instructions\": \"instructions, generated by LLM\",\r\n  \"Instances\": [\r\n    {\r\n      \"input\": \"use's init instruction, from use agent\",\r\n      \"output\": \"final output, from assistant agent\",\r\n      \"Final Thought\": \"the final thought before output, from assistant agent\",\r\n      \"intermediate_steps\": [\r\n        [\r\n          [\r\n            \"action, from assistant agent\",\r\n            \"action input, str(json), from assistant agent\",\r\n            \"thought + action + action input, assistant agent's output\"\r\n          ]\r\n          \"bbservation, from [user agent, type check python code, tool executor agent]\"\r\n        ]\r\n      ]\r\n    }\r\n  ]\r\n}\r\n```\r\n\r\n## Dataset Generation\r\n- Clone this repository and install packages\r\n```bash\r\ngit clone git@github.com:tangqiaoyu/ToolAlpaca.git\r\ncd ToolAlpaca\r\npip install -r requirements.txt\r\n```\r\n\r\n- download public-api data\r\n```bash\r\npython tool_maker/preprocess_public_apis.py -api data/public_apis.json\r\n```\r\n\r\n- toolset construction\r\n```bash\r\nexport PYTHONPATH=$PYTHONPAT:$(pwd)\r\nexport OPENAI_API_KEY=\"\"\r\n\r\npython tool_maker/get_elements.py -api data/public_apis.json -out ./data\r\n\r\npython tool_maker/natural_language_documentation.py -api ./data/api_data.json\r\n```\r\n\r\n- tool-use instances generation\r\n```bash\r\npython instance_generation/instruction.py -api ./data/api_data.json -out ./data\r\n\r\npython instance_generation/simulator.py -api ./data/api_data.json\r\n\r\npython instance_generation/generation.py -api ./data/api_data.json -out ./data --use_cache\r\n```\r\n\r\n## Train\r\nTo train Toolapaca, we need to create a prompt to organize the dataset in a format that the standard SFT training code can read, similar to what is done in `build_dataset.py`. Afterward, we can proceed with training using the standard SFT method, only optimizing the loss on `thought`, `action`, and `action input`.\r\n\r\n```bash\r\ndeepspeed --num_gpus=2 --master_port=12345 train.py \\\r\n    --deepspeed ${deepspeed config path} \\\r\n    --model_name_or_path ${path to base model like vicuna-7b}  \\\r\n    --data_path ${data path} \\\r\n    --bf16 True \\\r\n    --output_dir outputs/vicuna-7b-toolalpaca/ \\\r\n    --num_train_epochs 3 \\\r\n    --per_device_train_batch_size 32 \\\r\n    --per_device_eval_batch_size 1 \\\r\n    --gradient_accumulation_steps 2 \\\r\n    --evaluation_strategy \"no\" \\\r\n    --save_strategy \"epoch\" \\\r\n    --save_total_limit 10 \\\r\n    --learning_rate 2e-5 \\\r\n    --weight_decay 0. \\\r\n    --warmup_ratio 0.03 \\\r\n    --lr_scheduler_type \"cosine\" \\\r\n    --logging_steps 1 \\\r\n    --tf32 True \\\r\n    --model_max_length 2048 \\\r\n    --gradient_checkpointing True \\\r\n    --lazy_preprocess True\r\n```\r\n\r\nYou can Find our models on huggingface hub: [ToolAlpaca-7B](https://huggingface.co/TangQiaoYu/ToolAlpaca-7B), [ToolAlpaca-13B](https://huggingface.co/TangQiaoYu/ToolAlpaca-13B).\r\n\r\n## Evaluation\r\n- for simulated APIs:\r\n```bash\r\n# start the api simulator\r\npython instance_generation/simulator.py -api ./data/eval_simulated.json\r\n\r\n# get LLM outputs\r\npython instance_generation/generation.py \\\r\n  -api ./data/eval_simulated.json \\\r\n  -out ./eval \\\r\n  -llm TangQiaoYu/ToolAlpaca-13B \\\r\n  --agent_prompt test_v1 \\\r\n  --use_cache\r\n\r\n# evaluation with LLM like GPT-4\r\npython evaluation.py -api ${api_data_path} -out ./eval\r\n```\r\n\r\n- for real APIs:\r\nYou should register the websites and get the API_KEYs.\r\n\r\n```bash\r\npython instance_generation/generation.py \\\r\n  -api ./data/eval_real.json \\\r\n  -out ./data \\\r\n  -llm TangQiaoYu/ToolAlpaca-13B \\\r\n  --agent_prompt test_v1 \\\r\n  --real\r\n\r\npython evaluation.py -api ${api_data_path} -out ./eval\r\n```\r\n\r\n\r\n## Citation\r\n\r\nIf you find our work helpful, please cite as\r\n\r\n```bibtex\r\n@misc{tang2023toolalpaca,\r\n      title={ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases}, \r\n      author={Qiaoyu Tang and Ziliang Deng and Hongyu Lin and Xianpei Han and Qiao Liang and Le Sun},\r\n      year={2023},\r\n      eprint={2306.05301},\r\n      archivePrefix={arXiv},\r\n      primaryClass={cs.CL}\r\n}\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftangqiaoyu%2FToolAlpaca","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftangqiaoyu%2FToolAlpaca","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftangqiaoyu%2FToolAlpaca/lists"}