{"id":27984488,"url":"https://github.com/JieyuZ2/EcoAssistant","last_synced_at":"2025-05-08T05:02:01.152Z","repository":{"id":196426794,"uuid":"696086319","full_name":"JieyuZ2/EcoAssistant","owner":"JieyuZ2","description":"EcoAssistant: using LLM assistant more affordably and accurately","archived":false,"fork":false,"pushed_at":"2024-06-30T06:08:19.000Z","size":966,"stargazers_count":132,"open_issues_count":1,"forks_count":7,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-03T11:02:56.226Z","etag":null,"topics":["chatbot","gpt","large-language-models","llm-inference","nlp"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2310.03046","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JieyuZ2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-25T03:50:08.000Z","updated_at":"2025-03-10T12:30:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"f159d771-ba00-4242-be63-cabe96bb5455","html_url":"https://github.com/JieyuZ2/EcoAssistant","commit_stats":{"total_commits":15,"total_committers":2,"mean_commits":7.5,"dds":"0.19999999999999996","last_synced_commit":"99ec822d73e44b9147c5a2a75d579f73a47ba0d7"},"previous_names":["jieyuz2/ecoassistant"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JieyuZ2%2FEcoAssistant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JieyuZ2%2FEcoAssistant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JieyuZ2%2FEcoAssistant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JieyuZ2%2FEcoAssistant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JieyuZ2","download_url":"https://codeload.github.com/JieyuZ2/EcoAssistant/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253002857,"owners_count":21838640,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbot","gpt","large-language-models","llm-inference","nlp"],"created_at":"2025-05-08T05:01:59.534Z","updated_at":"2025-05-08T05:02:01.067Z","avatar_url":"https://github.com/JieyuZ2.png","language":"Python","funding_links":[],"categories":["2.3 Ensemble After Inference"],"sub_categories":["2.3.2 (c,2) Cascade"],"readme":"# EcoAssistant\n\nEcoAssistant: using LLM assistant more affordably and accurately\n\n**Check out our [blog](https://microsoft.github.io/autogen/blog/2023/11/09/EcoAssistant) in AutoGen website!**\n\n**A simplified version with latest AutoGen is in `simplified_demo.py`**\n\n\n## Introduction\n\n### What is EcoAssistant?\n\nEcoAssistant is a framework that can make the LLM assistant more affordable and accurate for code-driven question answering. It is based on the idea of **assistant hierarchy** and **solution demonstration**.\nIt is built upon [AutoGen](https://github.com/microsoft/autogen).\n\n![](https://github.com/JieyuZ2/EcoAssistant/blob/main/assets/system.png?raw=true)\n\n### What is LLM assistant?\n\nLLM assistant is an assistant agent backed by conversational LLM such as ChatGPT and GPT-4 and is able to address user query in a conversation.\n\n### What is code-driven question answering?\n\nCode-driven question answering is a task that requires the LLM assistant to write code to call external APIs to answer the question. \nFor example, given the question \"What is the average temperature of the city X in the next 5 days?\", the assistant needs to write code to get the weather information via certain APIS and calculate the average temperature of the city X in the next 5 days.\n\nCode-driven question answering requires iterative coding, because, just like human, LLM can hardly write the correct code at the first attempt.\nTherefore, the assistant needs to interact with the user to get feedback and revise the code iteratively until the code is correct.\nWe built our system on a **two-agent conversation framework**, where the LLM assistant is coupled with a code executor agent that can automatically execute the code and return the output to the LLM assistant.\n\n### What is assistant hierarchy?\n\nAssistant hierarchy is a hierarchy of assistants, where the LLM assistants are ranked by their cost (e.g., GPT-3.5-turbo -\u003e GPT-4). \nWhen addressing a user query, the EcoAssistant first asks the cheapest assistant to answer the query. \nOnly when it fails, we invoke the more expensive assistant. It is designed to save the cost by reducing the usage of expensive assistants.\n\n### What is solution demonstration?\n\nSolution demonstration is a technique that leverage the past successful query-code pair to help future queries.\nEverytime when a query is successfully addressed, we save the query-code pair to a database. \nWhen a new query comes, we retrieve the most similar query from the database, and then use the query and its associated code as in-context demonstration.\nIt is designed to improve the accuracy by leveraging the past successful query-code pairs.\n\n**The combination of assistant hierarchy and solution demonstration amplifies the individual benefits because the solution from high-performing model would be naturally leveraged to guide weaker model without specific designs.**\n\n### Why use EcoAssistant?\n\nFor queries about weather, stock, and places, EcoAssistant surpass individual GPT-4 assistant by 10 points of success rate with less than 50% of GPT-4's cost.\nMore details can be found in our paper.\n\n## Preparation\n\nAll the data are included in this repository.\n\nYou only need to set your API keys in `keys.json`\n\nInstall required libraries (we recommend Python3.10):\n\n```bash\npip3 install -r requirements.txt\n```\n\n## Instructions\n\nWe use the **Mixed-100** dataset as an example. For other dataset, just change the dataset name to google_places/stock/weather/mixed_1/mixed_2/mixed_3 in the following commands.\n\nThe outputting results can be found in the `results` folder.\n\nThe following commands are for the autonomous systems without human feedback described in the Section 4.5.\n\n\n**Run the GPT-3.5-turbo assistant**\n\n```bash\npython3 run.py --data mixed_100 --seed 0 --api --model gpt-3.5-turbo \n```\n\n**Run the GPT-3.5-turbo assistant + Chain-of-Thought**\n\nturn on `cot`\n\n```bash\npython3 run.py --data mixed_100 --seed 0 --api --cot --model gpt-3.5-turbo \n```\n\n\n**Run the GPT-3.5-turbo assistant + solution demonstration**\n\nturn on `solution_demonstration`\n\n```bash\npython3 run.py --data mixed_100 --seed 0 --api --solution_demonstration --model gpt-3.5-turbo \n```\n\n\n**Run the assistant hierarchy (GPT-3.5-turbo + GPT-4)**\n\nset `model` to `gpt-3.5-turbo,gpt-4` \n\n```bash\npython3 run.py --data mixed_100 --seed 0 --api --model gpt-3.5-turbo,gpt-4\n```\n\n\n**Run the EcoAssistant: assistant hierarchy (GPT-3.5-turbo + GPT-4) + solution demonstration**\n\n```bash\npython3 run.py --data mixed_100 --seed 0 --api --solution_demonstration --model gpt-3.5-turbo,gpt-4\n```\n\n**Enable human feedback**\n\nFor systems with human judgement, please set `eval` to `human` (which is by default `llm`) like the following example command.\n\n```bash\npython3 run.py --data mixed_100 --seed 0 --api --solution_demonstration --model gpt-3.5-turbo,gpt-4 --eval human\n```\n\n**Run the gold code for Mixed-100 we collect as described in Section 4.4**\n\nThis script would print the code outputs.\n\n```bash\npython3 run_gold_code_for_mix_100.py\n```\n\n## :paperclip: Cite\n\nIf you find this repository useful, please consider citing:\n```bibtex\n@article{zhang2023ecoassistant,\n  title={EcoAssistant: Using LLM Assistant More Affordably and Accurately},\n  author={Zhang, Jieyu and Krishna, Ranjay and Awadallah, Ahmed H and Wang, Chi},\n  journal={arXiv preprint arXiv:2310.03046},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJieyuZ2%2FEcoAssistant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJieyuZ2%2FEcoAssistant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJieyuZ2%2FEcoAssistant/lists"}