{"id":19520157,"url":"https://github.com/osu-nlp-group/middleware","last_synced_at":"2025-04-15T23:53:01.186Z","repository":{"id":215479223,"uuid":"738835066","full_name":"OSU-NLP-Group/Middleware","owner":"OSU-NLP-Group","description":"Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments (EMNLP'2024)","archived":false,"fork":false,"pushed_at":"2024-12-29T13:36:06.000Z","size":2055,"stargazers_count":36,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-29T03:02:41.169Z","etag":null,"topics":["agents","llm","tool-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSU-NLP-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-04T06:43:10.000Z","updated_at":"2025-03-17T14:40:25.000Z","dependencies_parsed_at":"2024-11-11T00:29:25.562Z","dependency_job_id":"a21e65a0-794d-4f75-a910-20e4d2ec6d5a","html_url":"https://github.com/OSU-NLP-Group/Middleware","commit_stats":null,"previous_names":["osu-nlp-group/fuxi","osu-nlp-group/middleware"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FMiddleware","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FMiddleware/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FMiddleware/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FMiddleware/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSU-NLP-Group","download_url":"https://codeload.github.com/OSU-NLP-Group/Middleware/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249173061,"owners_count":21224481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","llm","tool-learning"],"created_at":"2024-11-11T00:24:06.358Z","updated_at":"2025-04-15T23:53:01.165Z","avatar_url":"https://github.com/OSU-NLP-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-red.svg)](https://opensource.org/licenses/Apache-2.0)\n[![language-python3](https://img.shields.io/badge/Language-Python3-blue.svg?style=flat-square)](https://www.python.org/)\n[![paper](https://img.shields.io/badge/Paper-EMNLP2024-gold?style=flat-square)](https://aclanthology.org/2024.emnlp-main.436.pdf)\n\n\u003e The applications of large language models (LLMs) have expanded well beyond the confines of text processing, signaling a new era where LLMs are envisioned as generalist agents capable of operating within complex environments. These environments are often highly expansive, making it impossible for the LLM to process them within its short-term memory. Motivated by recent research on extending the capabilities of LLMs with tools, we seek to investigate the intriguing potential of tools to augment LLMs in handling such complexity by introducing a novel class of tools, termed *middleware*, to aid in the proactive exploration within these massive environments. Such specialized tools can serve as a middleware layer shielding the LLM from environmental complexity. In two representative complex environments—knowledge bases (KBs) and databases—we demonstrate the significant potential of augmenting language agents with tools in complex environments. Notably, equipped with the middleware, GPT-4 achieves **2.8**X the performance of the best baseline in tasks requiring access to database content and **2.2**X in KB tasks. Our findings illuminate the path for advancing language agents in real-world applications.\n\u003cimg width=\"605\" alt=\"image\" src=\"https://github.com/user-attachments/assets/05530abb-74e5-4914-9fe8-576adaf3a928\" /\u003e\n\n\n## Setup\n```\nconda create -n middleware python=3.9\nconda activate middleware\npip install -r requirements.txt\n```\n\n### Setup for KGs\nTo run our experiments on Freebase, please follow [Freebase Setup](https://github.com/dki-lab/Freebase-Setup]) to set up a Virtuoso triplestore service. You can set up your Virtuoso server on port 3093. If you use a different port, please remember to correspondingly update the url [here](https://github.com/OSU-NLP-Group/Middleware/blob/990a6beb8d749932bb2891416fbd88f9a16e614b/src/tasks/knowledgegraph/utils/sparql_executer.py#L24).\n\n### Setup for DBs\nFor our experiments on BIRD, please first download the databases associated with its dev set from the [official link](https://bird-bench.github.io) and put all files under `./data/birdbench/`.\nNote that, in the original dev set, no information regarding whether a task requires DB content to solve is provided. We have provided the information in our own `dev.json` file under `./data/birdbench/`. Specifically, whether a task requires content-level information to solve is indicated by the following json field:\n```json\n{\n    ...\n     \"require_content_info\": [true|false],\n    ...\n}\n```\n\n## Evaluation\nOur codebase is mostly adapted from the 0.1 version of [AgentBench](https://github.com/THUDM/AgentBench). \nFor a more detailed description of the structure of the source code and config files, please find more detailed information there (but this is not necessary if you just want to reproduce the experiments in our Middleware work).\n\nTo run experiments using our codebase, simply do\n```\npython eval.py \\\n --task configs/tasks/\u003cyour_task\u003e.yaml \\\n --agent configs/agents/\u003cyour_agent\u003e.yaml \\\n --workers \u003cnum_of_threads\u003e\n```\nFor example,\n```\npython eval.py \\\n --task configs/tasks/knowledgegraph/dev.yaml \\\n --agent configs/agents/api_agents/gpt-3.5-turbo.yaml \\\n --workers 10\n```\nThis command can be used to evaluate on dev.yaml of our KG experiments using gpt-3.5-turbo, running with 10 threads in parallel.\n\nFor agent config files, please remember to replace placeholders with your actual API keys.\n\n\n\n## Citation\n```\n@inproceedings{gu-etal-2024-middleware,\n    title = \"Middleware for {LLM}s: Tools Are Instrumental for Language Agents in Complex Environments\",\n    author = \"Gu, Yu  and\n      Shu, Yiheng  and\n      Yu, Hao  and\n      Liu, Xiao  and\n      Dong, Yuxiao  and\n      Tang, Jie  and\n      Srinivasa, Jayanth  and\n      Latapie, Hugo  and\n      Su, Yu\",\n    editor = \"Al-Onaizan, Yaser  and\n      Bansal, Mohit  and\n      Chen, Yun-Nung\",\n    booktitle = \"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing\",\n    month = nov,\n    year = \"2024\",\n    address = \"Miami, Florida, USA\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2024.emnlp-main.436\",\n    doi = \"10.18653/v1/2024.emnlp-main.436\",\n    pages = \"7646--7663\",\n    abstract = \"The applications of large language models (LLMs) have expanded well beyond the confines of text processing, signaling a new era where LLMs are envisioned as generalist agents capable of operating within complex environments. These environments are often highly expansive, making it impossible for the LLM to process them within its short-term memory. Motivated by recent research on extending the capabilities of LLMs with tools, we seek to investigate the intriguing potential of tools to augment LLMs in handling such complexity by introducing a novel class of tools, termed *middleware*, to aid in the proactive exploration within these massive environments. Such specialized tools can serve as a middleware layer shielding the LLM from environmental complexity. In two representative complex environments{---}knowledge bases (KBs) and databases{---}we demonstrate the significant potential of augmenting language agents with tools in complex environments. Notably, equipped with the middleware, GPT-4 achieves **2.8**X the performance of the best baseline in tasks requiring access to database content and **2.2**X in KB tasks. Our findings illuminate the path for advancing language agents in real-world applications.\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fmiddleware","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosu-nlp-group%2Fmiddleware","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fmiddleware/lists"}