{"id":16283339,"url":"https://github.com/zhudotexe/redel","last_synced_at":"2025-04-05T16:03:45.017Z","repository":{"id":191227816,"uuid":"678578816","full_name":"zhudotexe/redel","owner":"zhudotexe","description":"ReDel is a toolkit for researchers and developers to build, iterate on, and analyze recursive multi-agent systems. (EMNLP 2024 Demo)","archived":false,"fork":false,"pushed_at":"2025-03-17T21:20:45.000Z","size":1120723,"stargazers_count":74,"open_issues_count":2,"forks_count":11,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-29T15:02:57.220Z","etag":null,"topics":["large-language-models","multi-agent-systems","visualization"],"latest_commit_sha":null,"homepage":"https://aclanthology.org/2024.emnlp-demo.17/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhudotexe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"zhudotexe","ko_fi":"zhuexe"}},"created_at":"2023-08-14T22:09:46.000Z","updated_at":"2025-03-26T19:31:27.000Z","dependencies_parsed_at":"2024-05-28T02:38:32.439Z","dependency_job_id":"33c1f2bc-6a0e-4302-9d3d-558e61c405ef","html_url":"https://github.com/zhudotexe/redel","commit_stats":null,"previous_names":["zhudotexe/kan-pi","zhudotexe/kanpai","zhudotexe/redel"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhudotexe%2Fredel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhudotexe%2Fredel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhudotexe%2Fredel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhudotexe%2Fredel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhudotexe","download_url":"https://codeload.github.com/zhudotexe/redel/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247361600,"owners_count":20926641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["large-language-models","multi-agent-systems","visualization"],"created_at":"2024-10-10T19:13:09.930Z","updated_at":"2025-04-05T16:03:44.973Z","avatar_url":"https://github.com/zhudotexe.png","language":"Python","funding_links":["https://github.com/sponsors/zhudotexe","https://ko-fi.com/zhuexe"],"categories":[],"sub_categories":[],"readme":"# ReDel\n\n*A framework for recursive delegation of LLMs*\n\n[Check out the paper!](https://aclanthology.org/2024.emnlp-demo.17/)\n\nReDel is a toolkit for researchers and developers to build, iterate on, and analyze recursive multi-agent systems.\n\nBuilt using the [kani](https://github.com/zhudotexe/kani) framework, it offers best-in-class support for modern\nLLMs with tool usage.\n\n## Features\n\n- **Modular design** - ReDel makes it easy to experiment by providing a modular interface for creating tools, different\n  delegation methods, and logs for later analysis.\n- **Event-driven architecture** - Granular logging and a central event system makes it easy to listen for signals\n  from anywhere in your system. Every event is automatically logged so you can run your favorite data analysis tools.\n- **Bundled visualization** - Multi-agent systems can be hard to reason about from a human perspective. We provide a\n  web-based visualization that allows you to interact with a configured system directly or view replays of saved runs\n  (e.g. your own experiments!).\n- **Built with open, unopinionated tech** - ReDel won't force you to learn bizarre library-specific tooling and isn't\n  built by a big tech organization with their own motives. Everything in ReDel is implemented in pure, idiomatic Python\n  and permissively licensed.\n\n## Quickstart\n\nRequires Python 3.10+\n\n```shell\n# install python dependencies\n$ pip install \"redel[all] @ git+https://github.com/zhudotexe/redel.git@main\"\n# run web visualization of a ReDel system with web browsing\n$ OPENAI_API_KEY=\"...\" python -m redel.server\n```\n\n## Screenshots\n\n![The ReDel homepage](docs/_static/home.png)\n\n![Interactive](docs/_static/delegate2.png)\n\n![Loading saved logs](docs/_static/loader.png)\n\n![Replay](docs/_static/replay.png)\n\n## Usage\n\nThere are two primary ways to interact with a system: interactively, through the web\ninterface, or programmatically. The former is particularly useful to debug your system's behaviour, iterate on prompts,\nor otherwise provide an interactive experience. The latter is useful for running experiments and batch queries.\n\nSee the docs for more usage information at https://redel.readthedocs.io!\n\n### Server\n\n```python\nfrom kani.engines.openai import OpenAIEngine\nfrom redel import AUTOGENERATE_TITLE, ReDel\nfrom redel.server import VizServer\nfrom redel.tools.browsing import Browsing\n\n# Define the LLM engines to use for each node\nengine = OpenAIEngine(model=\"gpt-4\", temperature=0.8, top_p=0.95)\n\n# Define the configuration for each interactive session\nredel_proto = ReDel(\n    root_engine=engine,\n    delegate_engine=engine,\n    title=AUTOGENERATE_TITLE,\n    tool_configs={\n        Browsing: {\"always_include\": True},\n    },\n)\n\n# configure and start the server\nserver = VizServer(redel_proto)\nserver.serve()\n```\n\n### Programmatic\n\n```python\nimport asyncio\nfrom kani import ChatRole\nfrom kani.engines.openai import OpenAIEngine\nfrom redel import ReDel, events\nfrom redel.tools.browsing import Browsing\n\n# Define the LLM engines to use for each node\nengine = OpenAIEngine(model=\"gpt-4\", temperature=0.8, top_p=0.95)\n\n# Define the configuration for the session\nai = ReDel(\n    root_engine=engine,\n    delegate_engine=engine,\n    title=\"Airspeed of a swallow\",\n    tool_configs={\n        Browsing: {\"always_include\": True},\n    },\n)\n\n\n# ReDel is async, so define an async function and use asyncio.run()\nasync def main():\n    async for event in ai.query(\"What is the airspeed velocity of an unladen swallow?\"):\n        if isinstance(event, events.RootMessage) and event.msg.role == ChatRole.ASSISTANT:\n            if event.msg.text:\n                print(event.msg.text)\n\n\nasyncio.run(main())\n```\n\n## EMNLP Demo Experiments\n\n\u003e [!NOTE]\n\u003e This section is specific to the `demo/emnlp` branch of this repository. You can switch branches in the top-left of\n\u003e the GitHub UI or by using this link: https://github.com/zhudotexe/redel/tree/demo/emnlp\n\nThis repository includes the logs of every single experiment run included in our paper in\nthe `experiments/` directory. You can load any of these runs in the visualization to view what the ReDel system did!\n\nThe experiments directory is broken down into the following\nstructure: `experiments/BENCHMARK_NAME/BENCHMARK_SPLIT/[RUN_ID]/SYSTEM_ID/QUERY_ID`, where:\n\n- `BENCHMARK_NAME` is the name of the benchmark (fanoutqa, travelplanner, or webarena)\n- `BENCHMARK_SPLIT` is the split of the benchmark we ran (usually the dev/validation split)\n- `RUN_ID` is an internal split in the FanOutQA experiment to analyze an edge-case behaviour wrt parallel function\n  calling and long contexts\n- `SYSTEM_ID` is the system under test, configured as in the table below\n- `QUERY_ID` is the benchmark-specific ID of a single run (loadable in the visualizer).\n\n### System Configurations\n\n| System ID      | Root Model    | Delegate Model | Root Functions? | Delegation? | Root Context | Delegate Context |\n|----------------|---------------|----------------|-----------------|-------------|--------------|------------------|\n| full           | gpt-4o        | gpt-4o         | no              | yes         | 128000       | 128000           |\n| root-fc        | gpt-4o        | gpt-4o         | yes             | yes         | 128000       | 128000           |\n| baseline       | gpt-4o        | N/A            | yes             | no          | 128000       | N/A              |\n| small-leaf     | gpt-4o        | gpt-3.5-turbo  | no              | yes         | 128000       | 16385            |\n| small-all      | gpt-3.5-turbo | gpt-3.5-turbo  | no              | yes         | 16385        | 16385            |\n| small-baseline | gpt-3.5-turbo | N/A            | yes             | no          | 16385        | N/A              |\n| short-context  | gpt-4o        | gpt-4o         | no              | yes         | 8192         | 8192             |\n| short-baseline | gpt-4o        | N/A            | yes             | no          | 8192         | N/A              |\n\n### Reproducing Experiments\n\nTo reproduce the experiments included in this repository, we include scripts to run each benchmark.\n\nFollow these steps to setup the environment, then follow the instructions in each benchmark. We recommend setting up\na virtual environment for this project.\n\n1. First, you'll need to clone this repository and check out the `demo/emnlp`\n   branch: `git clone -b demo/emnlp https://github.com/zhudotexe/redel`\n2. Install the necessary dependencies: `pip install -r requirements.txt`\n\n#### FanOutQA\n\n*output path: `experiments/fanoutqa/dev/trial2/SYSTEM_ID`*\n\n**Run**\n\n```shell\npython bench_fanoutqa.py \u003cfull|root-fc|baseline|small-leaf|small-all|small-baseline|short-context|short-baseline\u003e\n```\n\nThis will run the given system on the FanOutQA dev set in the Open Book setting.\n\n**Evaluate**\n\nSet the `FANOUTQA_OPENAI_API_KEY` environment variable to a valid OpenAI API key. You can\nuse `export FANOUTQA_OPENAI_API_KEY=$OPENAI_API_KEY` to copy an existing API key from environment variables.\n\n```shell\npython score_fanoutqa.py experiments/fanoutqa/**/results.jsonl\n```\n\nThis will output a `score.json` file in the output path with the final scores.\n\n#### TravelPlanner\n\n*output path: `experiments/travelplanner/validation/SYSTEM_ID`*\n\n**Setup**\n\n1. Install the TravelPlanner database:\n    1. Download the database\n       from [this link](https://drive.google.com/file/d/1pF1Sw6pBmq2sFkJvm-LzJOqrmfWoQgxE/view?usp=drive_link)\n    2. Extract the zip file in `redel/tools/travelplanner`. This should create a directory named `db`.\n2. In another directory, clone our fork of the TravelPlanner repository. This will be used for scoring, and includes the\n   fixes discussed in our paper.\n    1. `git clone https://github.com/zhudotexe/TravelPlanner`\n\n**Run**\n\n```shell\npython bench_travelplanner.py \u003cfull|root-fc|baseline|small-leaf|small-all|small-baseline\u003e\n```\n\nNote: This benchmark does not test the `short-ctx` systems since this benchmark doesn't have a long-context requirement.\n\n**Evaluate**\n\n```shell\npython score_travelplanner.py experiments/travelplanner/**/results.jsonl\n```\n\nThis script will write files in the correct format for the TravelPlanner evaluation in the output path, and\nprint the command to run to score the results.\n\nYou should now switch to the TravelPlanner repository you cloned in the setup step and run the commands output by this\nscript.\n\n#### WebArena\n\n*output path: `experiments/webarena/test/SYSTEM_ID`*\n\n**Setup**\n\nWe reproduce some of the scripts and data contained in the WebArena repository in this repo under the terms of the\nApache-2.0 license, contained in `experiments/webarena/vendor/LICENSE`.\n\nFirst, you'll need to set up your own WebArena environment.\nSee https://github.com/web-arena-x/webarena/blob/main/environment_docker/README.md for instructions.\n\nNext, run the following to setup the webarena configuration:\n\n```shell\n# setup env vars (see https://github.com/web-arena-x/webarena/blob/main/environment_docker/README.md for env setup)\nexport SHOPPING=\"\u003cyour_shopping_site_domain\u003e:7770\"\nexport SHOPPING_ADMIN=\"\u003cyour_e_commerce_cms_domain\u003e:7780/admin\"\nexport REDDIT=\"\u003cyour_reddit_domain\u003e:9999\"\nexport GITLAB=\"\u003cyour_gitlab_domain\u003e:8023\"\nexport MAP=\"\u003cyour_map_domain\u003e:3000\"\nexport WIKIPEDIA=\"\u003cyour_wikipedia_domain\u003e:8888/wikipedia_en_all_maxi_2022-05/A/User:The_other_Kiwix_guy/Landing\"\nexport HOMEPAGE=\"\u003cyour_homepage_domain\u003e:4399\"\n# generate config files\npython experiments/webarena/generate_test_data.py\n```\n\nYou'll also need to ensure Playwright is installed:\n\n```shell\nplaywright install chromium\n```\n\n**Run**\n\nFirst, make sure you have reset your WebArena environment\n(see https://github.com/web-arena-x/webarena/blob/main/environment_docker/README.md#environment-reset).\n\nThen, launch the WebArena environment.\n\nAs the default WebArena script is incompatible with asyncio, ReDel launches a separate process to handle the\nWebArena environment, which it communicates with over a pipe. This is done automatically.\n\nFinally, run the bench script:\n\n```shell\npython bench_webarena.py \u003cfull|root-fc|baseline|small-leaf|small-all|small-baseline|short-context|short-baseline\u003e\n```\n\n## License\n\nWe release ReDel under the terms of the MIT license, included in `LICENSE`. ReDel is intended for academic and personal\nuse only. To use ReDel for commercial purposes, please contact us.\n\n## Citation\n\nIf you use our code or findings in your research, please cite us as:\n\n```\n@inproceedings{zhu-etal-2024-redel,\n    title = \"{R}e{D}el: A Toolkit for {LLM}-Powered Recursive Multi-Agent Systems\",\n    author = \"Zhu, Andrew  and\n      Dugan, Liam  and\n      Callison-Burch, Chris\",\n    editor = \"Hernandez Farias, Delia Irazu  and\n      Hope, Tom  and\n      Li, Manling\",\n    booktitle = \"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations\",\n    month = nov,\n    year = \"2024\",\n    address = \"Miami, Florida, USA\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2024.emnlp-demo.17\",\n    pages = \"162--171\",\n}\n```\n\n## Who we are\n\n\u003cimg alt=\"University of Pennsylvania Logo\" src=\"docs/_static/penn-logo.jpg\" width=\"300\"\u003e\n\nThe core development team is made of two PhD students in the Department of Computer and Information Science at the\nUniversity of Pennsylvania. We're members of\n[Prof. Chris Callison-Burch's](https://www.cis.upenn.edu/~ccb/) lab, working towards advancing the future of NLP.\n\n- [**Andrew Zhu**](https://zhu.codes/) started in Fall 2022. His research interests include natural language processing,\n  programming languages, distributed systems, and more. He's also a full-stack software engineer, proficient in all\n  manner of backend, devops, database, and frontend engineering. Andrew strives to make idiomatic, clean, performant,\n  and low-maintenance code — philosophies that are often rare in academia. His research is supported by the NSF Graduate\n  Research Fellowship.\n- [**Liam Dugan**](https://liamdugan.com/) started in Fall 2021. His research focuses primarily on large language models\n  and how humans interact with them. In particular, he is interested in human detection of generated text and whether we\n  can apply those insights to automatic detection systems. He is also interested in the practical application of large\n  language models to education.\n\n**Library Support**:\nWe are working on researching recursive multi-agent systems using ReDel, and we aim to keep it up-to-date with modern\nNLP practices. As an active research library, the release schedule is somewhat ad-hoc. Published releases should be\nstable for wide use, and any experiment-specific code should be contained within a branch or fork.\n\n## Acknowledgements\n\nThis research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced\nResearch Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200005.\nThis material is based upon work supported by the National Science Foundation Graduate Research Fellowship, under Grant\nNo. DGE-2236662.\nThe views and conclusions contained herein are those of the authors and should not be interpreted as necessarily\nrepresenting the official policies or views, either expressed or implied, of ODNI, IARPA, the NSF, or the U.S.\nGovernment. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes\nnotwithstanding any copyright annotation therein.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhudotexe%2Fredel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhudotexe%2Fredel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhudotexe%2Fredel/lists"}