{"id":44293625,"url":"https://github.com/estuary-ai/mangrove","last_synced_at":"2026-02-11T00:09:25.053Z","repository":{"id":245562647,"uuid":"752097194","full_name":"estuary-ai/mangrove","owner":"estuary-ai","description":"Mangrove is the backend module of Estuary, a framework for building multimodal real-time Socially Intelligent Agents (SIAs).","archived":false,"fork":false,"pushed_at":"2025-07-11T00:06:08.000Z","size":2369,"stargazers_count":11,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-11T05:52:38.045Z","etag":null,"topics":["affective-computing","agents","artificial-intelligence","automatic-speech-recognition","digital-assistant","framework","human-computer-interaction","large-language-models","socially-aware-agents","socially-intelligent-agents","speech-recognition","speech-synthesis"],"latest_commit_sha":null,"homepage":"https://estuary-ai.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/estuary-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-03T02:41:59.000Z","updated_at":"2025-07-11T00:06:12.000Z","dependencies_parsed_at":"2024-06-22T20:14:21.885Z","dependency_job_id":"4943cbf4-af27-48ee-9dad-4216053e8135","html_url":"https://github.com/estuary-ai/mangrove","commit_stats":null,"previous_names":["al-estuary/mangrove","estuary-ai/mangrove"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/estuary-ai/mangrove","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estuary-ai%2Fmangrove","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estuary-ai%2Fmangrove/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estuary-ai%2Fmangrove/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estuary-ai%2Fmangrove/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/estuary-ai","download_url":"https://codeload.github.com/estuary-ai/mangrove/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estuary-ai%2Fmangrove/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29322769,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-10T20:44:44.282Z","status":"ssl_error","status_checked_at":"2026-02-10T20:44:43.393Z","response_time":65,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["affective-computing","agents","artificial-intelligence","automatic-speech-recognition","digital-assistant","framework","human-computer-interaction","large-language-models","socially-aware-agents","socially-intelligent-agents","speech-recognition","speech-synthesis"],"created_at":"2026-02-11T00:09:24.375Z","updated_at":"2026-02-11T00:09:25.044Z","avatar_url":"https://github.com/estuary-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mangrove\nMangrove is the backend module of Estuary, a framework for building multimodal real-time Socially Intelligent Agents (SIAs).\n\n## Give us a Star! ⭐\nIf you find Estuary helpful, please give us a star!  Your support means a lot! \nIf you find any bugs or would like to request a new feature, feel free to open an\nissue!\n\n## Citing Estuary\nIf you would like to use Estuary for your work, please cite:\n\n```bash\n   @inproceedings{10.1145/3652988.3696198,\n   author = {Lin, Spencer and Rizk, Basem and Jun, Miru and Artze, Andy and Sullivan, Caitl\\'{\\i}n and Mozgai, Sharon and Fisher, Scott},\n   title = {Estuary: A Framework For Building Multimodal Low-Latency Real-Time Socially Interactive Agents},\n   year = {2024},\n   isbn = {9798400706257},\n   publisher = {Association for Computing Machinery},\n   address = {New York, NY, USA},\n   url = {https://doi.org/10.1145/3652988.3696198},\n   doi = {10.1145/3652988.3696198},\n   booktitle = {Proceedings of the 24th ACM International Conference on Intelligent Virtual Agents},\n   articleno = {50},\n   numpages = {3},\n   location = {GLASGOW, United Kingdom},\n   series = {IVA '24}}\n```\n\n## Supported Endpoints\n\n### Speech-To-Text (STT/ASR)\n* Faster-Whisper\n\n### Large Language Models (LLMs)\n* ChatGPT\n* Ollama\n\n### Text-To-Speech (TTS)\n* ElevenLabs\n* XTTS-v2\n* Google gTTS\n* pyttsx3\n\n\n# Setup\n## Environment Setup\n1. **[WSL Ubuntu 22.04]** Currently, Mangrove is tested to work in WSL Ubuntu 22.04.  To install WSL, follow this [official guide]((https://learn.microsoft.com/en-us/windows/wsl/install)) from Microsoft.\n2. **[Updating WSL]** Run `sudo apt update` and `sudo apt upgrade` in WSL.\n3. **[Installing pipx]** Run `sudo apt install pipx` in WSL.\n4. **[Installing pdm]** Run `pipx install pdm` in WSL.\n5. **[Installing Conda]** Refer to the Miniconda installation\n   \u003ca href=\"https://docs.anaconda.com/miniconda/install/\" target=\"blank\"\u003eguide\u003c/a\u003e.\n\n## Installing Dependencies\n1. Run the following command to install packages:\n    ```bash\n    sudo apt-get install libcairo2-dev pulseaudio portaudio19-dev libgirepository1.0-dev libespeak-dev sox ffmpeg gstreamer-1.0 clang\n    ```\n2. Open a powershell terminal window and restart your WSL shell (some packages require a restart to finish installation)\n    ```bash\n    wsl --shutdown\n    ```\n3. Clone this repository into your WSL environment and navigate into it\n    ```bash\n    git clone https://github.com/estuary-ai/mangrove.git\n    cd mangrove\n    ```\n4. Create a Python 3.9.19 virtual environment with Conda:\n    ```bash\n    conda create -n mangrove python=3.9.19\n    conda activate mangrove\n    ```\n5. Enter the command `pdm use` and select the correct Python interpreter to use e.g. `/home/username/miniconda3/envs/mangrove/bin/python`\n6. Install Python dependencies.\n    ```bash\n    pdm install -G :all\n    ```\n\nCongrats!  This is the end of the initial installation for Mangrove.  Please refer to the next section for running Mangrove for the first time!\n\n## Running Mangrove for the First Time\n\n### Initial Steps\n1. Navigate to the Mangrove root directory.\n     ```bash\n    cd mangrove\n     ```\n2. Activate the Conda virtual environment that was previously set up.\n    ```bash\n    conda activate mangrove\n    ```\n### Selecting an LLM\n* ChatGPT: Refer to the [API Keys](https://github.com/estuary-ai/mangrove?tab=readme-ov-file#api-keys) section below for set up if you would like to use OpenAI\n    * Flag: `--bot_endpoint openai`      \n* Ollama: If you would like to use offline LLMs and have the VRAM to run them, you may consult the [Ollama](https://github.com/estuary-ai/mangrove?tab=readme-ov-file#ollama) section for set up instructions.\n    * Flag: `--bot_endpoint ollama`\n\n### Selecting a TTS module\n* XTTS: This is a popular offline TTS module that produces both high quality results and is performant at runtime.  You can refer to the [XTTS](https://github.com/estuary-ai/mangrove?tab=readme-ov-file#xtts) section for set up instructions.\n    * Flag: `--tts_endpoint xtts`\n* gTTS: This is a free cloud-based TTS module offered by Google.\n    * Flag: `--tts_endpoint gtts`\n\n ### Other Configurations\n * You may specify which Port number you would like to use with the `--port` flag.\n * You may use CPU for processing with the `--cpu` flag.\n\n### Example Commands\n* Default run command which uses OpenAI and ElevenLabs and port 4000:\n  ```bash\n  python launcher.py\n  ```\n* Example run command which uses the above flags:\n  ```bash\n  python launcher.py --bot_endpoint ollama --tts_endpoint xtts --port 4000\n  ```\n\n### Connecting a Client\n* Python Client: This option is recommended for Python projects or for quick debugging purposes.\n    * Navigate to the client/python directory.\n      ```bash\n      cd client/python/\n      ```\n    * Run the following command to start the client on port 4000:\n      ```bash\n      python client.py\n      ```\n    * You may also specify the address and port for the client to connect to with the `--address` and `--port` flags.\n* Unity Client: If you are building a Unity application, refer to the Estuary Unity SDK [Documentation](https://github.com/estuary-ai/Estuary-Unity-SDK). \n\n## Further Setup as Required\n\n### API Keys\n- Mangrove supports the usage of APIs (e.g., OpenAI), which require API keys. Create `.env` file in the root directory of the project and add your API keys as follows:\n    ```bash\n    OPENAI_API_KEY=[your OpenAI API Key]\n    ELEVENLABS_API_KEY=[your ElevenLabs API Key]\n    ```\n\n### Ollama\n- Install Ollama inside of wsl by running the command:\n  ```bash\n  curl -fsSL https://ollama.com/install.sh | sh\n  ```\n- Install an LLM from [Ollama's model library](https://ollama.com/search) e.g.\n  ```bash\n  ollama run nemotron-mini\n  ```\n\n### XTTS\n- Running XTTS (using Deepspeed) requires a standlone version of cuda library (the same version as the one used by `torch.version.cuda`):\n    1. Install `dkms` package to avoid issues with the installation of the cuda library: `sudo apt-get install dkms`\n    2. Install CUDA 12.1 from the [NVIDIA website](https://developer.nvidia.com/cuda-12-1-0-download-archive?target_os=Linux\u0026target_arch=x86_64\u0026Distribution=WSL-Ubuntu\u0026target_version=2.0\u0026target_type=runfile_local).\n    3. Follow the instructions given by the installation process including installing the driver.\n       ```bash\n       sudo sh cuda_12.1.0_530.30.02_linux.run --silent --driver\n       ```\n    4. Add the following to the .bashrc file with any code editor ie. `nano ~/.bashrc`\n        ```bash\n        export PATH=/usr/local/cuda-12.1/bin${PATH:+:${PATH}}\n        export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}\n        ```\n    5. Add a 6s-30s voice training clip to the root of the project directory.  Make sure to name it `speaker.wav`. \n    6. Make sure to restart WSL afterwards with `wsl --shutdown` in Powershell.\n\n### Networked Configuration\n\nIf you are running Mangrove in WSL and would like to configure Local Area Network (LAN) communications for a remote client, WSL must be set to mirrored network configuration.  You can do this with the following steps:\n\n1. Open Powershell and create/open the .wslconfig file in the `C:\\Users\\[username]\\` directory.\n2. Add the following to the .wslconfig file:\n```bash\n[wsl2]\nnetworkingMode=mirrored\n[experimental] \ndnsTunneling=true\nautoProxy=true\nhostAddressLoopback=true\n```\n3. Add an inbound network rule in Windows Security Settings \u003e Firewall \u0026 Network Protection \u003e Advanced Settings \u003e Inbound Rules \u003e New Rule...\n    - Port \u003e TCP, Specific local ports: 4000 \u003e Allow the connection \u003e Check: Domain, Private, Public \u003e Name: Mangrove\n\n#### Tips\n\n- Ensure both Mangrove and the client are connected to the same LAN and both the machine running Mangrove and the LAN allow for device-to-device communications.\n- Try restarting after applying the above Network Configurations if they do not initially work \n- [OPTIONAL] You may refer to the Microsoft WSL documentation on Mirrored Networking [here](https://learn.microsoft.com/en-us/windows/wsl/networking#mirrored-mode-networking).\n\n# Acknowledgements\nMangrove was built from our base code of developing **Traveller**, the digital assistant of **SENVA**, a prototype Augmented Reality (AR) Heads-Up Display (HUD) solution for astronauts.  Thank you to **Team Aegis** for participating in the **NASA SUITs Challenge** for the following years:\n\n- **2023**: **University of Southern California (USC)** with **University of Berkley (UCBerkley)**\n\n- **2022**: **University of Southern California (USC)** with **University of Arizona (UA)**.\n\nThe Estuary team would also like to acknowledge the developers, authors, and creatives whose work contributed to the success of this project:\n\n- SocketIO Protocol: https://socket.io/docs/v4/socket-io-protocol/\n- FlaskSocketIO Library: https://github.com/miguelgrinberg/Flask-SocketIO\n- Python SocketIO Library: https://github.com/miguelgrinberg/python-socketio\n- Silero-VAD: https://github.com/snakers4/silero-vad\n- Faster-Whisper: https://github.com/SYSTRAN/faster-whisper\n- PyAudio: https://people.csail.mit.edu/hubert/pyaudio/\n- [XTTs](https://arxiv.org/abs/2406.04904): https://github.com/coqui-ai/TTS\n\nMore to come soon! Stay tuned and Fight On!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Festuary-ai%2Fmangrove","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Festuary-ai%2Fmangrove","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Festuary-ai%2Fmangrove/lists"}