{"id":28857712,"url":"https://github.com/ictnlp/stream-omni","last_synced_at":"2026-04-27T18:32:00.201Z","repository":{"id":299459032,"uuid":"1003107219","full_name":"ictnlp/Stream-Omni","owner":"ictnlp","description":"Stream-Omni is an end-to-end language-vision-speech chatbot that simultaneously supports interaction across various modality combinations.","archived":false,"fork":false,"pushed_at":"2025-06-16T16:52:09.000Z","size":11115,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-16T17:51:56.877Z","etag":null,"topics":["chatbot","gpt-4o","interaction","large-language-models","large-multimodal-models","llama","llm","mutlimodal","speech","speech-recognition","speech-synthesis","vision","vision-language-model"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ictnlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-16T16:24:46.000Z","updated_at":"2025-06-16T17:34:39.000Z","dependencies_parsed_at":"2025-06-16T18:02:47.729Z","dependency_job_id":null,"html_url":"https://github.com/ictnlp/Stream-Omni","commit_stats":null,"previous_names":["ictnlp/stream-omni"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ictnlp/Stream-Omni","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictnlp%2FStream-Omni","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictnlp%2FStream-Omni/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictnlp%2FStream-Omni/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictnlp%2FStream-Omni/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ictnlp","download_url":"https://codeload.github.com/ictnlp/Stream-Omni/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictnlp%2FStream-Omni/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260862924,"owners_count":23074181,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbot","gpt-4o","interaction","large-language-models","large-multimodal-models","llama","llm","mutlimodal","speech","speech-recognition","speech-synthesis","vision","vision-language-model"],"created_at":"2025-06-20T02:02:07.506Z","updated_at":"2026-04-27T18:32:00.196Z","avatar_url":"https://github.com/ictnlp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model\n\n[![arXiv](https://img.shields.io/badge/arXiv-2506.13642-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2506.13642)\n[![model](https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface%20-stream--omni--8b-orange.svg)](https://huggingface.co/ICTNLP/stream-omni-8b)\n[![data](https://img.shields.io/badge/%F0%9F%93%91%20Datasets%20-InstructOmni-green.svg)](https://huggingface.co/datasets/ICTNLP/InstructOmni)\n[![Badge](https://hitscounter.dev/api/hit?url=https%3A%2F%2Fgithub.com%2Fictnlp%2FStream-Omni\u0026label=Visitors\u0026icon=graph-up\u0026color=%23dc3545)](https://github.com/ictnlp/Stream-Omni)\n\n\u003e [**Shaolei Zhang**](https://zhangshaolei1998.github.io/), [**Shoutao Guo**](https://scholar.google.com.hk/citations?user=XwHtPyAAAAAJ), [**Qingkai Fang**](https://fangqingkai.github.io/), [**Yan Zhou**](https://zhouyan19.github.io/zhouyan/), [**Yang Feng**](https://people.ucas.edu.cn/~yangfeng?language=en)\\*\n\n\nStream-Omni is a GPT-4o-like language-vision-speech chatbot that simultaneously supports interaction across various modality combinations, with the following features💡:\n- **Omni Interaction**: Support multimodal inputs including text, vision, and speech, and generate both text and speech responses.\n- **Seamless \"see-while-hear\" Experience**: Simultaneously output *intermediate textual results* (e.g., ASR transcriptions and model responses) during speech interactions, like the advanced voice service of GPT-4o.\n- **Efficient Training**: Require only a small amount of omni-modal data for training.\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"./assets/stream-omni.png\" alt=\"stream-omni\" style=\"width: 90%; min-width: 300px; display: block; margin: auto;\"\u003e\n\u003c/p\u003e\n\n## 🖥 Demo\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n🎧 Vision-grounded Speech Interaction (simultaneously produce intermediate text) 🎧\n\u003c/p\u003e\n\nhttps://github.com/user-attachments/assets/25807982-aa95-4633-9e92-10d995900258\n\nhttps://github.com/user-attachments/assets/df8d79ba-63db-487c-a4a9-f183372168a1\n\n\u003e [!NOTE]\n\u003e\n\u003e **Stream-Omni can produce intermediate textual results (ASR transcription and text response) during speech interaction, offering users a seamless \"see-while-hear\" experience.**\n\n\n\n\n- Downlaod Stream-Omni model from [here](https://huggingface.co/ICTNLP/stream-omni-8b), put in `${STREAMOMNI_CKPT}`.\n- Downlaod CosyVoice (Tokenizer \u0026 Flow Model) from [here](https://modelscope.cn/models/iic/CosyVoice-300M-25Hz/files), put in `COSYVOICE_CKPT=./CosyVoice-300M-25Hz`:\n    ```python\n    from modelscope import snapshot_download\n    snapshot_download('iic/CosyVoice-300M-25Hz', local_dir='./CosyVoice-300M-25Hz')\n    ```\n- Run these scripts to launch the API and interface, and then interact through the browser (http://localhost:7860):\n    ```bash\n    # controller\n    python stream_omni/serve/controller.py --host 0.0.0.0 --port 10000\n    \n    # CosyVoice worker\n    COSYVOICE_CKPT=path_to_CosyVoice-300M-25Hz # e.g., ./CosyVoice-300M-25Hz\n    WAV_DIR=path_to_save_generated_audio\n    CUDA_VISIBLE_DEVICES=0 PYTHONPATH=CosyVoice/third_party/Matcha-TTS python ./CosyVoice/cosyvoice_worker.py --port 21003 --model ${COSYVOICE_CKPT} --wav_dir ./gen_wavs/\n    \n    # Stream-Omni worker, add --load-8bit for VRAM lower than 32GB \n    STREAMOMNI_CKPT=path_to_stream-omni-8b # e.g., ./stream-omni-8b\n    CUDA_VISIBLE_DEVICES=1  python ./stream_omni/serve/model_worker.py --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path ${STREAMOMNI_CKPT} --model-name stream-omni\n    \n    # Interface\n    python stream_omni/serve/gradio_web.py --controller http://localhost:10000 --model-list-mode reload  --port 7860\n    ```\n- You can also refer to [`api.py`](./api.py) for the usage of API.\n\n## 🔥 Quick Start\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"./assets/model.png\" alt=\"model\" style=\"width: 100%; min-width: 300px; display: block; margin: auto;\"\u003e\n\u003c/p\u003e\n\n\u003e [!Tip]\n\u003e\n\u003e **Stream-Omni achieves modality alignments through sequence-dimension concatenation for vision-text alignment and layer-dimension mapping for speech-text alignment.**\n\n\n\n### Requirements\n\n- Install packages:\n    ```bash\n    conda create -n streamomni python=3.10 -y\n    conda activate streamomni\n    pip install -e .\n    pip install flash-attn --no-build-isolation\n    pip install -r requirements.txt\n    pip install -r CosyVoice/requirements.txt\n    ```\n### Command Interaction\n- Run these scripts for vision-grounded speech interaction:\n    ```bash\n    export CUDA_VISIBLE_DEVICES=0\n    export PYTHONPATH=CosyVoice/third_party/Matcha-TTS\n    \n    STREAMOMNI_CKPT=path_to_stream-omni-8b\n    \n    # Replace the path of cosyvoice model in run_stream_omni.py (e.g., cosyvoice = CosyVoiceModel('./CosyVoice-300M-25Hz')) \n    # add --load-8bit for VRAM lower than 32GB \n    python ./stream_omni/eval/run_stream_omni.py \\\n        --model-path ${STREAMOMNI_CKPT} \\\n        --image-file ./stream_omni/serve/examples/cat.jpg --conv-mode stream_omni_llama_3_1 --model-name stream-omni  \\\n        --query ./stream_omni/serve/examples/cat_color.wav\n    ```\n    \n    You should get the following outputs:\n    \n    ```yaml\n    ASR Outputs:\n    What is the color of the cat\n    LLM Outputs:\n    The cat is gray and black.\n    Speech Tokens:\n    \u003cAudio_2164\u003e\u003cAudio_2247\u003e\u003cAudio_671\u003e\u003cAudio_246\u003e\u003cAudio_2172\u003e\u003cAudio_1406\u003e\u003cAudio_119\u003e\u003cAudio_203\u003e\u003cAudio_2858\u003e\u003cAudio_2099\u003e\u003cAudio_1716\u003e\u003cAudio_22\u003e\u003cAudio_1736\u003e\u003cAudio_1038\u003e\u003cAudio_4082\u003e\u003cAudio_1655\u003e\u003cAudio_2409\u003e\u003cAudio_2104\u003e\u003cAudio_571\u003e\u003cAudio_2255\u003e\u003cAudio_73\u003e\u003cAudio_760\u003e\u003cAudio_822\u003e\u003cAudio_701\u003e\u003cAudio_2583\u003e\u003cAudio_1038\u003e\u003cAudio_2203\u003e\u003cAudio_1185\u003e\u003cAudio_2103\u003e\u003cAudio_1718\u003e\u003cAudio_2610\u003e\u003cAudio_1883\u003e\u003cAudio_16\u003e\u003cAudio_792\u003e\u003cAudio_8\u003e\u003cAudio_8\u003e\u003cAudio_535\u003e\u003cAudio_67\u003e\n    Speech Outputs:\n    Audio saved at ./output_893af1597afe2551d76c37a75c813b16.wav\n    ```\n    \n- Interaction across various modality combinations:\n\n    | Inputs                    | Outputs | Intermediate Outputs                                         | Scripts                                                       |\n    | ------------------------- | ------- | ------------------------------------------------------------ | ------------------------------------------------------------ |\n    | Text + Vision (or None)   | Text    | /                                                            | [`run_stream_omni_t2t.py`](./stream_omni/eval/run_stream_omni_t2t.py) |\n    | Text + Vision (or None)   | Speech  | Text result of model outputs                                 | [`run_stream_omni_t2s.py`](./stream_omni/eval/run_stream_omni_t2s.py) |\n    | Speech + Vision (or None) | Text    | ASR transciption of user inputs                              | [`run_stream_omni_s2t.py`](./stream_omni/eval/run_stream_omni_s2t.py) |\n    | Speech + Vision (or None) | Speech  | Text result of model outputs, ASR transciption of user inputs | [`run_stream_omni_s2s.py`](./stream_omni/eval/run_stream_omni_s2s.py) |\n\n    \u003e Control the interaction mode via `inference_type` in `model.generate()` (select from `text_to_text`, `text_to_speech`, `speech_to_text`, `speech_to_speech`)\n\n### Evaluation\n- Refer to [`./scripts/stream_omni/`](./scripts/stream_omni/) for evaluation scripts.\n\n## 🤝 Acknowledgement\n- [LLaVA](https://github.com/haotian-liu/LLaVA)/[LLaVA-NeXT](https://github.com/LLaVA-VL/LLaVA-NeXT)/[LLaVA-OneVision](https://github.com/LLaVA-VL/LLaVA-NeXT): Stream-Omni is built upon the LLaVA and LLaVA-NeXT codebases and incorporates image instruction data from LLaVA-OneVision.\n- [CosyVoice](https://github.com/FunAudioLLM/CosyVoice): Stream-Omni uses the tokenizer and flow model of CosyVoice.\n- [UltraEval-Audio](https://github.com/OpenBMB/UltraEval-Audio): Some normalization processing during evaluation refer to UltraEval-Audio.\n- [VisIT-Bench](https://visit-bench.github.io/): Stream-Omni constructs SpokenVisIT benchmark based on VisIT-Bench for the evaluation of vision-grounded speech interaction.\n\n\n## 🖋Citation\n\nIf this repository is useful for you, please cite as:\n\n```\n@misc{streamomni,\n      title={Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model}, \n      author={Shaolei Zhang and Shoutao Guo and Qingkai Fang and Yan Zhou and Yang Feng},\n      year={2025},\n      eprint={2506.13642},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2506.13642}, \n}\n```\n\nIf you have any questions, please feel free to submit an issue or contact `zhangshaolei20z@ict.ac.cn`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fictnlp%2Fstream-omni","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fictnlp%2Fstream-omni","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fictnlp%2Fstream-omni/lists"}