{"id":31034650,"url":"https://github.com/freedomintelligence/s2s-arena","last_synced_at":"2025-09-14T02:46:45.182Z","repository":{"id":268885359,"uuid":"883698431","full_name":"FreedomIntelligence/S2S-Arena","owner":"FreedomIntelligence","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-27T06:26:47.000Z","size":4804,"stargazers_count":16,"open_issues_count":0,"forks_count":0,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-05-27T07:31:29.619Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-11-05T12:24:44.000Z","updated_at":"2025-05-27T06:26:51.000Z","dependencies_parsed_at":"2024-12-19T14:25:05.602Z","dependency_job_id":"d3b8527d-8e40-4dc0-8944-4437ec8cd134","html_url":"https://github.com/FreedomIntelligence/S2S-Arena","commit_stats":null,"previous_names":["freedomintelligence/s2s-bench","freedomintelligence/s2s-arena"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FreedomIntelligence/S2S-Arena","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FS2S-Arena","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FS2S-Arena/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FS2S-Arena/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FS2S-Arena/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/S2S-Arena/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FS2S-Arena/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275054971,"owners_count":25397576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-14T02:00:10.474Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-14T02:46:40.321Z","updated_at":"2025-09-14T02:46:45.161Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# S2S-Arena\r\n[中文版本](./README_zh.md)\r\n\r\nWelcome to our project repository. This project primarily focuses on the reproduction and evaluation of Speech Large Models (SLMs), with a particular emphasis on speech-to-speech (S2S) models that support both speech input and output. The repository includes:\r\n* 1) Reproduction code for various models;\r\n* 2) Arena web code for testing and demonstration;\r\n* 3) Dataset for testing.\r\n\r\nIn existing research, benchmarks for evaluating models’ instruction-following abilities often overlook paralinguistic information in both input and output, and lack direct comparison of speech output across models. To address these issues, we introduce a novel arena-style S2S benchmark that covers multiple real-world task scenarios, using the ELO rating system for performance analysis. Preliminary experiments show that although some models excel in knowledge-intensive tasks, they still face challenges in generating expressive speech. This study provides critical insights for the further development of S2S models and establishes a robust framework for evaluating model performance in both semantic and paralinguistic dimensions.\r\n\r\nWe invite researchers to include their models in our evaluation system. For inquiries, please contact us via issue submission or email: `jeffreyjiang@cuhk.edu.cn`, `bufan@cuhk.edu.cn`.\r\n\r\nIf you have any additional interesting tests, feel free to contact us.\r\n\r\n## Model Reproduction Guide\r\n### Cascade Model\r\nThe Cascade Model consists of three components: ASR, LLMs, and TTS.\r\n* For ASR, we use `whisper-large-v3`;\r\n* For LLMs, we use `gpt-4o-2024-08-06 (text version)`;\r\n* For TTS, we use `CosyVoice-300M-Instruct`.\r\n\r\nThe related code is located in [./CascadeModel](./CascadeModel).\r\n\r\nEnvironment setup for each component is as follows:\r\n#### Whisper-ASR\r\nRefer to the [Whisper model page on Hugging Face](https://huggingface.co/openai/whisper-large-v3) for ASR environment configuration.\r\n\r\n#### GPT-4o-LLMs\r\nTo set up the environment for LLMs, use the following command:\r\n```shell\r\npip install openai==0.28.0\r\n```\r\n\r\n#### CosyVoice-TTS\r\nFor TTS environment configuration, refer to [FunAudio’s official GitHub](https://github.com/FunAudioLLM/CosyVoice).\r\n\r\nMake sure to set `PATH_TO_COSYVOICE` to the path of your CosyVoice code directory to import the required packages. If you encounter a module-not-found error for `Matcha-TTS`, you can resolve it by running:\r\n```shell\r\nexport PYTHONPATH=third_party/Matcha-TTS\r\n```\r\n\r\n### GPT-4o\r\nWe reproduced the `gpt-4o-realtime-preview-2024-10-01` version by calling the API directly. Refer to [./GPT-4o](./GPT-4o) for related code and environment configuration. If you experience issues with model responses or speech format, try using the [conversion code](./GPT-4o/input/convert.py).\r\n\r\n### SpeechGPT\r\nWe started from the [SpeechGPT open-source code](https://github.com/0nutation/SpeechGPT/tree/main/speechgpt) and have extended its inference logic to support batch processing and better integrate with this project’s workflow.  \r\n\r\nThe related code is located in [./SpeechGPT](./SpeechGPT).\r\n\r\n### GLM-4-Voice\r\nWe started from the [GLM-4-Voice open-source code](https://github.com/THUDM/GLM-4-Voice) and have extended its inference logic to support batch processing and better integrate with this project’s workflow.  \r\n\r\nThe related code is located in [./GLM-4-Voice](./GLM-4-Voice).\r\n\r\n### FunaudioLLMs-Qwen72B\r\nWe reproduced two versions of this model: the official version (using Qwen-72B as the LLMs) and a version using GPT-4o. This section introduces the former; the next section introduces the latter.\r\n\r\nThe code using Qwen-72B as the LLMs is in [./Funaudio_qwen](./Funaudio_qwen). To run this code, you will need the following configurations:\r\n\r\n#### SenseVoice\r\nRefer to [SenseVoice’s official GitHub](https://github.com/FunAudioLLM/SenseVoice) for environment setup, and set `'PATH_TO_SENSEVOICE'` in `sensevoice.py`.\r\n\r\n#### Qwen72B\r\nEnvironment installation and obtaining API Key:\r\n1. Install the environment:\r\n   ```shell\r\n   pip install dashscope\r\n   ```\r\n2. Obtain the API Key by:\r\n   * Visiting [Aliyun's website](https://www.aliyun.com/) and logging in.\r\n   * Accessing the console, locating \"Machine Learning\" under \"Products\" or \"Services,\" or directly searching for \"DashScope.\"\r\n   * Navigating to the DashScope service page and following instructions to activate the service.\r\n   * After activation, create a project or application if required.\r\n   * Once created, locate API Key or access key settings in the project or application management interface.\r\n   * Follow the prompts to generate or view your API Key.\r\n\r\n#### CosyVoice\r\nEnvironment setup is the same as in the Cascade Model section.\r\n\r\n### LLaMA-omni\r\nWe modified the invocation method based on the [LLaMA-Omni open-source project](https://github.com/ictnlp/LLaMA-Omni). Follow these steps for setup:\r\n\r\n1. Download and configure the original project.\r\n2. Place `run_arena.sh` in the `./omni_speech/infer` folder, modifying the script with the correct model and dataset paths.\r\n3. Run `run_model.py`.\r\n4. Run `change_filename.py`.\r\n\r\n### Mini-Omni\r\nWe modified the invocation method based on the [Mini-Omni open-source project](https://github.com/gpt-omni/mini-omni). Follow these steps for setup:\r\n\r\nOur modified code can be found in the [./Mini-Omni](Mini-Omni) directory.\r\n1. Download and configure the original project;\r\n2. Place `inference_arena.py` in the folder containing the downloaded data;\r\n3. Configure the file paths in `inference_arena.py`;\r\n4. Run `inference_arena.py`.\r\n\r\n\r\n## Website Code\r\n\r\ncoming soon\r\n\r\n## Dataset\r\n\r\nYou can refer to: [Hugging Face Dataset](https://huggingface.co/datasets/FreedomIntelligence/S2S-Arena)\r\n\r\n## BIb\r\nSince our project is still ongoing and requires the involvement of many friends and partners in the evaluation process, we have decided to publicly release the first draft of our paper to help everyone quickly understand our research progress and encourage participation. However, it is important to note that this is only our preliminary version, not the final draft.\r\n[Our Paper](./S2S_Arena.pdf)\r\n```\r\n@article{jiang2025s2s,\r\n  title={S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following with Paralinguistic Information},\r\n  author={Jiang, Feng and Lin, Zhiyu and Bu, Fan and Du, Yuhao and Wang, Benyou and Li, Haizhou},\r\n  journal={arXiv preprint arXiv:2503.05085},\r\n  year={2025}\r\n}\r\n\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fs2s-arena","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fs2s-arena","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fs2s-arena/lists"}