{"id":26923827,"url":"https://github.com/nighthawk42/morpheus","last_synced_at":"2025-04-02T01:42:49.427Z","repository":{"id":283618640,"uuid":"952347678","full_name":"Nighthawk42/mOrpheus","owner":"Nighthawk42","description":"Whisper STT + Orpheus TTS  + Gemma 3 using LM Studio to create a virtual assistant. ","archived":false,"fork":false,"pushed_at":"2025-03-31T22:33:13.000Z","size":90,"stargazers_count":36,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-31T23:27:49.191Z","etag":null,"topics":["ai","gemma3","llm","lmstudio","orpheus-tts","virtualassistant","whisper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Nighthawk42.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-21T06:12:13.000Z","updated_at":"2025-03-31T22:33:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"ea344082-8c29-4af4-bde3-20eef245c0ad","html_url":"https://github.com/Nighthawk42/mOrpheus","commit_stats":null,"previous_names":["nighthawk42/morpheus"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nighthawk42%2FmOrpheus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nighthawk42%2FmOrpheus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nighthawk42%2FmOrpheus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nighthawk42%2FmOrpheus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Nighthawk42","download_url":"https://codeload.github.com/Nighthawk42/mOrpheus/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246741185,"owners_count":20826063,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","gemma3","llm","lmstudio","orpheus-tts","virtualassistant","whisper"],"created_at":"2025-04-02T01:42:48.850Z","updated_at":"2025-04-02T01:42:49.415Z","avatar_url":"https://github.com/Nighthawk42.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Status Update\n\nI'm finishing up the last month or so of university, so my progression will likely be non-existent on this. I suggest using the latest Dev tree build. \n\n# ------------\n# mOrpheus Virtual Assistant Demo\n\nThis project implements the mOrpheus Virtual Assistant, which integrates speech recognition, text generation, and text-to-speech (TTS) synthesis. The assistant leverages several state-of-the-art components:\n\n- **Whisper** for speech recognition.\n- **LM Studio API** for both text generation (chat) and text-to-speech synthesis.\n- **SNAC-based decoder** to convert TTS token streams into PCM audio.\n\n## Features\n\n- **Speech Recognition:**  \n  Captures audio from the microphone and transcribes it using the Whisper model.\n\n- **Text Generation:**  \n  Uses LM Studio’s chat API to generate natural language responses based on transcribed input.\n\n- **Text-to-Speech:**  \n  Converts the generated text into speech via LM Studio’s TTS API. The text is cleaned (removing newlines, markdown symbols, and emojis) before being sent to the TTS engine, and the resulting token stream is decoded using a SNAC-based decoder.\n  \n\n- **Audio Playback \u0026 Activation:**  \n  Plays the generated audio and waits for the next activation. Long responses are segmentented then recombined at playback. The assistant supports both push‑to‑talk (keypress) and hotword (“Hey Assistant”) activation, or both simultaneously, as configured.\n\n## Requirements\n\n- Python 3.7+\n- [PyTorch](https://pytorch.org/)\n- [Whisper](https://github.com/openai/whisper)\n- [sounddevice](https://python-sounddevice.readthedocs.io/)\n- [scipy](https://www.scipy.org/)\n- [numpy](https://numpy.org/)\n- [requests](https://docs.python-requests.org/)\n- [PyYAML](https://pyyaml.org/)\n- [Transformers](https://huggingface.co/transformers/)\n- [SNAC](https://github.com/hubertsiuzdak/snac)\n\n## Setup\n\n1. **Clone the repository:**\n\n   ```bash\n   git clone https://github.com/yourusername/morpheus-virtual-assistant.git\n   cd morpheus-virtual-assistant\n   ```\n\n2. **Install the dependencies:**\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n   *Ensure that your `requirements.txt` includes all required packages.*\n\n3. **Configure the application:**\n\n   - Create a `settings.yml` file in the project root.\n   - Populate it with your configuration details for Whisper, LM Studio API, audio settings, interaction mode, etc.\n\n   Example `settings.yml`:\n\n   ```yaml\n   # -------------------------------\n   # Configuration for Whisper STT\n   # -------------------------------\n   whisper:\n     model: \"small.en\"\n     sample_rate: 16000\n\n   # -------------------------------\n   # Configuration for LM Studio (Chat \u0026 TTS)\n   # -------------------------------\n   lm:\n     api_url: \"http://127.0.0.1:1234/v1\"\n     \n     chat:\n       endpoint: \"/chat/completions\"\n       model: \"gemma-3-12b-it\"\n       system_prompt: \"You are a helpful assistant.\"\n       max_tokens: 256\n       temperature: 0.7\n       top_p: 0.9\n       repetition_penalty: 1.1\n       max_response_time: 10.0\n\n     tts:\n       endpoint: \"/completions\"\n       model: \"orpheus-3b-ft.gguf@q2_k\"\n       default_voice: \"tara\"\n       max_tokens: 4096\n       temperature: 0.6\n       top_p: 0.9\n       repetition_penalty: 1.0\n       speed: 1.0\n       max_segment_duration: 20\n\n   # -------------------------------\n   # TTS Audio Output Configuration\n   # -------------------------------\n   tts:\n     sample_rate: 24000\n\n   # -------------------------------\n   # Audio Device Configuration\n   # -------------------------------\n   audio:\n     input_device: null\n     output_device: null\n     hotword_sample_rate: 16000\n\n   # -------------------------------\n   # Voice Activity Detection (VAD) Configuration\n   # -------------------------------\n   vad:\n     mode: 2\n     frame_duration_ms: 30\n     silence_threshold_ms: 1000\n     min_record_time_ms: 2000\n\n   # -------------------------------\n   # Hotword Detection Configuration\n   # -------------------------------\n   hotword:\n     enabled: true\n     phrase: \"Hey Assistant\"\n     sensitivity: 0.7\n     timeout_sec: 5\n     retries: 3\n\n   # -------------------------------\n   # Segmentation Configuration for TTS\n   # -------------------------------\n   segmentation:\n     max_words: 60\n\n   # -------------------------------\n   # Speech Quality Configuration\n   # -------------------------------\n   speech:\n     normalize_audio: false\n     default_pitch: 0\n     min_speech_confidence: 0.5\n     max_retries: 3\n\n   # -------------------------------\n   # Interaction Configuration\n   # -------------------------------\n   interaction:\n     mode: \"both\"  # Options: \"push_to_talk\", \"hotword\", or \"both\"\n     post_audio_delay: 0.5\n   ```\n\n4. **Run LM Studio**\nBefore activating the assistant you need to have LM Studio running both the LLM and Orpheus model as defined in the settings.yml in API mode. This is only accessibly in Power User or Developer Mode respectively.\n\n5. **Run the Assistant:**\n\n   ```bash\n   python morpheus.py\n   ```\n\n\n## Suggested Models\n\nThese are the models I tested with, your milage may vary with additional models.\n\n**LLM Models:**\n- gemma-3-12b-it-GGUF/gemma-3-12b-it-Q3_K_L.gguf\n\n**Orpheus Models:**\n- lex-au/Orpheus-3b-FT-Q2_K.gguf - Fastest inference (~50% faster tokens/sec than Q8_0).\n- lex-au/Orpheus-3b-FT-Q4_K_M.gguf - Balanced quality/speed.\n- lex-au/Orpheus-3b-FT-Q8_0.gguf - Original high-quality model.\n\n## Usage\n\n- **Activation:**  \n  The assistant listens for activation either via a hotword (\"Hey Assistant\") or a push-to-talk keypress (or both, depending on your settings).  \n- **Speech Processing:**  \n  It records your speech, transcribes it using Whisper, and generates a text response via LM Studio’s chat API.\n- **TTS Synthesis:**  \n  The response is cleaned to remove unwanted characters (e.g., emojis, newlines, markdown formatting) and then sent to LM Studio’s TTS API. The SNAC-based decoder converts the TTS token stream into PCM audio.\n- **Audio Playback:**  \n  The generated audio is played back, and a brief delay is applied before the assistant waits for the next activation.\n\n## Contributing\n\nFeel free to open issues or submit pull requests with improvements or bug fixes. I'm fairly new at this and feel this could see a lot of improvements.\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n\n## Notice\n\nSee the [NOTICE](NOTICE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnighthawk42%2Fmorpheus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnighthawk42%2Fmorpheus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnighthawk42%2Fmorpheus/lists"}