{"id":29246577,"url":"https://github.com/xorengine/marvin4000","last_synced_at":"2026-05-09T07:02:11.843Z","repository":{"id":302347479,"uuid":"1001100406","full_name":"XOREngine/Marvin4000","owner":"XOREngine","description":"Real-time audio translation using Whisper + SeamlessM4T / NLLB-200","archived":false,"fork":false,"pushed_at":"2025-07-01T21:53:34.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-01T22:29:55.868Z","etag":null,"topics":["ai","asr","audio-processing","consumer-hardware","cuda","gpu-accelerated","machine-learning","multilingual","nllb","nmt","pytorch","real-time","seamlessm4t","speech-recognition","transcription","translation","whisper"],"latest_commit_sha":null,"homepage":"https://xorengine.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/XOREngine.png","metadata":{"files":{"readme":"README.es.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-12T20:28:17.000Z","updated_at":"2025-07-01T21:56:57.000Z","dependencies_parsed_at":"2025-07-01T22:40:39.072Z","dependency_job_id":null,"html_url":"https://github.com/XOREngine/Marvin4000","commit_stats":null,"previous_names":["xorengine/marvin4000"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/XOREngine/Marvin4000","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XOREngine%2FMarvin4000","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XOREngine%2FMarvin4000/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XOREngine%2FMarvin4000/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XOREngine%2FMarvin4000/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/XOREngine","download_url":"https://codeload.github.com/XOREngine/Marvin4000/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XOREngine%2FMarvin4000/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263416475,"owners_count":23463144,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","asr","audio-processing","consumer-hardware","cuda","gpu-accelerated","machine-learning","multilingual","nllb","nmt","pytorch","real-time","seamlessm4t","speech-recognition","transcription","translation","whisper"],"created_at":"2025-07-03T23:02:12.802Z","updated_at":"2026-05-09T07:02:06.794Z","avatar_url":"https://github.com/XOREngine.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- Marvin4000 - Real-time Audio Transcription \u0026 Translation --\u003e\n\u003c!-- © 2025 XOREngine (WallyByte) --\u003e\n\u003c!-- https://github.com/XOREngine/marvin4000 --\u003e\n\n# Marvin4000\n\n\u003e Transcripción y traducción de audio en tiempo real con Whisper y modelos multilingües (SeamlessM4T / NLLB‑200)\n\n[![License](https://img.shields.io/badge/license-MIT-lightgrey)](LICENSE)\n[![Python](https://img.shields.io/badge/Python-3.10%2B-blue)](https://www.python.org/downloads/)\n[![CUDA](https://img.shields.io/badge/GPU-Accelerated-green)](https://developer.nvidia.com/cuda-toolkit)\n\n**🌐 Idiomas:** [English](README.md) | [Español](README.es.md)\n\n\u003cbr\u003e\n\n**Marvin4000** captura, transcribe y traduce audio del sistema en tiempo real usando hardware local.\n\n\u003cbr\u003e\n\n\u003e ⚠️ **IMPORTANTE:**\n\u003e\n\u003e * Si estás en **Windows**, la captura de audio debe ser implementada manualmente mediante una alternativa a `parec` que proporcione datos de audio del sistema en formato `float32`.\n\n\u003cbr\u003e\n\n## 📊 Rendimiento probado\n\n| GPU \u0026 Modelos usados                                                | Latencia (s) | WER       | BLEU-1/4/Corpus | VRAM        |\n| ---------------------------------------------------------------- | ----------- | --------- | --------------- | ----------- |\n| **RTX 4060 Ti 16GB\u003cbr\u003ewhisper-large-v3-turbo + nllb-200-3.3B** | **2-3**     | **6 %** | **75/38/54**    | **14.2 GB** |\n| RTX 4060 Ti 16GB\u003cbr\u003ewhisper-large-v3-turbo + seamless-m4t-v2-large | 2-3     | 6 % | 74/39/52    | 11.4 GB |\n\n#### Corpus de prueba\n\n* **Audio**: 25 fragmentos aleatorios de audiolibros de [LibriSpeech](https://www.openslr.org/12) (media: 5 min/fragmento)\n* **Transcripción de referencia**: Transcripciones oficiales de LibriSpeech\n* **Traducción de referencia**: Generada con Claude \u0026 GPT y revisada manualmente (Inglés → Español)\n* **Total evaluado**: \\~120 minutos de audio\n\n#### Cálculo de métricas\n\n* **WER**: Calculado con [jiwer](https://github.com/jitsi/jiwer), normalizado para puntuación\n* **BLEU**: Implementación corpus-level con tokenización lowercase, clipping de n-gramas y brevity penalty\n* **BLEU-1/4/Corpus**: Precisión 1-grama / 4-grama / score corpus completo\n* **Latencia**: Medida en condiciones reales con RTX 4060 Ti 16GB y RTX 2060 6GB\n\n#### Limitaciones\n\nAunque las traducciones de referencia son de alta calidad, reconocemos que no son equivalentes a traducciones humanas profesionales. Sin embargo, proveen un estándar consistente para comparar el rendimiento del sistema, siguiendo metodologías similares a las empleadas en evaluaciones como [FLEURS](https://arxiv.org/abs/2205.12446) y [CoVoST 2](https://arxiv.org/abs/2007.10310).\n\n\u003cbr\u003e\n\n## 🚀 Instalación y uso\n\n### Requisitos\n\n```bash\nsudo apt install python3-pip pulseaudio-utils ffmpeg\ngit clone https://github.com/XOREngine/marvin4000.git\ncd marvin4000\npip install -r requirements.txt\n```\n\n### Ejecución básica\n\n```bash\n# 1. Reproducir algún contenido con audio en tu sistema\nvlc video_ejemplo.mp4\n# ffmpeg.ffplay -nodisp -autoexit -ss 1 example.mp3\n# o reproducir audio desde el navegador, etc.\n\n# 2. Detectar dispositivos de audio válidos\npython detect_audio_devices.py\n# Ejemplo salida:\n# $ python marvin4000_seam.py --audio-device \"alsa_output.pci-0000_00_1f.3.analog-stereo.monitor\"\n\n# 3. Iniciar transcripción/traducción con el dispositivo monitor adecuado\npython marvin4000_seam.py --audio-device \"alsa_output.pci-0000_00_1f.3.analog-stereo.monitor\"\n\npython marvin4000_nllb.py --audio-device \"alsa_output.pci-0000_00_1f.3.analog-stereo.monitor\" --asr-lang \"de\" --nmt-source \"deu_Latn\" --nmt-target \"spa_Latn\"\n```\n\n### Configuración de idiomas\n\nMarvin4000 utiliza SeamlessM4T y NLLB‑200 para transcripción y traducción entre más de 100 idiomas. Soporta aplicaciones multilingües en tiempo real.\n\n\u003cbr\u003e\n\n## 🔬 Arquitectura técnica\n\n* **Separación de hilos (Threading)**: Captura de audio | ASR | NMT. Reducción 68% latencia\n* **Cuantización Int8**: Implementación bits-and-bytes para los modelos\n* **VAD inteligente**: WebRTC + segmentación conservadora (1.2s silencio mínimo) + validación lingüística\n* **Memoria eficiente**: Buffer circular + caché de traducciones (similitud 0.95)\n* **Latencia híbrida**: Parciales progresivos (2-3s percibida) con `attention_mask` explícito para mayor control en ASR\n* **Segmentación adaptativa**: Evita fragmentos \u003c0.5s, cortes mínimos 2.5s\n* **Decodificación forzada**: Uso de `forced_decoder_ids` para indicar idioma y tarea a Whisper, mejorando precisión de transcripción\n\n\u003cbr\u003e\n\n### Parámetros de configuración ajustables\n\n\u003e **Nota:** Si experimentas demasiada latencia, puedes reducir `num_beams` o acortar `max_new_tokens`. Esto hará las inferencias más rápidas a costa de una leve pérdida de calidad.\n\n**Segmentación y flujo:**\n\n```python\nTIMEOUT_SEC = 12.0           # Tiempo máximo sin flush\nMIN_SEGMENT_SEC = 0.5        # Mínima duración aceptada de segmento\nMIN_PARTIAL_WORDS = 5        # Palabras mínimas para mostrar parcial\nREUSE_THRESHOLD = 0.95       # Umbral de similitud para cache\nSILENCE_SEC = 0.8            # Silencio requerido para segmentar\nVAD_SILENCE_DURATION_SEC = 1.2\nMIN_CUT_DURATION_SEC = 2.5\nAUDIO_RMS_THRESHOLD = 0.0025 # Nivel mínimo de volumen aceptado\n```\n\n**Inferencia ASR (Whisper):**\n\n```python\ngen = self.asr.generate(\n    feats,\n    attention_mask=attn,\n    forced_decoder_ids=forced,\n    max_length=448,\n    num_beams=3,\n    early_stopping=True,\n    temperature=0.0,\n    repetition_penalty=1.1,\n    no_repeat_ngram_size=3,\n    return_timestamps=False,\n    use_cache=True,\n)\n```\n\n**Inferencia NMT (NLLB-200):**\n\n```python\ngenerated_tokens = self.nmt_model.generate(\n    **inputs,\n    forced_bos_token_id=forced_bos_token_id,\n    max_length=120,              \n    min_length=8,                \n    num_beams=4,                 \n    do_sample=False,             \n    repetition_penalty=1.1,      \n    no_repeat_ngram_size=2,      \n    early_stopping=True,         \n)\n```\n\n### Optimizaciones para hardware potente\n\nPara GPUs con \u003e20GB VRAM (RTX 4090, A40, A100), se pueden implementar **CUDA streams** para paralelización ASR/NMT:\n\n```python\n# Modificaciones sugeridas para hardware potente:\nasr_lock = threading.Lock()     # En lugar de gpu_lock compartido\nnmt_lock = threading.Lock()     # Locks independientes\n\nstream_asr = torch.cuda.Stream()\nstream_nmt = torch.cuda.Stream()\n# Potencial mejora estimada: +15-25% throughput\n```\n\n\u003cbr\u003e\n\n## 📜 Modelos y licencias\n\n* Código Marvin4000: [MIT](LICENSE)\n* Whisper: [MIT](https://github.com/openai/whisper/blob/main/LICENSE) (OpenAI)\n* SeamlessM4T: [CC-BY-NC 4.0](https://github.com/facebookresearch/seamless_communication/blob/main/LICENSE) (Meta AI)\n* NLLB-200: [CC-BY-NC 4.0](https://huggingface.co/facebook/nllb-200-3.3B) (Meta AI)\n\n\u003cbr\u003e\n\n## 🙏 Agradecimientos y referencias\n\n### Modelos y librerías usadas\n\n* [OpenAI Whisper](https://github.com/openai/whisper)\n* [Meta SeamlessM4T](https://github.com/facebookresearch/seamless_communication)\n* [Meta NLLB-200](https://github.com/facebookresearch/fairseq/tree/nllb)\n* [WebRTC VAD](https://webrtc.org/)\n\n### Inspiración técnica y papers\n\n* [ggerganov/whisper.cpp](https://github.com/ggerganov/whisper.cpp) – ejecución tiempo real\n* [TimDettmers/bitsandbytes](https://github.com/TimDettmers/bitsandbytes) – cuantización\n* [guillaumekln/faster-whisper](https://github.com/guillaumekln/faster-whisper) – buffering eficiente\n* [snakers4/silero-vad](https://github.com/snakers4/silero-vad) – VAD optimizado\n* [Whisper: Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356)\n* [SeamlessM4T: Massively Multilingual \u0026 Multimodal Machine Translation](https://arxiv.org/abs/2308.11596)\n* [NLLB-200: No Language Left Behind](https://arxiv.org/abs/2207.04672)\n* [Efficient Low-Bit Quantization of Transformer-Based Language Models](https://arxiv.org/abs/2305.12889)\n\n---\n\n\u003cbr\u003e\n\nEste proyecto está pensado como una base flexible. Si quieres modificarlo, usarlo de forma creativa, mejorarlo o simplemente adaptarlo a tus necesidades...\n\n\u003e 💪 **Hazlo.**\n\nSi además compartes mejoras o nos mencionas como referencia, será siempre bien recibido 🙌😜.\n\n\u003cbr\u003e\n\n© [XOREngine](https://xorengine.com) · Compromiso open source\n\n\u003cbr\u003e\n\n\u003c!-- keywords: whisper, seamlessM4T, realtime transcription, translation, streaming audio, cuda, multilingual, vad, low latency, NLLB, ASR, NMT --\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxorengine%2Fmarvin4000","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxorengine%2Fmarvin4000","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxorengine%2Fmarvin4000/lists"}