{"id":51176499,"url":"https://github.com/dashed/whisperx-subtitles-replicate","last_synced_at":"2026-06-27T04:03:51.331Z","repository":{"id":366473952,"uuid":"865750398","full_name":"dashed/whisperx-subtitles-replicate","owner":"dashed","description":"Generates synchronized, readable SRT subtitles from transcribed audio with WhisperX (faster-whisper-large-v3)","archived":false,"fork":false,"pushed_at":"2026-06-22T01:11:16.000Z","size":282,"stargazers_count":13,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-22T03:09:35.858Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://replicate.com/dashed/whisperx-subtitles-replicate","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dashed.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-01T04:20:57.000Z","updated_at":"2026-05-14T18:11:32.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dashed/whisperx-subtitles-replicate","commit_stats":null,"previous_names":["dashed/whisperx-subtitles-replicate"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/dashed/whisperx-subtitles-replicate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dashed%2Fwhisperx-subtitles-replicate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dashed%2Fwhisperx-subtitles-replicate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dashed%2Fwhisperx-subtitles-replicate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dashed%2Fwhisperx-subtitles-replicate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dashed","download_url":"https://codeload.github.com/dashed/whisperx-subtitles-replicate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dashed%2Fwhisperx-subtitles-replicate/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34840912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-27T02:00:06.362Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-27T04:03:50.700Z","updated_at":"2026-06-27T04:03:51.326Z","avatar_url":"https://github.com/dashed.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# whisperx-subtitles-replicate\n\n\u003e Generates synchronized, readable SRT subtitles from transcribed audio with WhisperX (faster-whisper-large-v3)\n\nThis a fork of:\n\n- https://github.com/victor-upmeet/whisperx-replicate\n- https://replicate.com/victor-upmeet/whisperx\n\nThis script processes transcribed audio data to generate properly formatted subtitle files (`.srt`). It handles the splitting and merging of subtitle cues based on duration, line length, line count, and desired reading speed, ensuring that the resulting subtitles are readable and synchronized with the audio.\n\nCode based on: https://github.com/m-bain/whisperX/issues/883\n\nHere’s a high-level overview of how it achieves this:\n\n1. **Generate transcription:** Uses WhisperX (faster-whisper-large-v3) to generate transcription with word-level timestamps.\n2. **Sentence Segmentation:** Splits the transcribed text into sentences with `PySBD` (Python Sentence Boundary Disambiguation) for the languages it supports; for languages it does not (e.g. Thai), it falls back to the neural [`SaT`/`wtpsplit`](https://github.com/segment-any-text/wtpsplit) segmenter, which handles non-space-delimited and unpunctuated scripts.\n3. **Initial Cue Creation:** For each sentence, the script creates an initial subtitle cue, including start and end times based on word-level timings.\n4. **Cue Merging:**\n   - Merges short cues that don't meet a minimum duration (e.g., 3 seconds) to ensure subtitles are displayed long enough for viewers to read.\n   - Merges are performed without exceeding maximum line lengths or line counts.\n   - Considers time gaps between cues to avoid merging cues that are too far apart in time.\n5. **Cue Splitting:**\n   - Splits long cues that exceed maximum line lengths or line counts into smaller cues.\n   - Uses word-level timings to maintain accurate synchronization when splitting.\n   - Avoids splitting in the middle of words or sentences when possible.\n6. **Adjusting Cue Durations:**\n   - Adjusts cue durations based on the desired words per second (e.g., 4 WPS) for comfortable reading.\n   - Ensures that cue durations are not shorter than the minimum duration and do not exceed the maximum duration.\n   - Re-adjusts durations after merging to match the new speech/reading rate.\n7. **Handling Time Gaps:**\n   - When merging cues or chunks, the script considers the time gap between them.\n   - Avoids merging cues/chunks if the time gap exceeds a maximum acceptable duration (e.g., 1.5 seconds).\n8. **SRT File Generation:** Formats each cue according to the SRT specification, including numbering, timing, and text formatting.\n\n# Development\n\nThis project uses [uv](https://docs.astral.sh/uv/) for dependency management,\n[ruff](https://docs.astral.sh/ruff/) for linting/formatting, and\n[ty](https://docs.astral.sh/ty/) for type checking. `pyproject.toml` is the\nsingle source of truth for dependencies.\n\n## Dependencies\n\nThe runtime stack (`torch`, `whisperx`, `pyannote.audio`, ...) targets CUDA 12.8\non Linux/x86_64 — the Replicate GPU image — and has no macOS wheels, so the\nlockfile is resolved for that platform only. Cog still builds from\n`requirements.txt`, which is generated from the lock:\n\n```sh\nuv lock                  # resolve / update uv.lock from pyproject.toml\nuv export --frozen --no-dev --no-emit-project --no-hashes \\\n  --format requirements-txt -o requirements.txt\n```\n\nAfter exporting, re-add the `--extra-index-url https://download.pytorch.org/whl/cu128`\nline near the top of `requirements.txt` (uv omits explicit indexes on export, and\npip needs it to find the `+cu128` torch wheels). See the header in that file.\n\n## Lint, format and type-check\n\nThese run without installing the heavy GPU stack, so they work on any machine:\n\n```sh\nuvx ruff check .          # lint\nuvx ruff format .         # format\nuvx ty check              # type check\n```\n\n(On the Linux GPU image, where the full dependencies are installed, you can also\nrun them via `uv run ruff ...` / `uv run ty check` with full import resolution.)\n\n## Tests\n\nThe subtitle/formatting logic lives in the `whisperx_subtitles` package and is\ndeliberately free of the GPU stack, so the test suite runs on any machine\nwithout installing torch/whisperx:\n\n```sh\nuvx --with pysbd --with ffmpeg-python --with numpy pytest   # any machine\nuv run pytest                                                # on the Linux image (full env)\n```\n\n## Project layout\n\n```\npredict.py                     # Cog entry point: Runner (run/setup) + Output (thin glue)\nwhisperx_subtitles/\n  config.py                    # runtime + subtitle-formatting constants (line length, CPS, durations)\n  types.py                     # Word / Segment / Cue TypedDicts\n  subtitles.py                 # pure subtitle logic (split, merge, timing normalization, SRT)\n  audio.py                     # ffmpeg probing + pure segment-timing math\n  transcription.py             # whisperx glue: language detection, alignment, diarization\ntests/                         # pytest suite for the pure modules + mocked pipeline\n```\n\n## Subtitle formatting\n\nReadability follows EBU-TT / Netflix-style guidelines, all configurable as model\ninputs (`max_line_length`, `max_lines`, `max_cps`, `min_duration`, `max_duration`;\ndefaults in `config.py`): max characters per line, max 2 lines, a\ncharacters-per-second reading-speed ceiling, and min/max on-screen duration. The\n`normalize_cues` pass guarantees cues are ordered and non-overlapping and don't\nlinger far past the spoken audio. When `diarization` is enabled, each cue is\nprefixed with its `[SPEAKER_xx]` label.\n\n## Multilingual\n\nTranscription (Whisper large-v3) covers ~99 languages. Word-level timestamps —\nwhich the subtitle engine needs to time cues — are produced by forced alignment:\n\n- **WhisperX's built-in alignment** (wav2vec2) covers ~41 languages natively.\n- For any **other** language, the model falls back to **MMS forced alignment**\n  (via [`ctc-forced-aligner`](https://github.com/MahmoudAshraf97/ctc-forced-aligner),\n  1000+ languages) so those languages still get word-level cue timing instead of\n  segment-level only. If alignment fails for any reason it degrades gracefully to\n  segment-level timing — it never breaks a request.\n\n\u003e ⚠️ **License:** the MMS alignment weights\n\u003e ([`MahmoudAshraf/mms-300m-1130-forced-aligner`](https://huggingface.co/MahmoudAshraf/mms-300m-1130-forced-aligner),\n\u003e derived from Meta MMS) are **CC-BY-NC 4.0 (non-commercial)**. As of 2026 there is\n\u003e no permissively-licensed broad multilingual aligner. If you need commercial use,\n\u003e swap in a permissively-licensed per-language CTC model for the languages you care\n\u003e about. The SaT segmenter weights (`sat-3l-sm`) and `wtpsplit` are MIT-licensed.\n\nKnown limitation: for scripts with no word spacing (Thai, Lao, Khmer, Burmese),\nASR output is often a single unsegmented run, so MMS produces one timestamp span\nrather than true per-word timing. Per-word timing there needs a word segmenter\n(e.g. `pythainlp`), which is out of scope for now.\n\n## Translation\n\nSet the `translate_to` input (an ISO code like `en`, `es`, `ja`) to output\nsubtitles in a different language than the audio. Naive per-cue translation is\ndeliberately **not** used — it loses sentence context and breaks on word-order\ndifferences (e.g. Korean SOV → English SVO) and length changes. Instead the model\nruns a **cascade**:\n\n1. Transcribe + align in the **source** language → word-level timestamps.\n2. Reconstruct full **sentences** with their `[start, end]` spans.\n3. Translate each whole sentence (context preserved) with\n   [MADLAD-400](https://huggingface.co/google/madlad400-3b-mt) (Apache-2.0, 400+\n   languages, any→any).\n4. Re-segment each translation into cues and distribute the source sentence's\n   time span across them **proportionally by character count**, so the\n   translation stays synced to the audio.\n\nThe translation model is loaded lazily (only when `translate_to` is set), so\nnormal transcription requests don't pay its cost. For higher translation quality\nyou can swap `MT_MODEL` in `config.py` for a larger MADLAD or an LLM translator.\n\nLimitation: translation re-timing is proportional, not word-aligned (the\ntranslated words aren't in the audio); if a translated sentence is much longer\nthan its source span, reading speed may exceed the target CPS.\n\n## Download models\n\n```sh\n./build.sh\n```\n\nThe multilingual model weights (SaT segmenter and the MMS alignment model) are\nnot fetched by `build.sh` — they are pre-cached into the image at build time by\nthe `build.run` steps in `cog.yaml`, so cold starts don't download them.\n\n## Publish to cog\n\n```sh\ncog login\ncog push r8.im/dashed/whisperx-subtitles-replicate\n```\n\n# Usage\n\nExtract audio with:\n\n```\nffmpeg -i input_video.mp4 -vn -acodec pcm_s16le -ar 16000 -ac 1 output_audio.wav\nffmpeg -i input_video.mp4 -vn -acodec aac -b:a 192k output_audio.m4a\n```\n\n# Model Information\n\nWhisperX provides fast automatic speech recognition (70x realtime with large-v3) with word-level timestamps and speaker diarization.\n\nWhisper is an ASR model developed by OpenAI, trained on a large dataset of diverse audio. Whilst it does produces highly accurate transcriptions, the corresponding timestamps are at the utterance-level, not per word, and can be inaccurate by several seconds. OpenAI’s whisper does not natively support batching, but WhisperX does.\n\nModel used is for transcription is large-v3 from faster-whisper.\n\nFor more information about WhisperX, including implementation details, see the [WhisperX github repo](https://github.com/m-bain/whisperX).\n\n## Diarization\n\nWhen `diarization` is enabled, WhisperX uses pyannote's\n[`pyannote/speaker-diarization-community-1`](https://huggingface.co/pyannote/speaker-diarization-community-1)\npipeline. You must accept that model's user agreement on Hugging Face and pass a\nread token via `huggingface_access_token`.\n\n# Citation\n\n```\n@misc{bain2023whisperx,\n      title={WhisperX: Time-Accurate Speech Transcription of Long-Form Audio},\n      author={Max Bain and Jaesung Huh and Tengda Han and Andrew Zisserman},\n      year={2023},\n      eprint={2303.00747},\n      archivePrefix={arXiv},\n      primaryClass={cs.SD}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdashed%2Fwhisperx-subtitles-replicate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdashed%2Fwhisperx-subtitles-replicate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdashed%2Fwhisperx-subtitles-replicate/lists"}