Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/BingLingGroup/autosub
Command-line utility to transcribe/translate from video/audio/subtitles to subtitles
https://github.com/BingLingGroup/autosub
audio-segmentation baidu-api cloud-speech-api substation-alpha subtitles voice-activity-detection xfyun xunfei-api
Last synced: 8 days ago
JSON representation
Command-line utility to transcribe/translate from video/audio/subtitles to subtitles
- Host: GitHub
- URL: https://github.com/BingLingGroup/autosub
- Owner: BingLingGroup
- License: gpl-2.0
- Fork: true (iWangJiaxiang/autosub)
- Created: 2019-02-10T08:13:05.000Z (almost 6 years ago)
- Default Branch: dev
- Last Pushed: 2023-12-21T09:24:18.000Z (11 months ago)
- Last Synced: 2024-10-01T20:42:33.390Z (about 1 month ago)
- Topics: audio-segmentation, baidu-api, cloud-speech-api, substation-alpha, subtitles, voice-activity-detection, xfyun, xunfei-api
- Language: Python
- Homepage:
- Size: 1.29 MB
- Stars: 1,979
- Watchers: 35
- Forks: 245
- Open Issues: 38
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Autosub
[简体中文](docs/README.zh-Hans.md)
This repo is not the same as [the original autosub repo](https://github.com/agermanidis/autosub).
This repo has been modified by several people. See the [Changelog](CHANGELOG.md).
[autosub icon](docs/icon/autosub.svg) designed by BingLingGroup.
Software: [inkscape](https://inkscape.org/zh/)
Font: [source-han-sans](https://github.com/adobe-fonts/source-han-sans) ([SIL](https://github.com/adobe-fonts/source-han-sans/blob/master/LICENSE.txt))
Color: [Solarized](https://en.wikipedia.org/wiki/Solarized_(color_scheme)#Colors)
### TOC
1. [Description](#description)
2. [License](#license)
3. [Dependencies](#dependencies)
- 3.1 [Optional Dependencies](#optional-dependencies)
- 3.2 [Required Dependencies](#required-dependencies)
4. [Download and Installation](#download-and-installation)
- 4.1 [Branches](#branches)
- 4.2 [Install on Ubuntu](#install-on-ubuntu)
- 4.3 [Install on Windows](#install-on-windows)
5. [Workflow](#workflow)
- 5.1 [Input](#input)
- 5.2 [Split](#split)
- 5.3 [Speech-to-Text/Translation API request](#speech-to-texttranslation-api-request)
- 5.4 [Speech-to-Text/Translation language support](#speech-to-texttranslation-language-support)
- 5.5 [Output](#Output)
6. [Usage](#usage)
- 6.1 [Typical usage](#typical-usage)
- 6.1.1 [Pre-process Audio](#pre-process-audio)
- 6.1.2 [Detect Regions](#detect-regions)
- 6.1.3 [Split Audio](#split-audio)
- 6.1.4 [Transcribe Audio To Subtitles](#transcribe-audio-to-subtitles)
- 6.1.4.1 [Google Speech V2](#google-speech-v2)
- 6.1.4.2 [Google Cloud Speech-to-Text](#google-cloud-speech-to-text)
- 6.1.4.3 [Google speech config](#google-speech-config)
- 6.1.4.4 [Output API full response](#output-api-full-response)
- 6.1.4.5 [Xfyun speech config](#xfyun-speech-config)
- 6.1.4.6 [Baidu speech config](#baidu-speech-config)
- 6.1.5 [Translate Subtitles](#translate-subtitles)
- 6.2 [Options](#Options)
- 6.3 [Internationalization](#internationalization)
7. [FAQ](#FAQ)
- 7.1 [Other APIs supports](#other-apis-supports)
- 7.2 [Batch processing](#batch-processing)
- 7.3 [proxy support](#proxy-support)
- 7.4 [macOS locale issue](#macos-locale-issue)
- 7.5 [Accuracy](#accuracy)
8. [Bugs report](#bugs-report)
9. [Build](#build)
Click up arrow to go back to TOC.
### Description
Autosub is an automatic subtitles generating utility. It can detect speech regions automatically by using Auditok, split the audio files according to regions by using ffmpeg, transcribe speech based on several APIs and translate the subtitles' text by using py-googletrans.
The new features mentioned above are only available in the latest alpha branch. Not available on PyPI or the original repo.
### License
This repo has a different license from [the original repo](https://github.com/agermanidis/autosub).
[GPLv2](LICENSE)
### Dependencies
Autosub depends on these third party softwares or Python site-packages. Much appreciation to all of these projects.
#### Optional dependencies
- [ffmpeg](https://ffmpeg.org/)
- [ffprobe](https://ffmpeg.org/ffprobe.html)
- [langcodes](https://github.com/LuminosoInsight/langcodes)
- [ffmpeg-normalize](https://github.com/slhck/ffmpeg-normalize)
- [python-Levenshtein](https://github.com/ztane/python-Levenshtein)(Used by [fuzzywuzzy](https://github.com/seatgeek/fuzzywuzzy))
For windows user:
- [Build Tools for Visual Studio 2019](https://visualstudio.microsoft.com/downloads/)
- Used by [marisa-trie](https://github.com/pytries/marisa-trie) when installing.
- [marisa-trie](https://github.com/pytries/marisa-trie) is the dependency of the [langcodes](https://github.com/LuminosoInsight/langcodes).
- Probable components installation: MSVC VS C++ build tools, windows 10 SDK.
#### Required dependencies
- [auditok](https://github.com/amsehili/auditok)
- [pysubs2](https://github.com/tkarabela/pysubs2)
- [wcwidth](https://github.com/jquast/wcwidth)
- [requests](https://github.com/psf/requests)
- [fuzzywuzzy](https://github.com/seatgeek/fuzzywuzzy)
- [progressbar2](https://github.com/WoLpH/python-progressbar)
- [websocket-client](https://github.com/websocket-client/websocket-client)
- [py-googletrans](https://github.com/ssut/py-googletrans)
[requirements.txt](requirements.txt).
About how to install these dependencies, see [Download and Installation](#download-and-installation).
### Download and Installation
Except the PyPI version, others include non-original codes not from the original repository.
0.4.0 > autosub
- These versions are only compatible with Python 2.7.
0.5.6a >= autosub >= 0.4.0
- These versions are compatible with both Python 2.7 and Python 3. It doesn't matter if you change the Python version in the installation commands below.
autosub >= 0.5.7a
- These versions are only compatible with Python 3.
ffmpeg, ffprobe, ffmpeg-normalize need to be put on one of these places to let the autosub detect and use them. The following codes are in the [constants.py](autosub/constants.py). Priority is determined in order.
1. Set the following environment variables before running the program: `FFMPEG_PATH`, `FFPROBE_PATH` and `FFMPEG_NORMALIZE_PATH`. It will override the ones located at the environment variable `PATH`. This will be helpful if you don't want to use the one in the `PATH`.
2. Add them to the environment variable `PATH`. No need to worry about if using package manager to install such as using pip to install ffmpeg-normalize and using chocolatey to install ffmpeg.
3. Add them to the same directory as the autosub executable.
4. Add them to the current command line working directory.
About the git installation. If you don't want to install git to use pip [VCS](https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support) support to install python package or just confused with git environment variables, you can manually click that clone and download button to download the source code and use pip to install the source code [locally](https://pip.pypa.io/en/stable/reference/pip_install/#description) by input these commands.
```batch
cd the_directory_contains_the_source_code
pip install .
```
Due to the autosub PyPI project is maintained by the original autosub repo's owner, I can't modify it or upload a project with the same name. Perhaps later when this version of autosub becomes stabler, I will rename and duplicate this repo and then upload it to PyPI.
#### Branches
[alpha branch](https://github.com/BingLingGroup/autosub/tree/alpha)
- Include many changes from [the original repo](https://github.com/agermanidis/autosub). Details in [Changelog](CHANGELOG.md). Codes will be updated when an alpha version have been released. It is stabler than the dev branch
[origin branch](https://github.com/BingLingGroup/autosub/tree/origin)
- Include the least changes from [the original repo](https://github.com/agermanidis/autosub) except all new features in the [alpha branch](https://github.com/BingLingGroup/autosub/tree/alpha). The changes in [origin branch](https://github.com/BingLingGroup/autosub/tree/dev) just make sure there's no critical bugs when the program is running on Windows. Currently isn't maintained.
[dev branch](https://github.com/BingLingGroup/autosub/tree/dev)
- The latest codes will be pushed to this branch. If it works fine, it will be merged to alpha branch when new version is released.
- Only used to test or pull request. Don't install them unless you know what you are doing.
#### Install on Ubuntu
Include dependencies installation commands.
Install from `alpha` branch.(latest autosub alpha release)
```bash
apt install ffmpeg python3 python3-dev curl git -y
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py
pip install git+https://github.com/BingLingGroup/autosub.git@alpha ffmpeg-normalize langcodes
```
Install from `dev` branch.(latest autosub dev version)
```bash
apt install ffmpeg python3 python3-dev curl git -y
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py
pip install git+https://github.com/BingLingGroup/autosub.git@dev ffmpeg-normalize langcodes
```
Install from `origin` branch.(autosub-0.4.0a)
```bash
apt install ffmpeg python python-pip git -y
pip install git+https://github.com/BingLingGroup/autosub.git@origin
```
Install from PyPI.(autosub-0.3.12)
```bash
apt install ffmpeg python python-pip -y
pip install autosub
```
Recommend using `python3` and `python-pip3` instead of `python` and `python-pip` after autosub-0.4.0.
#### Install on Windows
You can just go to the [release page](https://github.com/BingLingGroup/autosub/releases) and download the latest release(standalone version) for Windows. Thre release version can run without the python environment. The click-and-run batches are also in the package. You can manually edit by using Notepad++. Or add the executable files' directory to system environment variables so you can use it as a universal command everywhere in the system if permission is Ok.
Tips: `Shift - Right Click` is the keyboard shortcut for opening a Powershell on current directory. To open an exe at current directory, the format is like `.\autosub`.
Or you can just directly open it and input the args manually though I don't recommend doing this due to its less efficiency.
- The one without pyinstaller suffix is compiled by Nuitka. It's faster than the pyinstaller due to its compiling feature different from pyinstaller which just bundles the application.
- ffmpeg and ffmpeg-normalize are also in the package. The original ffmpeg-normalize doesn't have a standalone version. The standalone version of ffmpeg-normalize is built separately. Codes are [here](https://github.com/BingLingGroup/ffmpeg-normalize).
- If there's anything wrong on the both releases, or the package size and any other things are annoying you, you can just use the traditional pip installation method below.
Or install Python environment(if you still don't have one) from choco and then install the package.
Recommend using [chocolatey](https://chocolatey.org) on windows to install the environment and dependencies.
Choco installation command is for cmd.(not Powershell)
```batch
@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
```
If you don't have [Build Tools for Visual Studio 2019](https://visualstudio.microsoft.com/downloads/), please install autosub without langcodes.
Install from `alpha` branch.(latest autosub alpha release)
```batch
choco install git python curl ffmpeg -y
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install git+https://github.com/BingLingGroup/autosub.git@alpha ffmpeg-normalize langcodes
```
Install from `dev` branch.(latest autosub dev version)
```batch
choco install git python curl ffmpeg -y
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install git+https://github.com/BingLingGroup/autosub.git@dev ffmpeg-normalize langcodes
```
Install from `origin` branch.(autosub-0.4.0a)
```batch
choco install git python2 curl ffmpeg -y
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install git+https://github.com/BingLingGroup/autosub.git@origin
```
PyPI version(autosub-0.3.12) is not recommended using on windows because it just can't run successfully. See the [changelog on the origin branch](CHANGELOG.md#040-alpha---2019-02-17) and you will know the details.
Recommend using `python` instead of `python2` for autosub-0.4.0.
### Workflow
#### Input
A video/audio/subtitles file.
If it is a video or audio file, use ffmpeg to convert the format into [the proper one](https://github.com/gillesdemey/google-speech-v2#data) for API. Any format supported by ffmpeg is OK to input, but the output or processed format for API is limited by API and autosub codes.
Supported formats below:
[Google-Speech-v2](https://github.com/gillesdemey/google-speech-v2)
- 24bit/44100Hz/mono FLAC(default)
- Other format like OGG_OPUS isn't supported by API. (I've tried modifying requests headers or json requests and it just don't work) Or format like PCM has less bits per sample but more storage usage than FLAC. Although the API support it but I think it's unnecessary to modify codes to support it.
[Google Cloud Speech-to-Text API](https://cloud.google.com/speech-to-text/docs/encoding) [v1p1beta1](https://cloud.google.com/speech-to-text/docs/reference/rest/v1p1beta1/RecognitionConfig#AudioEncoding)
- Supported
- 24bit/44100Hz/mono FLAC(default)
- Supported but not default args (more info on [Transcribe Audio To Subtitles](#transcribe-audio-to-subtitles))
- 8000Hz|12000Hz|16000Hz|24000Hz|48000Hz/mono OGG_OPUS
- MP3
- 16bit/mono PCM
[Xfyun Speech-to-Text WebSocket API](https://www.xfyun.cn/doc/asr/voicedictation/API.html#%E6%8E%A5%E5%8F%A3%E8%A6%81%E6%B1%82)/[Baidu ASR API/Baidu ASR Pro API](https://ai.baidu.com/ai-doc/SPEECH/Vk38lxily)
- Supported
- 16bit/16000Hz/mono PCM
Also, you can use the built-in audio pre-processing function though Google [doesn't recommend](https://cloud.google.com/speech-to-text/docs/best-practices) doing this. Honestly speaking, if your audio volume is not been standardized like too loud or too quiet, it's recommended to use some tools or just the built-in function to standardize it. The default [pre-processing commands](https://github.com/agermanidis/autosub/issues/40#issuecomment-509928060) depend on the ffmpeg-normalize and ffmpeg. The commands include three commands. The [first](https://trac.ffmpeg.org/wiki/AudioChannelManipulation) is for converting stereo to mono. The [second](https://superuser.com/questions/733061/reduce-background-noise-and-optimize-the-speech-from-an-audio-clip-using-ffmpeg) is for filtering out the sound not in the frequency of speech. The third is to normalize the audio to make sure it is not too loud or too quiet. If you are not satisfied with the default commands, you can also modified them yourself by input `-apc` option. Still, it currently only supports 24bit/44100Hz/mono FLAC format.
If it is a subtitles file and you give the proper arguments, only translate it by py-googletrans.
#### Split
Audio length limits:
[Google-Speech-v2](https://github.com/gillesdemey/google-speech-v2)
- No longer than [10 to 15 seconds](https://github.com/gillesdemey/google-speech-v2#caveats).
- In autosub it is set as the [60-seconds-limit](https://github.com/BingLingGroup/autosub/blob/dev/autosub/constants.py#L74).
[Google Cloud Speech-to-Text API](https://cloud.google.com/speech-to-text/docs/encoding)
- No longer than [1 minute](https://cloud.google.com/speech-to-text/docs/sync-recognize).
- In autosub it is currently set the same as the [60-seconds-limit](https://github.com/BingLingGroup/autosub/blob/dev/autosub/constants.py#L74).
- Currently only support sync-recognize means only short-term audio supported.
[Xfyun Speech-to-Text WebSocket API](https://www.xfyun.cn/doc/asr/voicedictation/API.html#%E6%8E%A5%E5%8F%A3%E8%A6%81%E6%B1%82)/[Baidu ASR API/Baidu ASR Pro API](https://ai.baidu.com/ai-doc/SPEECH/Vk38lxily)
- Same limit above.
Autosub uses Auditok to detect speech regions. And then use them to split as well as convert the video/audio into many audio fragments. Each fragment per region per API request. All these audio fragments are converted directly from input to avoid any extra quality loss.
Or uses external regions from the file that pysubs2 supports like `.ass` or `.srt`. This will allow you to manually adjust the regions to get better recognition result.
#### Speech-to-Text/Translation API request
Makes parallel requests to generate transcriptions for those regions. One audio fragment per request. Recognition speed mostly depends on your network upload speed.
- Manually post-processing for the subtitles lines may be needed, some of which are too long to be fitted in a single line at the bottom of the video frame.
After Speech-to-Text, translates them to a different language. Combining multiple lines of text to a chunk of text to request for result. Details at [issue #49](https://github.com/BingLingGroup/autosub/issues/49). And finally saves the result subtitles to the local storage.
#### Speech-to-Text/Translation language support
Below is only for Google API language codes description. About other API: [Xfyun speech config](#xfyun-speech-config), [baidu speech config](#baidu-speech-config).
The Speech-to-Text lang codes are different from the Translation lang codes due to the difference between these two APIs. And of course, they are in *Google* formats, not following the iso standards, making users more confused to use.
To solve this problem, autosub uses [langcodes](https://github.com/LuminosoInsight/langcodes) to detect input lang code and convert it to a best match according to the lang code lists. Default it won't be enabled. To enable it in different phases, use `-bm all` option.
To manually match or see the full list of the lang codes, run the utility with the argument `-lsc`/`--list-speech-codes` and `-ltc`/ `--list-translation-codes`. Or open [constants.py](autosub/constants.py) and check.
To get the language of the first line of the subtitles file, you can use `-dsl` to detect.
- Currently, autosub allows to send the lang codes not from the `--list-speech-codes`, which means in this case the program won't stop.
- Though you can input the speech lang code whatever you want, need to point out that if not using the codes on the list but somehow the API accept it, [Google-Speech-v2](https://github.com/gillesdemey/google-speech-v2) recognizes your audio in the ways that depend on your IP address which is uncontrollable by yourself. This is a known issue and I ask for a [pull request](https://github.com/agermanidis/autosub/pull/136) in the original repo.
- On the other hand, [py-googletrans](https://github.com/ssut/py-googletrans) is stricter. When it receive a lang code not on its list, it will throw an exception and stop translation.
- Apart from the user input, another notable change is I split the `-S` option into two parts, `-S` and `-SRC`. `-S` option is for speech recognition lang code. `-SRC` is for translation source language. When not offering the arg of `-SRC`, autosub will automatically match the `-S` arg by using [langcodes](https://github.com/LuminosoInsight/langcodes) and get a best-match lang code for translation source language though [py-googletrans](https://github.com/ssut/py-googletrans) can auto-detect source language. Of course you can manually specify one by input `-SRC` option. `-D` is for translation destination language, still the same as before.
#### Output
Currently support the following formats to output.
```Python
OUTPUT_FORMAT = {
'srt': 'SubRip',
'ass': 'Advanced SubStation Alpha',
'ssa': 'SubStation Alpha',
'sub': 'MicroDVD Subtitle',
'mpl2.txt': 'Similar to MicroDVD',
'tmp': 'TMP Player Subtitle Format',
'vtt': 'WebVTT',
'json': 'json(Only times and text)',
'ass.json': 'json(Complex ass content json)',
'txt': 'Plain Text(Text or times)'
}
```
Or other subtitles types/output modes, depend on what you need. More info in help message.
```Python
DEFAULT_MODE_SET = {
'regions',
'src',
'full-src',
'dst',
'bilingual',
'dst-lf-src',
'src-lf-dst'
}
```
### Usage
For the original autosub usage, see [简体中文使用指南](https://binglinggroup.github.io/archives/autosub安装使用指南(windows及ubuntu).html).
For the modified alpha branch version, see the typical usage below.
#### Typical usage
##### Pre-process Audio
Use default [Audio pre-processing](https://github.com/agermanidis/autosub/issues/40).
Pre-processing only.
```
autosub -i input_file -ap o
```
Pre-processing as a part.
```
autosub -i input_file -ap y ...(other options)
```
##### Detect Regions
Detect regions by using Auditok.
Getting regions only.
```
autosub -i input_file
```
Getting regions as a part.
```
autosub -i input_file -of regions ...(other options)
```
##### Split Audio
Get audio fragments according to the regions.
Only get audio fragments according to auto regions detection.
```
autosub -i input_file -ap s
```
Only get audio fragments according to external regions.
```
autosub -i input_file -ap s -er external_regions_subtitles
```
Getting audio fragments as a part.
```
autosub -i input_file -k ...(other options)
```
##### Transcribe Audio To Subtitles
Speech audio fragments to speech language subtitles.
###### Google Speech V2
Use default [Google-Speech-v2](https://github.com/gillesdemey/google-speech-v2) to transcribe speech language subtitles only.
```
autosub -i input_file -S lang_code
```
Use default [Google-Speech-v2](https://github.com/gillesdemey/google-speech-v2) to transcribe speech language subtitles as a part.
```
autosub -i input_file -S lang_code -of src ...(other options)
```
###### Google Cloud Speech-to-Text
Use Google Cloud Speech-to-Text API service account(GOOGLE_APPLICATION_CREDENTIALS has already been set in system environment variable) to transcribe.
```
autosub -i input_file -sapi gcsv1 -S lang_code ...(other options)
```
Use Google Cloud Speech-to-Text API service account(GOOGLE_APPLICATION_CREDENTIALS is set by `-sa`) to transcribe.(Currently not available in Nuitka build.)
```
autosub -i input_file -sapi gcsv1 -S lang_code -sa path_to_key_file ...(other options)
```
Use Google Cloud Speech-to-Text API key to transcribe.
```
autosub -i input_file -sapi gcsv1 -S lang_code -skey API_key ...(other options)
```
Use 48000Hz OGG_OPUS in Google Cloud Speech-to-Text API. The conversion commands will be automatically modified by these [codes](https://github.com/BingLingGroup/autosub/blob/alpha/autosub/__init__.py#L135-L140).
```
autosub -i input_file -sapi gcsv1 -asf .ogg -asr 48000 ...(other options)
```
Use MP3 in Google Cloud Speech-to-Text API.(Not recommended because OGG_OPUS is better than MP3)
```
autosub -i input_file -sapi gcsv1 -asf .mp3 ...(other options)
```
###### Google Speech config
Use customized [speech config file](https://googleapis.dev/python/speech/latest/gapic/v1/types.html#google.cloud.speech_v1.types.RecognitionConfig) to send request to Google Cloud Speech API. If using the config file, override these options: `-S`, `-asr`, `-asf`.
`language_code` will be replaced by the best matching one if using option `-bm src` or `-bm all`. `encoding` string will be replaced by the enum in `google.cloud.speech_v1p1beta1.enums.RecognitionConfig.AudioEncoding` if using service account credentials. Default `encoding` is `FLAC`. Default `sample_rate_hertz` is `44100`.
Example speech config file:
```json
{
"language_code": "zh",
"enable_word_time_offsets": true
}
```
If not provide option `-asr` and `-asf`, equal to:
```json
{
"language_code": "zh",
"sample_rate_hertz": 44100,
"encoding": "FLAC",
"enable_word_time_offsets": true
}
```
otherwise:
```json
{
"language_code": "zh",
"sample_rate_hertz": "from --api-sample-rate",
"encoding": "from --api-suffix",
"enable_word_time_offsets": true
}
```
command:
```
autosub -i input_file -sconf config_json_file -bm all -sapi gcsv1 -skey API_key ...(other options)
```
###### Output API full response
Currently autosub can't handle many [advanced fields](https://cloud.google.com/speech-to-text/docs/reference/rpc/google.cloud.speech.v1p1beta1#google.cloud.speech.v1p1beta1.SpeechRecognitionResult) contained in the speech recognition result received from API, especially from Google Cloud Speech-to-Text API. With complex [speech config](#speech-config) input and option `-of full-src`, recognition results will be output into json file so you can customize them and handle them outside autosub.
Example json output:
```json
[
{
"start": 0.52,
"end": 1.31,
"content": {
"results": [
{
"alternatives": [
{
"confidence": 0.98267895,
"transcript": "how old is the Brooklyn Bridge"
}
]
}
]
}
}
]
```
"start" and "end" mean the start seconds and the end seconds in the whole audio file. "content" is the result received from API.
command:
```
autosub -i input_file -sconf config_json_file -bm all -sapi gcsv1 -skey API_key -of full-src ...(other options)
```
##### Xfyun speech config
For Xfyun Speech-to-Text WebSocket API usage, user must input its speech config.
Example speech config file:
```json
{
"app_id": "",
"api_secret": "",
"api_key": "",
"api_address": "",
"business": {
"language": "zh_cn",
"domain": "iat",
"accent": "mandarin"
}
}
```
`"business"`/`"api_address"` field is the same as the [xfyun document](https://www.xfyun.cn/doc/asr/voicedictation/API.html#%E4%B8%9A%E5%8A%A1%E5%8F%82%E6%95%B0) mentioned.
When the file doesn't include the `"business"` field, autosub will use the above default content instead.
If you add `"delete_chars": "，。"` in the configuration file (In this example, full-width comma and period are the punctuations to be deleted), autosub will automatically replace the specific punctuation with a space when receiving the transcript, and strip the space at the end of each sentence.
command:
```
autosub -sapi xfyun -i input_file -sconf xfyun_speech_config ...(other options)
```
##### Baidu speech config
For Baidu ASR API usage, user must input its speech config.
Example speech config file:
```json
{
"AppID": "",
"API key": "",
"Secret Key": "",
"config": {
"format": "pcm",
"rate": 16000,
"channel": 1,
"cuid": "python",
"dev_pid": 1537
}
}
```
`"config"` field is the same as the [Baidu ASR document](https://ai.baidu.com/ai-doc/SPEECH/ek38lxj1u) mentioned.
If you want to use the Pro ASR API, change the value of `"cuid"` into `80001`.
When the file doesn't include the `"config"` field, autosub will use the above default content instead.
Same `"delete_chars"` function above.
Practical speaking, since Baidu ASR/ASR Pro API doesn't allow concurrency by default, concurrency will be limited to 1. If you need to lift the limit, please add `"disable_qps_limit": true,` to the config file. If so, the concurrency will be set by the option `-sc`.
command:
```
autosub -sapi baidu -i input_file -sconf baidu_speech_config ...(other options)
```
##### Translate Subtitles
Translate subtitles to another language.
If not input option `-SRC`, the translation source language will be auto-detected by py-googletrans.
Translate subtitles from an audio/video file.
```
autosub -i input_file -S lang_code (-SRC lang_code) -D lang_code
```
Translate subtitles from a subtitles file.
```
autosub -i input_file (-SRC lang_code) -D lang_code
```
Translate subtitles by "translate.google.cn" which can be directly accessed from somewhere.
```
autosub -i input_file -surl "translate.google.cn" ...(other options)
```
Or use other available URLs like these [ssut/py-googletrans#165](https://github.com/ssut/py-googletrans/issues/165).
#### Options
Full list of the help message.
```
$ autosub -h
usage:
autosub [-i path] [options]
Auto-generate subtitles for video/audio/subtitles file.
Input Options:
Options to control input.
-i path, --input path
The path to the video/audio/subtitles file that needs
to generate subtitles. When it is a subtitles file,
the program will only translate it. (arg_num = 1)
-er path, --ext-regions path
Path to the subtitles file which provides external
speech regions, which is one of the formats that
pysubs2 supports and overrides the default method to
find speech regions. (arg_num = 1)
-sty [path], --styles [path]
Valid when your output format is "ass"/"ssa". Path to
the subtitles file which provides "ass"/"ssa" styles
for your output. If the arg_num is 0, it will use the
styles from the : "-esr"/"--external-speech-regions".
More info on "-sn"/"--styles-name". (arg_num = 0 or 1)
-sn [style_name [style_name ...]], --styles-name [style_name [style_name ...]]
Valid when your output format is "ass"/"ssa" and
"-sty"/"--styles" is given. Adds "ass"/"ssa" styles to
your events. If not provided, events will use the
first one from the file. If the arg_num is 1, events
will use the specific style from the arg of "-sty"/"--
styles". If the arg_num is 2, src language events use
the first. Dst language events use the second.
(arg_num = 1 or 2)
Language Options:
Options to control language.
-S lang_code, --speech-language lang_code
Lang code/Lang tag for speech-to-text. Recommend using
the Google Cloud Speech reference lang codes. WRONG
INPUT WON'T STOP RUNNING. But use it at your own risk.
Ref: https://cloud.google.com/speech-to-
text/docs/languages(arg_num = 1) (default: None)
-SRC lang_code, --src-language lang_code
Lang code/Lang tag for translation source language. If
not given, use py-googletrans to auto-detect the src
language. (arg_num = 1) (default: auto)
-D lang_code, --dst-language lang_code
Lang code/Lang tag for translation destination
language. (arg_num = 1) (default: None)
-bm [mode [mode ...]], --best-match [mode [mode ...]]
Use langcodes to get a best matching lang code when
your input is wrong. Only functional for py-
googletrans and Google Speech API. If langcodes not
installed, use fuzzywuzzy instead. Available modes: s,
src, d, all. "s" for "-S"/"--speech-language". "src"
for "-SRC"/"--src-language". "d" for "-D"/"--dst-
language". (3 >= arg_num >= 1)
-mns integer, --min-score integer
An integer between 0 and 100 to control the good match
group of "-lsc"/"--list-speech-codes" or "-ltc"/"--
list-translation-codes" or the match result in "-bm"/"
--best-match". Result will be a group of "good match"
whose score is above this arg. (arg_num = 1)
Output Options:
Options to control output.
-o path, --output path
The output path for subtitles file. (default: the
"input" path combined with the proper name tails)
(arg_num = 1)
-F format, --format format
Destination subtitles format. If not provided, use the
extension in the "-o"/"--output" arg. If "-o"/"--
output" arg doesn't provide the extension name, use
"srt" instead. In this case, if "-i"/"--input" arg is
a subtitles file, use the same extension from the
subtitles file. (arg_num = 1) (default: srt)
-y, --yes Prevent pauses and allow files to be overwritten. Stop
the program when your args are wrong. (arg_num = 0)
-of [type [type ...]], --output-files [type [type ...]]
Output more files. Available types: regions, src,
full-src, dst, bilingual, dst-lf-src, src-lf-dst, all.
"regions", "src", "full-src" are available only if
input is not a subtitles file. full-src: Full result
received from Speech-to-Text API in json format with
start and end time. dst-lf-src: dst language and src
language in the same event. And dst is ahead of src.
src-lf-dst: src language and dst language in the same
event. And src is ahead of dst. (6 >= arg_num >= 1)
(default: ['dst'])
-fps float, --sub-fps float
Valid when your output format is "sub". If input, it
will override the fps check on the input file. Ref:
https://pysubs2.readthedocs.io/en/latest/api-
reference.html#supported-input-output-formats (arg_num
= 1)
Speech Options:
Options to control speech-to-text. If Speech Options not given, it will only generate the times.
-sapi API_code, --speech-api API_code
Choose which Speech-to-Text API to use. Currently
support: gsv2: Google Speech V2
(https://github.com/gillesdemey/google-speech-v2).
gcsv1: Google Cloud Speech-to-Text V1P1Beta1
(https://cloud.google.com/speech-to-text/docs). xfyun:
Xun Fei Yun Speech-to-Text WebSocket API (https://www.
xfyun.cn/doc/asr/voicedictation/API.html). baidu:
Baidu Automatic Speech Recognition API
(https://ai.baidu.com/ai-doc/SPEECH/Vk38lxily)
(arg_num = 1) (default: gsv2)
-skey key, --speech-key key
The API key for Google Speech-to-Text API. (arg_num =
1) Currently support: gsv2: The API key for gsv2.
(default: Free API key) gcsv1: The API key for gcsv1.
(If used, override the credentials given by"-sa"/"--
service-account")
-sconf [path], --speech-config [path]
Use Speech-to-Text recognition config file to send
request. Override these options below: "-S", "-asr",
"-asf". Currently support: gcsv1: Google Cloud Speech-
to-Text V1P1Beta1 API key config reference:
https://cloud.google.com/speech-to-
text/docs/reference/rest/v1p1beta1/RecognitionConfig
Service account config reference: https://googleapis.d
ev/python/speech/latest/gapic/v1/types.html#google.clo
ud.speech_v1.types.RecognitionConfig xfyun: Xun Fei
Yun Speech-to-Text WebSocket API
(https://console.xfyun.cn/services/iat). baidu: Baidu
Automatic Speech Recognition API
(https://ai.baidu.com/ai-doc/SPEECH/ek38lxj1u). If
arg_num is 0, use const path. (arg_num = 0 or 1)
(const: config.json)
-mnc float, --min-confidence float
Google Speech-to-Text API response for text
confidence. A float value between 0 and 1. Confidence
bigger means the result is better. Input this argument
will drop any result below it. Ref:
https://github.com/BingLingGroup/google-
speech-v2#response (arg_num = 1) (default: 0.0)
-der, --drop-empty-regions
Drop any regions without speech recognition result.
(arg_num = 0)
-sc integer, --speech-concurrency integer
Number of concurrent Speech-to-Text requests to make.
(arg_num = 1) (default: 4)
py-googletrans Options:
Options to control translation. Default method to translate. Could be blocked at any time.
-slp second, --sleep-seconds second
(Experimental)Seconds to sleep between two translation
requests. (arg_num = 1) (default: 1)
-surl [URL [URL ...]], --service-urls [URL [URL ...]]
(Experimental)Customize request urls. Ref: https://py-
googletrans.readthedocs.io/en/latest/ (arg_num >= 1)
-ua User-Agent headers, --user-agent User-Agent headers
(Experimental)Customize User-Agent headers. Same docs
above. (arg_num = 1)
-doc, --drop-override-codes
Drop any .ass override codes in the text before
translation. Only affect the translation result.
(arg_num = 0)
-gt-dc [chars], --gt-delete-chars [chars]
Replace the specific chars with a space after
translation, and strip the space at the end of each
sentence. Only affect the translation result. (arg_num
= 0 or 1) (const: ，。！)
Subtitles Conversion Options:
Options to control subtitles conversions.(Experimental)
-mjs integer, --max-join-size integer
(Experimental)Max length to join two events. (arg_num
= 1) (default: 100)
-mdt second, --max-delta-time second
(Experimental)Max delta time to join two events.
(arg_num = 1) (default: 0.2)
-dms string, --delimiters string
(Experimental)Delimiters not to join two events.
(arg_num = 1) (default: !()*,.:;?[]^_`~)
-sw1 words_delimited_by_space, --stop-words-1 words_delimited_by_space
(Experimental)First set of Stop words to split two
events. (arg_num = 1)
-sw2 words_delimited_by_space, --stop-words-2 words_delimited_by_space
(Experimental)Second set of Stop words to split two
events. (arg_num = 1)
-ds, --dont-split (Experimental)Don't Split just merge. (arg_num = 0)
Network Options:
Options to control network.
-hsa, --http-speech-api
Change the Google Speech V2 API URL into the http one.
(arg_num = 0)
-hsp [URL], --https-proxy [URL]
Add https proxy by setting environment variables. If
arg_num is 0, use const proxy url. (arg_num = 0 or 1)
(const: https://127.0.0.1:1080)
-hp [URL], --http-proxy [URL]
Add http proxy by setting environment variables. If
arg_num is 0, use const proxy url. (arg_num = 0 or 1)
(const: http://127.0.0.1:1080)
-pu username, --proxy-username username
Set proxy username. (arg_num = 1)
-pp password, --proxy-password password
Set proxy password. (arg_num = 1)
Other Options:
Other options to control.
-h, --help Show autosub help message and exit. (arg_num = 0)
-V, --version Show autosub version and exit. (arg_num = 0)
-sa path, --service-account path
Set service account key environment variable. It
should be the file path of the JSON file that contains
your service account credentials. Can be overridden by
the API key. Ref:
https://cloud.google.com/docs/authentication/getting-
started Currently support: gcsv1
(GOOGLE_APPLICATION_CREDENTIALS) (arg_num = 1)
Audio Processing Options:
Options to control audio processing.
-ap [mode [mode ...]], --audio-process [mode [mode ...]]
Option to control audio process. If not given the
option, do normal conversion work. "y": pre-process
the input first then start normal workflow. If
succeed, no more conversion before the speech-to-text
procedure. "o": only pre-process the input audio.
("-k"/"--keep" is true) "s": only split the input
audio. ("-k"/"--keep" is true) Default command to pre-
process the audio: C:\Program
Files\ImageMagick-7.0.10-Q16\ffmpeg.exe -hide_banner
-i "{in_}" -vn -af "asplit[a],aphasemeter=video=0,amet
adata=select:key=lavfi.aphasemeter.phase:value=-0.005:
function=less,pan=1c|c0=c0,aresample=async=1:first_pts
=0,[a]amix" -ac 1 -f flac -loglevel error "{out_}" |
C:\Program Files\ImageMagick-7.0.10-Q16\ffmpeg.exe
-hide_banner -i "{in_}" -af
"lowpass=3000,highpass=200" -loglevel error "{out_}" |
C:\Python37\Scripts\ffmpeg-normalize.exe -v "{in_}"
-ar 44100 -ofmt flac -c:a flac -pr -p -o "{out_}"
(Ref: https://github.com/stevenj/autosub/blob/master/s
cripts/subgen.sh https://ffmpeg.org/ffmpeg-
filters.html) (2 >= arg_num >= 1)
-k, --keep Keep audio processing files to the output path.
(arg_num = 0)
-apc [command [command ...]], --audio-process-cmd [command [command ...]]
This arg will override the default audio pre-process
command. Every line of the commands need to be in
quotes. Input file name is {in_}. Output file name is
{out_}. (arg_num >= 1)
-ac integer, --audio-concurrency integer
Number of concurrent ffmpeg audio split process to
make. (arg_num = 1) (default: 4)
-acc command, --audio-conversion-cmd command
(Experimental)This arg will override the default audio
conversion command. "[", "]" are optional arguments
meaning you can remove them. "{", "}" are required
arguments meaning you can't remove them. (arg_num = 1)
(default: C:\Program
Files\ImageMagick-7.0.10-Q16\ffmpeg.exe -hide_banner
-y -i "{in_}" -vn -ac {channel} -ar {sample_rate}
-loglevel error "{out_}")
-asc command, --audio-split-cmd command
(Experimental)This arg will override the default audio
split command. Same attention above. (arg_num = 1)
(default: C:\Program
Files\ImageMagick-7.0.10-Q16\ffmpeg.exe -y -ss {start}
-i "{in_}" -t {dura} -vn -ac [channel] -ar
[sample_rate] -loglevel error "{out_}")
-asf file_suffix, --api-suffix file_suffix
(Experimental)This arg will override the default API
audio suffix. (arg_num = 1) (default: .flac)
-asr sample_rate, --api-sample-rate sample_rate
(Experimental)This arg will override the default API
audio sample rate(Hz). (arg_num = 1) (default: 44100)
-aac channel_num, --api-audio-channel channel_num
(Experimental)This arg will override the default API
audio channel. (arg_num = 1) (default: 1)
Auditok Options:
Options to control Auditok when not using external speech regions control.
-et energy, --energy-threshold energy
The energy level which determines the region to be
detected. Ref: https://auditok.readthedocs.io/en/lates
t/apitutorial.html#examples-using-real-audio-data
(arg_num = 1) (default: 45)
-mnrs second, --min-region-size second
Minimum region size. Same docs above. (arg_num = 1)
(default: 0.5)
-mxrs second, --max-region-size second
Maximum region size. Same docs above. (arg_num = 1)
(default: 10.0)
-mxcs second, --max-continuous-silence second
Maximum length of a tolerated silence within a valid
audio activity. Same docs above. (arg_num = 1)
(default: 0.2)
-nsml, --not-strict-min-length
If not input this option, it will keep all regions
strictly follow the minimum region limit. Ref: https:/
/auditok.readthedocs.io/en/latest/core.html#class-
summary (arg_num = 0)
-dts, --drop-trailing-silence
Ref: https://auditok.readthedocs.io/en/latest/core.htm
l#class-summary (arg_num = 0)
List Options:
List all available arguments.
-lf, --list-formats List all available subtitles formats. If your format
is not supported, you can use ffmpeg or SubtitleEdit
to convert the formats. You need to offer fps option
when input is an audio file and output is "sub"
format. (arg_num = 0)
-lsc [lang_code], --list-speech-codes [lang_code]
List all recommended "-S"/"--speech-language" Google
Speech-to-Text language codes. If no arg is given,
list all. Or else will list a group of "good match" of
the arg. Default "good match" standard is whose match
score above 90 (score between 0 and 100). Ref:
https://tools.ietf.org/html/bcp47 https://github.com/L
uminosoInsight/langcodes/blob/master/langcodes/__init_
_.py lang code example: language-script-region-
variant-extension-privateuse (arg_num = 0 or 1)
-ltc [lang_code], --list-translation-codes [lang_code]
List all available "-SRC"/"--src-language" py-
googletrans translation language codes. Or else will
list a group of "good match" of the arg. Same docs
above. (arg_num = 0 or 1)
-dsl path, --detect-sub-language path
Use py-googletrans to detect a sub file's first line
language. And list a group of matched language in
recommended "-S"/"--speech-language" Google Speech-to-
Text language codes. Ref:
https://cloud.google.com/speech-to-text/docs/languages
(arg_num = 1) (default: None)
Make sure the argument with space is in quotes.
The default value is used
when the option is not given at the command line.
"(arg_num)" means if the option is given,
the number of the arguments is required.
Arguments *ARE* the things given behind the options.
Author: Bing Ling
Email: [email protected]
Bug report: https://github.com/BingLingGroup/autosub
```
#### Internationalization
Autosub supports multi-language command line user interface by [GNU gettext](https://www.gnu.org/software/gettext/). Now supports `zh_CN` and default `en_US`. More info about this [lang codes format](https://www.gnu.org/software/gettext/manual/gettext.html#Locale-Names). The program will automatically detect the os locale and use the one supported. For windows 10, it seems adjusting the `Region`-`Regional format` is Ok.
Of course, autosub offers a method to override the os locale. Just create a txt file without extension named `locale`, containing the lang codes at the beginning of the file, at the command line current working directory. When autosub starts, it will detect this file and read the lang code inside it and apply it if supported.
If you want to translate this program into other languages, first install the gettext utilities. Then you can run `python scripts/update_po_files.py lang_code` to create the locale files which you want to translate into. And then use [POEditor](https://poeditor.com/) to edit po files. [update_po_files.py](scripts/update_po_files.py) can also automatically merge the position info into the old po files and compile the po files into mo files which the program read them. So it's useful when the codes changed, you can merge the positional changes into the translations automatically.
### FAQ
#### Other APIs supports
[issue #11](https://github.com/BingLingGroup/autosub/issues/11)
I won't add any new features unless I'm less busy in the future. However, pull requests are welcomed.
#### Batch processing
[issue #13](https://github.com/BingLingGroup/autosub/issues/13)
Same as above. Currently won't add it. You can use batch/powershell/bash to implement it.
Example for batch:(working at current directory)
```batch
@echo off
set "in_format=*.mp4 *.m4a"
@echo on
for /f "delims=^" %%i in ('dir /b %in_format%') do (
autosub -i "%%i" ...(other options)
)
@echo off
```
If you want do a recursive walk through the directories, replace `'dir /b %in_format%'` with `'dir /b/s/a:-d %in_format%'` is Ok. [Reference](https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/dir).
#### proxy support
[issue #17](https://github.com/BingLingGroup/autosub/issues/17)
Currently I only implement the proxy settings in the same way as setting environment variables in the command line. So it is necessary for you to open a http/https proxy server locally like [shadowsocks-windows](https://github.com/shadowsocks/shadowsocks-windows/releases) or [shadowsocks](https://github.com/shadowsocks/shadowsocks/tree/master).
If you often encounter empty result or connection error during speech-to-text or subtitles translation, perhaps you need to get a better proxy for a better connection with Google or just rent a Linux server which can reach Google's network.
#### macOS locale issue
[issue 83 (comment)](https://github.com/BingLingGroup/autosub/issues/83#issuecomment-586624157)
```Python
Traceback (most recent call last):
File "/usr/local/bin/autosub", line 5, in
from autosub import main
File "/usr/local/lib/python3.7/site-packages/autosub/__init__.py", line 15, in
from autosub import ffmpeg_utils
File "/usr/local/lib/python3.7/site-packages/autosub/ffmpeg_utils.py", line 25, in
fallback=True)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/gettext.py", line 518, in translation
mofiles = find(domain, localedir, languages, all=True)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/gettext.py", line 490, in find
for nelang in _expand_lang(lang):
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/gettext.py", line 212, in _expand_lang
loc = locale.normalize(loc)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/locale.py", line 401, in normalize
code = localename.lower()
AttributeError: 'NoneType' object has no attribute 'lower'
```
It seems environment variable `LANG` and `LC_ALL` are not set on some macOS versions. Please manually set it before running the program. [ewdurbin/evacuate_2stp#1 (comment)](https://github.com/ewdurbin/evacuate_2stp/issues/1#issuecomment-413736644)
[How to set environment variables on macOS](https://medium.com/@himanshuagarwal1395/setting-up-environment-variables-in-macos-sierra-f5978369b255).
#### Accuracy
Apart from the volume issue mentioned in the [input](#input) section above, you need to make sure the audio doesn't contain some kind of vocals otherwise you need to adjust the Auditok options or the speech recognition config.
### Bugs report
Bugs and suggestions can be reported at [issues](https://github.com/BingLingGroup/autosub/issues).
### Build
I only write the scripts for building standalone executable files on windows, [Nuitka script](scripts/nuitka_build.bat) and [pyinstaller script](scripts/pyinstaller_build.bat).
The version 0.5.4a doesn't support Nuitka build since 0.5.4a import google.cloud package and it contains `pkg_resources.get_distribution` which is not supported by Nuitka due to this [Nuitka issue #146](https://github.com/Nuitka/Nuitka/issues/146).
The version 0.5.5a catches the exception `pkg_resources.DistributionNotFound` to remove the Google Cloud service account support when it is built by Nuitka.
Nuitka build is pretty tricky. These environments I tried and worked.
1. Anaconda recommended by [Nuitka readme](https://github.com/Nuitka/Nuitka#id6).
- Python version 3.5
- mingw-w64 package [m2w64-gcc](https://anaconda.org/msys2/m2w64-gcc) (No need to set the environment variables separately if you run it on Anaconda Prompt)
2. Use other C Compiler rather than the [m2w64-gcc](https://anaconda.org/msys2/m2w64-gcc) by setting the value of environment variable `CC` which is the path to the C Compiler, including the executable name. For example, you install [MingW-W64-builds](http://mingw-w64.org/doku.php/download/mingw-builds) somewhere on your storage and you want Nuitka to use it. In this case, Python 3.5 is still recommended.
3. Nuitka 0.6.6 is the latest stable version I tried. Other later version like 0.6.7 doesn't support windows icon input.
And for those whose os language is not `en_US`, please set it to `en_US` and then start to build. Otherwise you may encounter this [known issue](https://github.com/Nuitka/Nuitka/issues/193).
About Pyinstaller build, you need to manually hook the gcloud module. [Source](https://stackoverflow.com/questions/40076795/pyinstaller-file-fails-to-execute-script-distributionnotfound).
> . So you need to create a hook file for that names
>
> Python_Path\Lib\site-packages\PyInstaller\hooks\hook-gcloud.py
>
> File contents:
```Python
from PyInstaller.utils.hooks import copy_metadata
datas = copy_metadata('gcloud')
```
[create_release.py](scripts/create_release.py) is used to make the two release packages. You need to create a `binaries` folder containing ffmpeg and ffmpeg-normalize executable files if you want to create a "fully" standalone release like the one I release. `ffmpeg.exe` and `ffprobe.exe` are from [Zeranoe ffmpeg windows build](https://ffmpeg.zeranoe.com/builds/). `ffmpeg-normalize.exe` built in the same way as I mentioned [above](#install-on-windows).
It should have the directory structure like this below.
```
binaries\ffmpeg.exe
binaries\ffprobe.exe
binaries\ffmpeg-normalize-Nuitka\ffmpeg-normalize.exe
binaries\ffmpeg-normalize-pyinstaller\ffmpeg-normalize.exe
```