https://github.com/linagora-labs/asr_benchmark
Toolkit to benchmark various speech recognition APIs (NeMo, Whisper...) and visualize the results
https://github.com/linagora-labs/asr_benchmark
asr benchmark nemo speech-recognition whisper
Last synced: 6 months ago
JSON representation
Toolkit to benchmark various speech recognition APIs (NeMo, Whisper...) and visualize the results
- Host: GitHub
- URL: https://github.com/linagora-labs/asr_benchmark
- Owner: linagora-labs
- License: agpl-3.0
- Created: 2024-04-22T15:29:53.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-04-02T09:14:10.000Z (6 months ago)
- Last Synced: 2025-04-02T10:25:13.505Z (6 months ago)
- Topics: asr, benchmark, nemo, speech-recognition, whisper
- Language: Jupyter Notebook
- Homepage:
- Size: 657 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# ASR Benchmark
Toolkit to benchmark various speech recognition APIs (NeMo, Whisper...) and visualize the results. Supported models are mostly french. It can compute WER, RTF (or latencies when streaming) and measure hardware usage.
## How to bench
Just run:
```
python benchmarker.py CONFIG_FILE
```
### Data
The input data file (manifest) is a jsonl file (one json per line). Each line must have these fields:
- "audio_filepath", the path to the audio file
- "text", the text associated with the segment
They can also have:
- "offset", the start of the segment in the audio (if not specified, it is equal to 0)
- "duration", the duration of the segment (if not specified, the whole audio is used)
- "name" or "dataset", the name of the dataset
### Examples
Examples are provided in the `examples` folder. There is a audio file to test with benchmark config file, and a notebook for generating plots.
## Requirements
You need to install the package so that benchmarker.py can find the source code:
```
pip install -e .
```
You also need to install [ssak](https://github.com/linagora-labs/ssak) repo. You can run:
```
pip install git+https://github.com/linagora-labs/ssak
```
Then depending on what you want to bench, you will need to install other packages like faster-whisper, nemo(nemo_toolkit['asr']), whisper and transformers.
## Tools
Some tools are avaialble in the `tools` folder:
- add_silence.py: a script for adding white noise to audio files
- subsample_data.py: for selecting a subset of specified datasets
Don't hesitate to submit your tools (for converting datasets to the jsonl format for example). I used scripts from ssak to do it but datasets were in kaldi format.
## Backends (interfaces)
The current available backends:
- HTTP-API ("http-api")
- LinTo-STT ("linto-stt"): for using whisper, kaldi or nemo models. Can be streaming (can compute latencies) or offline
- Whisper ("openai")
- Faster Whisper ("faster-whisper")
- Transformers ("transformers"): work for Whisper
- Transformers Intel ("intel-transformers"): for using intel extension
- Transformers Facebook ("transformers-facebook"): for MMS model
- Transformers Bofenghuang ("transformers-bofenghuang"): for the french finetuned wav2vec
- NeMo ("nemo")
If the available interfaces don't allow to bench a model you want, you can easily add it by folliwing these steps:
- You create new class that inherits from `asr_benchmark.benchmark.interfaces.Model`
- You implement the various functions (load, transcribe, ...)
- You add your backend in `asr_benchmark.benchmark.backend_to_model`