https://github.com/egorsmkv/optimized-whisper-intel

Run quantized Whisper models only on CPU with Intel hardware
https://github.com/egorsmkv/optimized-whisper-intel

intel onnx onnxruntime quantized-neural-networks whisper

Last synced: 5 days ago
JSON representation

Run quantized Whisper models only on CPU with Intel hardware

• Host: GitHub
• URL: https://github.com/egorsmkv/optimized-whisper-intel
• Owner: egorsmkv
• Created: 2024-08-24T09:18:03.000Z (4 months ago)
• Default Branch: master
• Last Pushed: 2024-09-03T12:02:27.000Z (4 months ago)
• Last Synced: 2024-10-31T17:44:56.160Z (about 2 months ago)
• Topics: intel, onnx, onnxruntime, quantized-neural-networks, whisper
• Language: Python
• Homepage:
• Size: 409 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Optimized Whisper for Intel

🚨🚨🚨 ATTENTION 🚨🚨🚨:

> Your CPU must have AVX512 support to run this model, run `lscpu` to check it.

## Install requirements

```

curl -LsSf https://astral.sh/uv/install.sh | sh

apt update && apt install git-lfs ffmpeg

```

## Install

```

git clone https://github.com/egorsmkv/whisper-intel-optimized

cd whisper-intel-optimized

uv venv --python 3.12 && source .venv/bin/activate

uv pip install -r requirements.txt

uv pip install -r requirements-dev.txt

```

## Run

### Download an audio file

```

wget -O ukrainian_speech.wav https://github.com/egorsmkv/wav2vec2-uk-demo/raw/master/short_1_16k.wav

```

### Download quantized models

```

huggingface-cli download Intel/whisper-large-v2-onnx-int4-inc --local-dir-use-symlinks False --local-dir ./whisper-large-v2-onnx-int4-inc

huggingface-cli download Intel/whisper-medium-onnx-int4-inc --local-dir-use-symlinks False --local-dir ./whisper-medium-onnx-int4-inc

huggingface-cli download Intel/whisper-small-onnx-int4-inc --local-dir-use-symlinks False --local-dir ./whisper-small-onnx-int4-inc

```

### Run the model

```

python run.py "openai/whisper-large-v2" "whisper-large-v2-onnx-int4-inc" "ukrainian_speech.wav"

python run.py "openai/whisper-medium" "whisper-medium-onnx-int4-inc" "ukrainian_speech.wav"

python run.py "openai/whisper-small" "whisper-small-onnx-int4-inc" "ukrainian_speech.wav"

python run.py "openai/whisper-small" "whisper-small-onnx-int4-inc" "sample_english.wav"

```

### Outputs

**whisper-large-v2-onnx-int4-inc**:

```

Taiwan and the United States are important strategic partners. But there is a difference. The United States has a special law that stipulates that if China attacks Taiwan, the American military must protect it.

```

**whisper-medium-onnx-int4-inc**:

```

Taiwan is a strategic partner, but there is a difference. The US has a special law that provides that if China attacks Taiwan, the US military must protect it.

```

**whisper-small-onnx-int4-inc**:

```

That we've had a

```

> **Note**: It didn't recognize the Ukrainian speech at all, only English speech at the end of the audio.

## Development

```

ruff check

ruff format

```