Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/fly-apps/cog-whisper

Run OpenAI Whisper as a Cog model on Fly GPUs
https://github.com/fly-apps/cog-whisper

ai cog gpu whisper

Last synced: about 1 month ago
JSON representation

Run OpenAI Whisper as a Cog model on Fly GPUs

Awesome Lists containing this project

README

        


Whisper on Fly GPUs


Run OpenAI Whisper as a Replicate Cog on Fly.io!


![cog](https://github.com/fly-apps/cog-whisper/assets/3727384/4d757f88-29c1-4966-ace9-01d0b1a630ac)

This app exposes the Whisper model via a simple HTTP server, thanks to Replicate Cog. Cog is an open-source tool that lets you package machine learning models in a standard, production-ready container. When you're up and running, you can trascribe audio using the /predictions endpoint.

## Launch

Create a deploy the app in one single command:

```console
fly launch --from https://github.com/fly-apps/cog-whisper --no-public-ips
```

Assign a [Flycast](https://fly.io/docs/networking/private-networking/#assign-a-flycast-address) IP to the app:

```console
fly ips allocate-v6 --private
```

That's it! You can now access the app at `http://.flycast/predictions`

> [!IMPORTANT]
> By default, the app runs on Fly GPUs — Nvidia L40s to be exact. This can be customized in the fly.toml `vm` settings. It will run on a standard Fly Machine — but performance will be reduced.

## Usage

```console
curl -X PUT \
-H "Content-Type: application/json" \
-d '{
"input": {
"audio": "https://fly.storage.tigris.dev/cogs/bun_on_fly.mp3"
}
}' \
http://cog-whisper.flycast/predictions/test | jq

```

## Local Setup

1. Clone the `cog-whisper` repository from GitHub:

```console
git clone [email protected]:fly-apps/cog-whisper.git
```

2. Navigate into the cloned directory:

```console
cd cog-whisper
```

3. Run locally. First, run `get_weights.sh` from the project root to download pre-trained weights, then build a container and run predictions:

```console
./scripts/get_weights.sh:
```

```console
cog predict -i audio=""
```

4. Build the Docker image using `cog`:

```console
cog build -t whisper
```

## Having trouble?

Create an issue or ask a question here: https://community.fly.io/