https://github.com/fly-apps/cog-whisper

Run OpenAI Whisper as a Cog model on Fly GPUs
https://github.com/fly-apps/cog-whisper

ai cog gpu whisper

Last synced: about 2 months ago
JSON representation

Run OpenAI Whisper as a Cog model on Fly GPUs

• Host: GitHub
• URL: https://github.com/fly-apps/cog-whisper
• Owner: fly-apps
• License: apache-2.0
• Created: 2024-02-29T15:37:37.000Z (7 months ago)
• Default Branch: main
• Last Pushed: 2024-07-17T08:23:16.000Z (2 months ago)
• Last Synced: 2024-07-17T10:30:11.637Z (2 months ago)
• Topics: ai, cog, gpu, whisper
• Language: Python
• Homepage:
• Size: 5.2 MB
• Stars: 2
• Watchers: 8
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.txt

Awesome Lists containing this project

README

        


    
Whisper on Fly GPUs

    
Run OpenAI Whisper as a Replicate Cog on Fly.io!



![cog](https://github.com/fly-apps/cog-whisper/assets/3727384/4d757f88-29c1-4966-ace9-01d0b1a630ac)

This app exposes the Whisper model via a simple HTTP server, thanks to Replicate Cog. Cog is an open-source tool that lets you package machine learning models in a standard, production-ready container. When you're up and running, you can trascribe audio using the /predictions endpoint.

## Launch

Create a deploy the app in one single command:

```console

fly launch --from https://github.com/fly-apps/cog-whisper --no-public-ips

```

Assign a [Flycast](https://fly.io/docs/networking/private-networking/#assign-a-flycast-address) IP to the app:

```console

fly ips allocate-v6 --private

```

That's it! You can now access the app at `http://.flycast/predictions`

> [!IMPORTANT]  

> By default, the app runs on Fly GPUs — Nvidia L40s to be exact. This can be customized in the fly.toml `vm` settings. It will run on a standard Fly Machine — but performance will be reduced.

## Usage

```console

curl -X PUT \

     -H "Content-Type: application/json" \

     -d '{

           "input": {

             "audio": "https://fly.storage.tigris.dev/cogs/bun_on_fly.mp3"

           }

         }' \

     http://cog-whisper.flycast/predictions/test | jq

```

## Local Setup

1. Clone the `cog-whisper` repository from GitHub:

    ```console

    git clone [email protected]:fly-apps/cog-whisper.git

    ```

2. Navigate into the cloned directory:

    ```console

    cd cog-whisper

    ```

    

3. Run locally. First, run `get_weights.sh` from the project root to download pre-trained weights, then build a container and run predictions:

    ```console

    ./scripts/get_weights.sh:

    ```

    ```console

    cog predict -i audio=""

    ```

4. Build the Docker image using `cog`:

    ```console

    cog build -t whisper

    ```

## Having trouble?

Create an issue or ask a question here: https://community.fly.io/