https://github.com/arihanv/Shush

Shush is an app that deploys a WhisperV3 model with Flash Attention v2 on Modal and makes requests to it via a NextJS app
https://github.com/arihanv/Shush

flash-attention-2 huggingface-transformers machine-learning modal shadcn-ui transcription whisper

Last synced: 3 months ago
JSON representation

Shush is an app that deploys a WhisperV3 model with Flash Attention v2 on Modal and makes requests to it via a NextJS app

• Host: GitHub
• URL: https://github.com/arihanv/Shush
• Owner: arihanv
• License: mit
• Created: 2023-11-18T21:34:55.000Z (8 months ago)
• Default Branch: main
• Last Pushed: 2023-12-29T06:06:27.000Z (7 months ago)
• Last Synced: 2023-12-29T07:23:35.250Z (7 months ago)
• Topics: flash-attention-2, huggingface-transformers, machine-learning, modal, shadcn-ui, transcription, whisper
• Language: TypeScript
• Homepage: https://shush.arihanv.com
• Size: 429 KB
• Stars: 74
• Watchers: 2
• Forks: 11
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Lists

• my-awesome-stars - arihanv/Shush - Shush is an app that deploys a WhisperV3 model with Flash Attention v2 on Modal and makes requests to it via a NextJS app (TypeScript)

README

        
# Shush

https://github.com/arihanv/Shush/assets/63890951/6e675260-c29a-4fd4-8ba5-0b70549f0bcd

Shush is an app that deploys a WhisperV3 model with Flash Attention v2 on Modal and makes requests to it via a NextJS app. The essential **goal** of this app is to provide a full-stack demo to those interested in running high-performance models and reliable APIs on demand with auto-scaling.

This is a demo app built with [Next.js](https://nextjs.org/) (Frontend) + [Modal](https://modal.com/) (Backend).

# Set Up

Visit [modal.com](https://modal.com/) and create a free account. Then follow the instructions to install the Modal python package and authenticate in your CLI.

## Deploy backend

We will be using Modal to deploy and serve [WhisperV3](https://github.com/openai/whisper), an audio transcription model built by OpenAI.

Execute the following commands in your terminal:

```

cd modal

modal deploy shush.py

```

This is should give you a url in the form: `https://[ORG_NAME]--[STUB_NAME]-entrypoint.modal.run`

## Deploy Frontend

Now let's run the NextJS app. After going back to the root of the repo, execute the following commands:

```

cd shush

```

Now create a `.env` file and add the url we got from Modal (view `.env.example` for reference)

Then we can just do:

```

bun i

bun run dev

```

And that's it! Open http://localhost:3000/ in your browser and test the app + model out!