Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/jimmylv/chatvox
"Chat With Any Video" project in 24 hours, challenge myself to complete in @Supabase's AI Hackathon.
https://github.com/jimmylv/chatvox
ai chatgpt openai supabase video whisper
Last synced: 28 days ago
JSON representation
"Chat With Any Video" project in 24 hours, challenge myself to complete in @Supabase's AI Hackathon.
- Host: GitHub
- URL: https://github.com/jimmylv/chatvox
- Owner: JimmyLv
- License: gpl-3.0
- Created: 2023-04-16T05:54:19.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-06-02T04:39:50.000Z (7 months ago)
- Last Synced: 2024-11-16T19:55:49.848Z (about 1 month ago)
- Topics: ai, chatgpt, openai, supabase, video, whisper
- Language: TypeScript
- Homepage: https://chatvox.aitodo.co
- Size: 1.84 MB
- Stars: 100
- Watchers: 6
- Forks: 15
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# ChatVox - "Chat With Any Video"
> based on Supabase's [Next.js OpenAI Doc Search Starter](https://github.com/supabase-community/nextjs-openai-doc-search?og=v2)
![ChatVox Cover Image](/public/og-image.png)
ChatVox ยท Chat With Any Video: This project takes the YouTube/Podcast link you enter or the local Video/Audio file (WIP) you upload on the page, then you can chat with the video, just like ChatGPT! ๐คฉ
Talking to a video is essentially talking to this person, It meant "chat with wisdom" ๐คฃ
I'm updating AI Hackathon on Twitter in real-time, This is the showcase of the whole process. https://twitter.com/Jimmy_JingLv/status/1647483720608415744?s=20
The technology behind it is to process them to use as the custom context within [OpenAI Text Completion](https://platform.openai.com/docs/guides/completion) prompts and LangChain to handle the document searching and Q&A answering.
## Deploy
Deploy this starter to Vercel. The Supabase integration will automatically set the required environment variables and configure your [Database Schema](./supabase/migrations/20230406025118_init.sql). All you have to do is set your `OPENAI_API_KEY` and you're ready to go!
[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?demo-title=Next.js%20OpenAI%20Doc%20Search%20Starter&demo-description=Template%20for%20building%20your%20own%20custom%20ChatGPT%20style%20doc%20search%20powered%20by%20Next.js%2C%20OpenAI%2C%20and%20Supabase.&demo-url=https%3A%2F%2Fsupabase.com%2Fdocs&demo-image=%2F%2Fimages.ctfassets.net%2Fe5382hct74si%2F1OntM6THNEUvlUsYy6Bjmf%2F475e39dbc84779538c8ed47c63a37e0e%2Fnextjs_openai_doc_search_og.png&project-name=Next.js%20OpenAI%20Doc%20Search%20Starter&repository-name=nextjs-openai-doc-search-starter&repository-url=https%3A%2F%2Fgithub.com%2FJimmyLv%2FChatVox%2F&from=github&integration-ids=oac_jUduyjQgOyzev1fjrW83NYOv&env=OPENAI_API_KEY&envDescription=Get%20your%20OpenAI%20API%20key%3A&envLink=https%3A%2F%2Fplatform.openai.com%2Faccount%2Fapi-keys&teamCreateStatus=hidden&external-id=nextjs-open-ai-doc-search)
## Technical Details
Building your own custom ChatGPT involves four steps:
1. [๐ท Build time] Pre-process the knowledge base (your `.mdx` files in your `pages` folder).
2. [๐ท Build time] Store embeddings in Postgres with [pgvector](https://supabase.com/docs/guides/database/extensions/pgvector).
3. [๐ Runtime] Perform vector similarity search to find the content that's relevant to the question.
4. [๐ Runtime] Inject content into OpenAI GPT-3 text completion prompt and stream response to the client.## ๐ท Build time
Step 1. and 2. happen at build time, e.g. when Vercel builds your Next.js app. During this time the [`generate-embeddings`](./lib/generate-embeddings.ts) script is being executed which performs the following tasks:
```mermaid
sequenceDiagram
participant Vercel
participant DB (pgvector)
participant OpenAI (API)
loop 1. Pre-process the knowledge base
Vercel->>Vercel: Chunk .mdx pages into sections
loop 2. Create & store embeddings
Vercel->>OpenAI (API): create embedding for page section
OpenAI (API)->>Vercel: embedding vector(1536)
Vercel->>DB (pgvector): store embedding for page section
end
end
```In addition to storing the embeddings, this script generates a checksum for each of your `.mdx` files and stores this in another database table to make sure the embeddings are only regenerated when the file has changed.
## ๐ Runtime
Step 3. and 4. happen at runtime, anytime the user submits a question. When this happens, the following sequence of tasks is performed:
```mermaid
sequenceDiagram
participant Client
participant Edge Function
participant DB (pgvector)
participant OpenAI (API)
Client->>Edge Function: { query: lorem ispum }
critical 3. Perform vector similarity search
Edge Function->>OpenAI (API): create embedding for query
OpenAI (API)->>Edge Function: embedding vector(1536)
Edge Function->>DB (pgvector): vector similarity search
DB (pgvector)->>Edge Function: relevant docs content
end
critical 4. Inject content into prompt
Edge Function->>OpenAI (API): completion request prompt: query + relevant docs content
OpenAI (API)-->>Client: text/event-stream: completions response
end
```The relevant files for this are the [`SearchDialog` (Client)](./components/SearchDialog.tsx) component and the [`vector-search` (Edge Function)](./pages/api/vector-search.ts).
The initialization of the database, including the setup of the `pgvector` extension is stored in the [`supabase/migrations` folder](./supabase/migrations/) which is automatically applied to your local Postgres instance when running `supabase start`.
## Local Development
### Configuration
- `cp .env.example .env`
- Set your `OPENAI_API_KEY` in the newly created `.env` file.### Start Supabase
Make sure you have Docker installed and running locally. Then run
```bash
supabase start
```### Start the Next.js App
In a new terminal window, run
```bash
pnpm dev
```## Deploy
Deploy this starter to Vercel. The Supabase integration will automatically set the required environment variables and configure your [Database Schema](./supabase/migrations/20230406025118_init.sql). All you have to do is set your `OPENAI_API_KEY` and you're ready to go!
[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?demo-title=Next.js%20OpenAI%20Doc%20Search%20Starter&demo-description=Template%20for%20building%20your%20own%20custom%20ChatGPT%20style%20doc%20search%20powered%20by%20Next.js%2C%20OpenAI%2C%20and%20Supabase.&demo-url=https%3A%2F%2Fsupabase.com%2Fdocs&demo-image=%2F%2Fimages.ctfassets.net%2Fe5382hct74si%2F1OntM6THNEUvlUsYy6Bjmf%2F475e39dbc84779538c8ed47c63a37e0e%2Fnextjs_openai_doc_search_og.png&project-name=Next.js%20OpenAI%20Doc%20Search%20Starter&repository-name=nextjs-openai-doc-search-starter&repository-url=https%3A%2F%2Fgithub.com%2Fsupabase-community%2Fnextjs-openai-doc-search%2F&from=github&integration-ids=oac_jUduyjQgOyzev1fjrW83NYOv&env=OPENAI_API_KEY&envDescription=Get%20your%20OpenAI%20API%20key%3A&envLink=https%3A%2F%2Fplatform.openai.com%2Faccount%2Fapi-keys&teamCreateStatus=hidden&external-id=nextjs-open-ai-doc-search)
## Learn More
- Read the blogpost on how we built [ChatGPT for the Supabase Docs](https://supabase.com/blog/chatgpt-supabase-docs).
- [[Docs] pgvector: Embeddings and vector similarity](https://supabase.com/docs/guides/database/extensions/pgvector)
- Watch [Greg's](https://twitter.com/ggrdson) "How I built this" [video](https://youtu.be/Yhtjd7yGGGA) on the [Rabbit Hole Syndrome YouTube Channel](https://www.youtube.com/@RabbitHoleSyndrome):[![Video: How I Built Supabaseโs OpenAI Doc Search](https://img.youtube.com/vi/Yhtjd7yGGGA/0.jpg)](https://www.youtube.com/watch?v=Yhtjd7yGGGA)