{"id":15763871,"url":"https://github.com/devjiwonchoi/nextjs-openai-doc-search-starter","last_synced_at":"2025-03-31T10:19:23.535Z","repository":{"id":236513300,"uuid":"792757624","full_name":"devjiwonchoi/nextjs-openai-doc-search-starter","owner":"devjiwonchoi","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-27T13:48:06.000Z","size":110,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-11T12:03:28.138Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devjiwonchoi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-27T13:48:01.000Z","updated_at":"2024-04-27T13:48:10.000Z","dependencies_parsed_at":"2024-04-27T14:52:29.810Z","dependency_job_id":null,"html_url":"https://github.com/devjiwonchoi/nextjs-openai-doc-search-starter","commit_stats":null,"previous_names":["devjiwonchoi/nextjs-openai-doc-search-starter"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devjiwonchoi%2Fnextjs-openai-doc-search-starter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devjiwonchoi%2Fnextjs-openai-doc-search-starter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devjiwonchoi%2Fnextjs-openai-doc-search-starter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devjiwonchoi%2Fnextjs-openai-doc-search-starter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devjiwonchoi","download_url":"https://codeload.github.com/devjiwonchoi/nextjs-openai-doc-search-starter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246450478,"owners_count":20779421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-04T12:01:16.402Z","updated_at":"2025-03-31T10:19:23.511Z","avatar_url":"https://github.com/devjiwonchoi.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Next.js OpenAI Doc Search Starter\n\nThis starter takes all the `.mdx` files in the `pages` directory and processes them to use as custom context within [OpenAI Text Completion](https://platform.openai.com/docs/guides/completion) prompts.\n\n## Deploy\n\nDeploy this starter to Vercel. The Supabase integration will automatically set the required environment variables and configure your [Database Schema](./supabase/migrations/20230406025118_init.sql). All you have to do is set your `OPENAI_KEY` and you're ready to go!\n\n[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?demo-title=Next.js%20OpenAI%20Doc%20Search%20Starter\u0026demo-description=Template%20for%20building%20your%20own%20custom%20ChatGPT%20style%20doc%20search%20powered%20by%20Next.js%2C%20OpenAI%2C%20and%20Supabase.\u0026demo-url=https%3A%2F%2Fsupabase.com%2Fdocs\u0026demo-image=%2F%2Fimages.ctfassets.net%2Fe5382hct74si%2F1OntM6THNEUvlUsYy6Bjmf%2F475e39dbc84779538c8ed47c63a37e0e%2Fnextjs_openai_doc_search_og.png\u0026project-name=Next.js%20OpenAI%20Doc%20Search%20Starter\u0026repository-name=nextjs-openai-doc-search-starter\u0026repository-url=https%3A%2F%2Fgithub.com%2Fsupabase-community%2Fnextjs-openai-doc-search%2F\u0026from=github\u0026integration-ids=oac_VqOgBHqhEoFTPzGkPd7L0iH6\u0026env=OPENAI_KEY\u0026envDescription=Get%20your%20OpenAI%20API%20key%3A\u0026envLink=https%3A%2F%2Fplatform.openai.com%2Faccount%2Fapi-keys\u0026teamCreateStatus=hidden\u0026external-id=https%3A%2F%2Fgithub.com%2Fsupabase-community%2Fnextjs-openai-doc-search%2Ftree%2Fmain)\n\n## Technical Details\n\nBuilding your own custom ChatGPT involves four steps:\n\n1. [👷 Build time] Pre-process the knowledge base (your `.mdx` files in your `pages` folder).\n2. [👷 Build time] Store embeddings in Postgres with [pgvector](https://supabase.com/docs/guides/database/extensions/pgvector).\n3. [🏃 Runtime] Perform vector similarity search to find the content that's relevant to the question.\n4. [🏃 Runtime] Inject content into OpenAI GPT-3 text completion prompt and stream response to the client.\n\n## 👷 Build time\n\nStep 1. and 2. happen at build time, e.g. when Vercel builds your Next.js app. During this time the [`generate-embeddings`](./lib/generate-embeddings.ts) script is being executed which performs the following tasks:\n\n```mermaid\nsequenceDiagram\n    participant Vercel\n    participant DB (pgvector)\n    participant OpenAI (API)\n    loop 1. Pre-process the knowledge base\n        Vercel-\u003e\u003eVercel: Chunk .mdx pages into sections\n        loop 2. Create \u0026 store embeddings\n            Vercel-\u003e\u003eOpenAI (API): create embedding for page section\n            OpenAI (API)-\u003e\u003eVercel: embedding vector(1536)\n            Vercel-\u003e\u003eDB (pgvector): store embedding for page section\n        end\n    end\n```\n\nIn addition to storing the embeddings, this script generates a checksum for each of your `.mdx` files and stores this in another database table to make sure the embeddings are only regenerated when the file has changed.\n\n## 🏃 Runtime\n\nStep 3. and 4. happen at runtime, anytime the user submits a question. When this happens, the following sequence of tasks is performed:\n\n```mermaid\nsequenceDiagram\n    participant Client\n    participant Edge Function\n    participant DB (pgvector)\n    participant OpenAI (API)\n    Client-\u003e\u003eEdge Function: { query: lorem ispum }\n    critical 3. Perform vector similarity search\n        Edge Function-\u003e\u003eOpenAI (API): create embedding for query\n        OpenAI (API)-\u003e\u003eEdge Function: embedding vector(1536)\n        Edge Function-\u003e\u003eDB (pgvector): vector similarity search\n        DB (pgvector)-\u003e\u003eEdge Function: relevant docs content\n    end\n    critical 4. Inject content into prompt\n        Edge Function-\u003e\u003eOpenAI (API): completion request prompt: query + relevant docs content\n        OpenAI (API)--\u003e\u003eClient: text/event-stream: completions response\n    end\n```\n\nThe relevant files for this are the [`SearchDialog` (Client)](./components/SearchDialog.tsx) component and the [`vector-search` (Edge Function)](./pages/api/vector-search.ts).\n\nThe initialization of the database, including the setup of the `pgvector` extension is stored in the [`supabase/migrations` folder](./supabase/migrations/) which is automatically applied to your local Postgres instance when running `supabase start`.\n\n## Local Development\n\n### Configuration\n\n- `cp .env.example .env`\n- Set your `OPENAI_KEY` in the newly created `.env` file.\n- Set `NEXT_PUBLIC_SUPABASE_ANON_KEY` and `SUPABASE_SERVICE_ROLE_KEY` run:\n  \u003e Note: You have to run supabase to retrieve the keys.\n\n### Start Supabase\n\nMake sure you have Docker installed and running locally. Then run\n\n```bash\nsupabase start\n```\n\nTo retrieve `NEXT_PUBLIC_SUPABASE_ANON_KEY` and `SUPABASE_SERVICE_ROLE_KEY` run:\n\n```bash\nsupabase status\n```\n\n### Start the Next.js App\n\nIn a new terminal window, run\n\n```bash\npnpm dev\n```\n\n### Using your custom .mdx docs\n\n1. By default your documentation will need to be in `.mdx` format. This can be done by renaming existing (or compatible) markdown `.md` file.\n2. Run `pnpm run embeddings` to regenerate embeddings.\n   \u003e Note: Make sure supabase is running. To check, run `supabase status`. If is not running run `supabase start`.\n3. Run `pnpm dev` again to refresh NextJS localhost:3000 rendered page.\n\n## Learn More\n\n- Read the blogpost on how we built [ChatGPT for the Supabase Docs](https://supabase.com/blog/chatgpt-supabase-docs).\n- [[Docs] pgvector: Embeddings and vector similarity](https://supabase.com/docs/guides/database/extensions/pgvector)\n- Watch [Greg's](https://twitter.com/ggrdson) \"How I built this\" [video](https://youtu.be/Yhtjd7yGGGA) on the [Rabbit Hole Syndrome YouTube Channel](https://www.youtube.com/@RabbitHoleSyndrome):\n\n[![Video: How I Built Supabase’s OpenAI Doc Search](https://img.youtube.com/vi/Yhtjd7yGGGA/0.jpg)](https://www.youtube.com/watch?v=Yhtjd7yGGGA)\n\n## Licence\n\nApache 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevjiwonchoi%2Fnextjs-openai-doc-search-starter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevjiwonchoi%2Fnextjs-openai-doc-search-starter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevjiwonchoi%2Fnextjs-openai-doc-search-starter/lists"}