{"id":20504228,"url":"https://github.com/mharrvic/serverless-llm-playground","last_synced_at":"2025-08-14T07:34:28.595Z","repository":{"id":174189104,"uuid":"651883287","full_name":"mharrvic/serverless-llm-playground","owner":"mharrvic","description":"Serverless LLM Playground with Modal","archived":false,"fork":false,"pushed_at":"2023-12-30T07:58:22.000Z","size":675,"stargazers_count":7,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T20:48:19.032Z","etag":null,"topics":["clerkauth","drizzle-orm","llm","modal","nextjs","serverless","vercel-postgres"],"latest_commit_sha":null,"homepage":"https://serverless-llm-playground.vercel.app","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mharrvic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-10T11:37:05.000Z","updated_at":"2025-03-07T13:55:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"2cc1eceb-0aa6-4250-af93-f18f02c927c0","html_url":"https://github.com/mharrvic/serverless-llm-playground","commit_stats":null,"previous_names":["mharrvic/serverless-llm-playground"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mharrvic%2Fserverless-llm-playground","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mharrvic%2Fserverless-llm-playground/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mharrvic%2Fserverless-llm-playground/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mharrvic%2Fserverless-llm-playground/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mharrvic","download_url":"https://codeload.github.com/mharrvic/serverless-llm-playground/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248782280,"owners_count":21160716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clerkauth","drizzle-orm","llm","modal","nextjs","serverless","vercel-postgres"],"created_at":"2024-11-15T19:36:57.048Z","updated_at":"2025-04-13T20:48:27.089Z","avatar_url":"https://github.com/mharrvic.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Serverless LLM Playground with Modal (Heavily Inspired by Vercel AI Playground and Nat.dev) [WIP]\n\nhttps://github.com/mharrvic/serverless-llm-playground/assets/15852818/dbbc9381-45e0-4f27-b422-5a5a1c5eb612\n\n## 1. Getting Started\n\nRequirements:\n\n1. Install packages via pnpm https://pnpm.io/installation and then `pnpm install`\n2. Setup your environment variables from `.env.example` and rename it to `.env` then provide the values\n\nModal setup\n\n1. Save the Modal LLM models to `llm/modal` directory and deploy. Here are some instructions to follow:\n\n   - https://modal.com/docs/guide/ex/openllama\n   - https://modal.com/docs/guide/ex/falcon_bitsandbytes#serve-the-model\n   - https://modal.com/docs/guide/ex/falcon_gptq\n     Make sure to add secrets on `llm-playground-secrets` collection with `AUTH_TOKEN` value for `web_endpoint` authentication layer.\n\n2. Copy the Modal URL and add it to the model config under `src/model-config.ts`\n\n## 2. Vercel Postgres with Drizzle ORM\n\nSetup your pg database https://vercel.com/storage/postgres\n\nUpdate your schema under `src/lib/db/schema.ts`\n\nTo generate migrations\n\n```bash\nnpm run migrations:generate\n```\n\nTo push the migrations to the database\n\n```bash\nnpm run migrations:push\n```\n\nTo seed\n\n```bash\nnpm run seed\n```\n\nTo delete the migrations from the database (if needed, use with caution, don't use in initial setup)\n\n```bash\nnpm run migrations:drop\n```\n\n## 3. Setup your Authentication Provider with Clerk\n\n1. Follow instructions here https://clerk.com/docs/nextjs/get-started-with-nextjs\n2. Then setup the webhook from the `api/auth-webhook` for Syncing Clerk data to the database. Read more here https://clerk.com/docs/users/sync-data-to-your-backend\n\n## 4. Run the app\n\n```bash\nnpm run dev\n```\n\nTODO:\n\n- [ ] Proxy Modal endpoint via Next API (for additional protection with Modal auth endpoint)\n- [ ] Dynamic model selection\n- [ ] Update LLM settings via UI\n- [ ] Add settings page\n- [ ] Add token count usage\n- [ ] Add support for other models from HuggingFace, OpenAI, Anthropic, Cohere, and Replicate\n\nFAQ: \u003cbr\u003e\nq: Why not use Next.js API routes instead for additional protection with Clerk and Modal endpoint?\na: Because llm generation with modal might take a while like 20-30 seconds and it will timeout the request (also this https://vercel.com/docs/concepts/limits/overview#general-limits)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmharrvic%2Fserverless-llm-playground","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmharrvic%2Fserverless-llm-playground","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmharrvic%2Fserverless-llm-playground/lists"}