{"id":13632870,"url":"https://github.com/r2d4/react-llm","last_synced_at":"2025-04-04T14:09:09.579Z","repository":{"id":165412008,"uuid":"639710310","full_name":"r2d4/react-llm","owner":"r2d4","description":"Easy-to-use headless React Hooks to run LLMs in the browser with WebGPU. Just useLLM().","archived":false,"fork":false,"pushed_at":"2023-06-27T21:20:05.000Z","size":2881,"stargazers_count":691,"open_issues_count":3,"forks_count":30,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-03-28T13:09:58.059Z","etag":null,"topics":["headless","llm","react","webgpu"],"latest_commit_sha":null,"homepage":"https://chat.matt-rickard.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/r2d4.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-05-12T03:54:53.000Z","updated_at":"2025-03-25T06:05:32.000Z","dependencies_parsed_at":"2023-07-19T14:32:27.427Z","dependency_job_id":null,"html_url":"https://github.com/r2d4/react-llm","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r2d4%2Freact-llm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r2d4%2Freact-llm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r2d4%2Freact-llm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r2d4%2Freact-llm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/r2d4","download_url":"https://codeload.github.com/r2d4/react-llm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247190253,"owners_count":20898702,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["headless","llm","react","webgpu"],"created_at":"2024-08-01T22:03:20.638Z","updated_at":"2025-04-04T14:09:09.559Z","avatar_url":"https://github.com/r2d4.png","language":"TypeScript","funding_links":[],"categories":["TypeScript","LLM Inference"],"sub_categories":[],"readme":"# @react-llm/headless\n\nEasy-to-use headless React Hooks to run LLMs in the browser with WebGPU. As simple as `useLLM()`.\n\n### [**Live Demo**](https://chat.matt-rickard.com)\n\n![image](assets/demo.webp)\n\n**Features**:\n\n* Supports [Vicuna 7B](https://lmsys.org/blog/2023-03-30-vicuna/)\n* Use custom system prompts and \"user:\"/\"assistant:\" role names\n* Completion options like `max tokens` and `stop sequences`\n* No data leaves the browser. Accelerated via WebGPU.\n* Hooks built to 'Bring your own UI'\n* Persistent storage for conversations in browser storage. Hooks for loading and saving conversations.\n* Model caching for faster subsequent loads\n\n## Installation\n\n```bash\nnpm install @react-llm/headless\n```\n\n## Packages in this repository\n- [@react-llm/model](packages/model) - The LLM model and tokenizer compiled for the browser\n- [@react-llm/retro-ui](packages/retro-ui) - Retro-themed UI for the hooks\n- [@react-llm/extension](packages/extension) - Chrome Extension that uses the hooks\n- [@react-llm/headless](packages/headless) - Headless React Hooks for running LLMs in the browser\n\n\n## **useLLM** API\n### Types\n```typescript\n// Model Initialization\ninit: () =\u003e void;\n\n// Model Generation\nsend: (msg: string, maxTokens: number, stopSequences: string[]) =\u003e void;\nonMessage: (msg: GenerateTextResponse) =\u003e void;\nsetOnMessage: (cb: (msg: GenerateTextResponse) =\u003e void) =\u003e void;\n\n// Model Status\nloadingStatus: InitProgressReport;\nisGenerating: boolean;\ngpuDevice: GPUDeviceInfo;\n\n// Model Configuration\nuserRoleName: string;\nsetUserRoleName: (roleName: string) =\u003e void;\nassistantRoleName: string;\nsetAssistantRoleName: (roleName: string) =\u003e void;\n\n// Conversation Management\nconversation: Conversation | undefined;\nallConversations: Conversation[] | undefined;\ncreateConversation: (title?: string, prompt?: string) =\u003e void;\nsetConversationId: (conversationId: string) =\u003e void;\ndeleteConversation: (conversationId: string) =\u003e void;\ndeleteAllConversations: () =\u003e void;\ndeleteMessages: () =\u003e void;\nsetConversationTitle: (conversationId: string, title: string) =\u003e void;\n```\n\n### Hooks\n```typescript\nimport useLLM from '@react-llm/headless';\n\nconst MyComponent = () =\u003e {\n  const {\n    conversation,\n    allConversations,\n    loadingStatus,\n    isGenerating,\n    createConversation,\n    setConversationId,\n    deleteConversation,\n    deleteAllConversations,\n    deleteMessages,\n    setConversationTitle,\n    onMessage,\n    setOnMessage,\n    userRoleName,\n    setUserRoleName,\n    assistantRoleName,\n    setAssistantRoleName,\n    gpuDevice,\n    send,\n    init,\n  } = useLLM();\n\n  // Component logic...\n\n  return null;\n};\n```\n\n### Provider\n```typescript\nimport { ModelProvider } from \"@react-llm/headless\";\n\nexport default function Home() {\n  return (\n    \u003cModelProvider\n      config={{\n        kvConfig: {\n          numLayers: 64,\n          shape: [32, 32, 128],\n          dtype: 'float32',\n        },\n        wasmUrl: 'https://your-custom-url.com/model.wasm',\n        cacheUrl: 'https://your-custom-url.com/cache/',\n        tokenizerUrl: 'https://your-custom-url.com/tokenizer.model',\n        sentencePieceJsUrl: 'https://your-custom-url.com/sentencepiece.js',\n        tvmRuntimeJsUrl: 'https://your-custom-url.com/tvmjs_runtime.wasi.js',\n        maxWindowSize: 2048,\n        persistToLocalStorage: true,\n      }}\n    \u003e\n      \u003cChat /\u003e\n    \u003c/ModelProvider\u003e\n  );\n}\n```\n\n### Packages\n\n* `@react-llm/headless` - Headless React Hooks for running LLMs in the browser\n* `@react-llm/retro-ui` - Retro-themed UI for the hooks\n\n## How does it work?\n\nThis library is a set of React Hooks that provide a simple interface to run LLMs in the browser. It uses Vicuna 13B.\n\n* SentencePiece tokenizer (compiled for the browser via Emscripten)\n* Vicuna 7B (transformed to Apache TVM format)\n* Apache TVM and MLC Relax (compiled for the browser via Emscripten)\n* Off-the-main-thread WebWorker to run the model (bundled with the library)\n\n\nThe model, tokenizer, and TVM runtime are loaded from a CDN (huggingface). The model is cached in browser storage for faster subsequent loads.\n\n\n\n\n### Example\nSee [packages/retro-ui](packages/retro-ui) for the full demo code. This is a simple example of how to use the hooks. To run it, after cloning the repo,\n\n```bash\ncd packages/retro-ui\npnpm install\npnpm dev\n```\n\n\n### License\nMIT\n\nThe code under `packages/headless/worker/lib/tvm` is licensed under Apache 2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fr2d4%2Freact-llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fr2d4%2Freact-llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fr2d4%2Freact-llm/lists"}