{"id":13601343,"url":"https://github.com/withcatai/node-llama-cpp","last_synced_at":"2026-01-26T05:01:35.309Z","repository":{"id":187949209,"uuid":"677847575","full_name":"withcatai/node-llama-cpp","owner":"withcatai","description":"Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation level","archived":false,"fork":false,"pushed_at":"2026-01-11T00:04:52.000Z","size":24232,"stargazers_count":1823,"open_issues_count":10,"forks_count":164,"subscribers_count":21,"default_branch":"master","last_synced_at":"2026-01-11T05:36:49.393Z","etag":null,"topics":["ai","bindings","catai","cmake","cmake-js","cuda","embedding","function-calling","gguf","gpu","grammar","json-schema","llama","llama-cpp","llm","metal","nodejs","prebuilt-binaries","self-hosted","vulkan"],"latest_commit_sha":null,"homepage":"https://node-llama-cpp.withcat.ai","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/withcatai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"giladgd","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2023-08-12T20:53:16.000Z","updated_at":"2026-01-11T01:53:52.000Z","dependencies_parsed_at":"2025-11-30T13:04:23.187Z","dependency_job_id":null,"html_url":"https://github.com/withcatai/node-llama-cpp","commit_stats":{"total_commits":174,"total_committers":6,"mean_commits":29.0,"dds":0.06321839080459768,"last_synced_commit":"4d387ded56aedf8f8b8a77509962d95f8b0d5ae8"},"previous_names":["withcatai/node-llama-cpp"],"tags_count":129,"template":false,"template_full_name":null,"purl":"pkg:github/withcatai/node-llama-cpp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/withcatai%2Fnode-llama-cpp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/withcatai%2Fnode-llama-cpp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/withcatai%2Fnode-llama-cpp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/withcatai%2Fnode-llama-cpp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/withcatai","download_url":"https://codeload.github.com/withcatai/node-llama-cpp/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/withcatai%2Fnode-llama-cpp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28767008,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T03:54:34.369Z","status":"ssl_error","status_checked_at":"2026-01-26T03:54:33.031Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","bindings","catai","cmake","cmake-js","cuda","embedding","function-calling","gguf","gpu","grammar","json-schema","llama","llama-cpp","llm","metal","nodejs","prebuilt-binaries","self-hosted","vulkan"],"created_at":"2024-08-01T18:01:00.869Z","updated_at":"2026-01-26T05:01:35.304Z","avatar_url":"https://github.com/withcatai.png","language":"TypeScript","readme":"\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://node-llama-cpp.withcat.ai\" target=\"_blank\"\u003e\u003cimg alt=\"node-llama-cpp Logo\" src=\"https://raw.githubusercontent.com/withcatai/node-llama-cpp/master/assets/logo.v3.roundEdges.avif\" width=\"360px\" /\u003e\u003c/a\u003e\n    \u003ch1\u003enode-llama-cpp\u003c/h1\u003e\n    \u003cp\u003eRun AI models locally on your machine\u003c/p\u003e\n    \u003csub\u003ePre-built bindings are provided with a fallback to building from source with cmake\u003c/sub\u003e\n    \u003cp\u003e\u003c/p\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\" class=\"main-badges\"\u003e\n\n[![Build](https://github.com/withcatai/node-llama-cpp/actions/workflows/build.yml/badge.svg)](https://github.com/withcatai/node-llama-cpp/actions/workflows/build.yml)\n[![License](https://badgen.net/badge/color/MIT/green?label=license)](https://www.npmjs.com/package/node-llama-cpp)\n[![Types](https://badgen.net/badge/color/TypeScript/blue?label=types)](https://www.npmjs.com/package/node-llama-cpp)\n[![Version](https://badgen.net/npm/v/node-llama-cpp)](https://www.npmjs.com/package/node-llama-cpp)\n\n\u003c/div\u003e\n\n✨ [`gpt-oss` is here!](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss) ✨\n\n## Features\n* Run LLMs locally on your machine\n* [Metal, CUDA and Vulkan support](https://node-llama-cpp.withcat.ai/guide/#gpu-support)\n* [Pre-built binaries are provided](https://node-llama-cpp.withcat.ai/guide/building-from-source), with a fallback to building from source _**without**_ `node-gyp` or Python\n* [Adapts to your hardware automatically](https://node-llama-cpp.withcat.ai/guide/#gpu-support), no need to configure anything\n* A Complete suite of everything you need to use LLMs in your projects\n* [Use the CLI to chat with a model without writing any code](#try-it-without-installing)\n* Up-to-date with the latest `llama.cpp`. Download and compile the latest release with a [single CLI command](https://node-llama-cpp.withcat.ai/guide/building-from-source#downloading-a-release)\n* Enforce a model to generate output in a parseable format, [like JSON](https://node-llama-cpp.withcat.ai/guide/chat-session#json-response), or even force it to [follow a specific JSON schema](https://node-llama-cpp.withcat.ai/guide/chat-session#response-json-schema)\n* [Provide a model with functions it can call on demand](https://node-llama-cpp.withcat.ai/guide/chat-session#function-calling) to retrieve information or perform actions\n* [Embedding and reranking support](https://node-llama-cpp.withcat.ai/guide/embedding)\n* [Safe against special token injection attacks](https://node-llama-cpp.withcat.ai/guide/llama-text#input-safety-in-node-llama-cpp)\n* Great developer experience with full TypeScript support, and [complete documentation](https://node-llama-cpp.withcat.ai/guide/)\n* Much more\n\n## [Documentation](https://node-llama-cpp.withcat.ai)\n* [Getting started guide](https://node-llama-cpp.withcat.ai/guide/)\n* [API reference](https://node-llama-cpp.withcat.ai/api/functions/getLlama)\n* [CLI help](https://node-llama-cpp.withcat.ai/cli/)\n* [Blog](https://node-llama-cpp.withcat.ai/blog/)\n* [Changelog](https://github.com/withcatai/node-llama-cpp/releases)\n* [Roadmap](https://github.com/orgs/withcatai/projects/1)\n\n## Try It Without Installing\nChat with a model in your terminal using [a single command](https://node-llama-cpp.withcat.ai/cli/chat):\n```bash\nnpx -y node-llama-cpp chat\n```\n\n## Installation\n```bash\nnpm install node-llama-cpp\n```\n\n[This package comes with pre-built binaries](https://node-llama-cpp.withcat.ai/guide/building-from-source) for macOS, Linux and Windows.\n\nIf binaries are not available for your platform, it'll fallback to download a release of `llama.cpp` and build it from source with `cmake`.\nTo disable this behavior, set the environment variable `NODE_LLAMA_CPP_SKIP_DOWNLOAD` to `true`.\n\n## Usage\n```typescript\nimport {fileURLToPath} from \"url\";\nimport path from \"path\";\nimport {getLlama, LlamaChatSession} from \"node-llama-cpp\";\n\nconst __dirname = path.dirname(fileURLToPath(import.meta.url));\n\nconst llama = await getLlama();\nconst model = await llama.loadModel({\n    modelPath: path.join(__dirname, \"models\", \"Meta-Llama-3.1-8B-Instruct.Q4_K_M.gguf\")\n});\nconst context = await model.createContext();\nconst session = new LlamaChatSession({\n    contextSequence: context.getSequence()\n});\n\n\nconst q1 = \"Hi there, how are you?\";\nconsole.log(\"User: \" + q1);\n\nconst a1 = await session.prompt(q1);\nconsole.log(\"AI: \" + a1);\n\n\nconst q2 = \"Summarize what you said\";\nconsole.log(\"User: \" + q2);\n\nconst a2 = await session.prompt(q2);\nconsole.log(\"AI: \" + a2);\n```\n\n\u003e For more examples, see the [getting started guide](https://node-llama-cpp.withcat.ai/guide/)\n\n## Contributing\nTo contribute to `node-llama-cpp` read the [contribution guide](https://node-llama-cpp.withcat.ai/guide/contributing).\n\n## Acknowledgements\n* llama.cpp: [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp)\n\n\n\u003cbr /\u003e\n\n\u003cdiv align=\"center\" width=\"360\"\u003e\n    \u003cimg alt=\"Star please\" src=\"https://raw.githubusercontent.com/withcatai/node-llama-cpp/master/assets/star.please.roundEdges.png\" width=\"360\" margin=\"auto\" /\u003e\n    \u003cbr/\u003e\n    \u003cp align=\"right\"\u003e\n        \u003ci\u003eIf you like this repo, star it ✨\u003c/i\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n    \u003c/p\u003e\n\u003c/div\u003e\n","funding_links":["https://github.com/sponsors/giladgd"],"categories":["A01_文本生成_文本对话","TypeScript","ai","self-hosted"],"sub_categories":["大语言对话模型及数据"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwithcatai%2Fnode-llama-cpp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwithcatai%2Fnode-llama-cpp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwithcatai%2Fnode-llama-cpp/lists"}