{"id":14964762,"url":"https://github.com/tib0/llama3-wrapper","last_synced_at":"2026-02-14T17:31:29.237Z","repository":{"id":244829493,"uuid":"792832687","full_name":"tib0/llama3-wrapper","owner":"tib0","description":"Node Llama Cpp wrapper for Node JS","archived":false,"fork":false,"pushed_at":"2025-03-07T16:04:47.000Z","size":1161,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-25T08:45:08.441Z","etag":null,"topics":["llama3","llamacpp","node","wrapper"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tib0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-27T17:17:19.000Z","updated_at":"2024-09-25T09:17:23.000Z","dependencies_parsed_at":"2024-06-17T19:03:33.449Z","dependency_job_id":"ad3ae0c9-fa90-4a0e-af34-5acb90cc741e","html_url":"https://github.com/tib0/llama3-wrapper","commit_stats":{"total_commits":19,"total_committers":2,"mean_commits":9.5,"dds":"0.052631578947368474","last_synced_commit":"e92a7aa634b0b18960d447387f07aaee12a624c2"},"previous_names":["tib0/llama3-wrapper"],"tags_count":0,"template":false,"template_full_name":"tib0/typescript-node-module-template","purl":"pkg:github/tib0/llama3-wrapper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tib0%2Fllama3-wrapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tib0%2Fllama3-wrapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tib0%2Fllama3-wrapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tib0%2Fllama3-wrapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tib0","download_url":"https://codeload.github.com/tib0/llama3-wrapper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tib0%2Fllama3-wrapper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29450878,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-14T15:52:44.973Z","status":"ssl_error","status_checked_at":"2026-02-14T15:52:11.208Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llama3","llamacpp","node","wrapper"],"created_at":"2024-09-24T13:33:44.649Z","updated_at":"2026-02-14T17:31:29.222Z","avatar_url":"https://github.com/tib0.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"![logo](./.github/llama3-wrapper-banner.png)\n\n# Llama3-Wrapper\n\nA wrapper class for interacting with a LLaMA model instance loaded locally.\nUse any gguf models, you can find models on [huggingface.co](https://huggingface.co/models?search=gguf).\nThis projects is based on llama.cpp by [ggerganov](https://github.com/ggerganov) used via node-llama-cpp by [withcatai](https://github.com/withcatai/).\n\n## METHODS\n\n### Constructor\n\nCreates a new instance of the LlamaWrapper class, assigning it a unique ID and initializing its internal state.\n\n### loadModule()\n\nLoads the LLaMA module required for interacting with the model. Throws an error if the module cannot be loaded.\n\n### loadLlama(gpu?: Gpu)\n\nLoads the LLaMA library instance, optionally specifying a GPU device to use. Throws an error if the module isn't initialized.\n\n### loadModel(modelPath: string)\n\nLoads a specific LLaMA model from the specified modelPath. Throws an error if Llama isn't loaded.\n\n### initSession(systemPrompt: string)\n\nInitializes a new chat session with the wrapped LLaMA model using the specified systemPrompt. Throws an error if the model isn't initialized.\n\n### getStatus()\n\nReturns the current status of the wrapper as an object containing the status and optional message. The status can be one of LlamaStatusType (Uninitialized, Ready, Loading, Generating or Error).\n\n### isReady()\n\nReturns a boolean indicating whether the wrapper is in ready state.\n\n### prompt(message: string, onToken?: (chunk: string) =\u003e void)\n\nGenerates an answer to the specified message and calls the optional onToken callback function with each generated chunk. Throws an error if the session isn't initialized.\n\n### getHistory()\n\nRetrieves the chat history associated with this wrapper's current session. Throws an error if no session is found.\n\n### setHistory(chatHistoryItem: ChatHistoryItem[]): Promise\n\nSets the chat history associated with this wrapper's current session to the specified chatHistoryItem array. Throws an error if no session is found.\n\n### getId()\n\nReturns the unique ID assigned to this wrapper instance.\n\n### getInfos()\n\nReturns an object containing various information about the wrapped LLaMA model, including its ID, model filename, train context size, and more.\n\n### disposeSession()\n\nDisposes of the current chat session associated with this wrapper. Throws an error if no session is found.\n\n### clearHistory()\n\nClears the history of the current chat sequence associated with this wrapper. Throws an error if no session or sequence is found.\n\n## TYPES\n\n### Gpu\n\nThe `Gpu` type represents a GPU device to use.\n\n- **auto**: The default value, which means the library will automatically select the best available GPU.\n- **cuda**: A CUDA-enabled GPU.\n- **vulkan**: A VULKAN-enabled GPU.\n- **metal**: A METAL-enabled GPU.\n- **false**: Skip GPU usage.\n\n### LlamaStatusType\n\nThe `LlamaStatusType` type defines the possible status values for a llama wrapper instance.\n\n- **uninitialized**: The initial state of a new llama session.\n- **loading**: The session is loading a model or initializing session.\n- **ready**: The session is ready to use.\n- **error**: An error occurred during initialization or loading.\n- **generating**: The session is currently generating a response to a message.\n\n### ChatHistoryItem\n\nThe `ChatHistoryItem` type represents a single item in the chat history, which contains the message and its corresponding response. Can be ChatSystemMessage | ChatUserMessage | ChatModelResponse\n\n- **type**: The original message sent by the user. can be 'system', 'user' or 'model'.\n- **text**: Represent the stored user text or system prompt. This property comes from ChatSystemMessage and ChatUserMessage types.\n- **response**: Represent the model answer, comes from ChatModelResponse type.\n\n## BUILDING FROM SOURCE\n\nBy following the steps below, you can build and install the module from source code.\n\n1. Clone the repository:\n\n```sh\ngit clone https://github.com/tib0/llama3-wrapper.git\n```\n\n2. Install dependencies:\n\n```sh\ncd ./llama3-wrapper\npnpm i\n```\n\n3. Build the module:\n\n```sh\npnpm build\n```\n\n4. Link the module globally:\n\n```sh\npnpm link -g\n```\n\n5. In the target project folder use the module:\n\n```sh\ncd /path/to/target-project\npnpm link -g llama3-wrapper\n```\n\n## CONFIGURATION\n\nAdd your GGUF model path in a .env file at the root of your project:\n\n```sh\nLLAMA_MODELS_PATH=/Users/me/example/LLM/Models/my-model-file.gguf\n```\n\n## EXAMPLE\n\nSample chat-like usage in terminal:\n\n```ts\nimport { type ChatHistoryItem, LlamaWrapper } from 'llama3-wrapper';\nimport readline from 'readline';\nimport { spawn } from 'node:child_process';\n\nconst rl = readline.createInterface({\n  input: process.stdin,\n  output: process.stdout,\n});\n\nconst run = async () =\u003e {\n  console.log(`# START LLAMA CHAT`);\n  console.log(`\\n`);\n\n  console.log(`# Feeding history traces`);\n  const history: ChatHistoryItem[] = [\n    { type: 'user', text: 'Hey.' },\n    { type: 'model', response: ['Hello !'] },\n  ];\n\n  console.log(`# Waiting seat allocation`);\n\n  const llamaNodeCPP = new LlamaWrapper();\n  await llamaNodeCPP.loadModule();\n  await llamaNodeCPP.loadLlama();\n  await llamaNodeCPP.loadModel(process.env.LLAMA_MODELS_PATH);\n  await llamaNodeCPP.initSession(promptSystem);\n  console.log(`# Prompt ready`);\n\n  console.log(`# Activated TTS (voice)`);\n\n  console.log(`\\n`);\n  rl.setPrompt('1 \u003e ');\n  rl.prompt();\n  let i = 1;\n\n  rl.on('line', async (q) =\u003e {\n    if (!q || q === '' || q === 'exit' || q === 'quit' || q === 'q') {\n      rl.close();\n    } else {\n      const a = await llamaNodeCPP.prompt(q);\n      console.log(`${i} @ ${a}`);\n      spawn('say', [a]);\n      console.log(`\\n`);\n      i++;\n    }\n    rl.setPrompt(`${i} \u003e `);\n    rl.prompt();\n  }).on('close', async () =\u003e {\n    console.log(`\\n`);\n    console.log(`Disposing session...`);\n    await llamaNodeCPP.disposeSession();\n    console.log(`\\n`);\n    const a = await llamaNodeCPP.getHistory();\n    console.log(`History:`);\n    console.log(JSON.stringify(a));\n\n    console.log(`\\n`);\n    console.log('# END LLAMA CHAT');\n\n    process.exit(0);\n  });\n};\n\nrun();\n```\n\n# RESSOURCES\n\n- [https://github.com/withcatai/node-llama-cpp](https://github.com/withcatai/node-llama-cpp): Run AI models locally on your machine with node.js bindings for llama.cpp. Force a JSON schema on the model output on the generation level.\n- [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp): official implementation of LLaMA in C++.\n- [https://huggingface.co](https://huggingface.co): let's you explore the models and datasets available on the Hub.\n- [https://github.com/facebookresearch/llama](https://github.com/facebookresearch/llama): official implementation of LLaMA.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftib0%2Fllama3-wrapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftib0%2Fllama3-wrapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftib0%2Fllama3-wrapper/lists"}