https://github.com/mybigday/llama.node

Node.js binding of Llama.cpp
https://github.com/mybigday/llama.node

llama llama-cpp llama2 llamacpp node-js nodejs

Last synced: about 5 hours ago
JSON representation

Node.js binding of Llama.cpp

• Host: GitHub
• URL: https://github.com/mybigday/llama.node
• Owner: mybigday
• Created: 2024-04-23T19:43:12.000Z (5 months ago)
• Default Branch: main
• Last Pushed: 2024-09-16T03:18:55.000Z (9 days ago)
• Last Synced: 2024-09-18T18:25:24.388Z (6 days ago)
• Topics: llama, llama-cpp, llama2, llamacpp, node-js, nodejs
• Language: C++
• Homepage:
• Size: 28.3 MB
• Stars: 5
• Watchers: 1
• Forks: 0
• Open Issues: 8
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# llama.node

[![CI](https://github.com/mybigday/llama.node/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/mybigday/llama.node/actions/workflows/ci.yml)

[![NPM Version](https://img.shields.io/npm/v/%40fugood%2Fllama.node)](https://www.npmjs.com/package/@fugood/llama.node)

![NPM Downloads](https://img.shields.io/npm/dw/%40fugood%2Fllama.node)

Node binding of [llama.cpp](https://github.com/ggerganov/llama.cpp).

[llama.cpp](https://github.com/ggerganov/llama.cpp): Inference of [LLaMA](https://arxiv.org/abs/2302.13971) model in pure C/C++

## Installation

```sh

npm install @fugood/llama.node

```

## Usage

```js

import { loadModel } from '@fugood/llama.node'

// Initial a Llama context with the model (may take a while)

const context = await loadModel({

  model: 'path/to/gguf/model',

  use_mlock: true,

  n_ctx: 2048,

  n_gpu_layers: 1, // > 0: enable GPU

  // embedding: true, // use embedding

  // lib_variant: 'opencl', // Change backend

})

// Do completion

const { text } = await context.completion(

  {

    prompt: 'This is a conversation between user and llama, a friendly chatbot. respond in simple markdown.\n\nUser: Hello!\nLlama:',

    n_predict: 100,

    stop: ['', 'Llama:', 'User:'],

    // n_threads: 4,

  },

  (data) => {

    // This is a partial completion callback

    const { token } = data

  },

)

console.log('Result:', text)

```

## Lib Variants

- [x] `default`: General usage, not support GPU except macOS (Metal)

- [x] `vulkan`: Support GPU Vulkan (Windows/Linux), but some scenario might unstable

## License

MIT

---



  

    

  

  


    Built and maintained by BRICKS.