Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/darksorrow/llamarn

React Native Turbo Module for llama.cpp integration, optimized for the New Architecture
https://github.com/darksorrow/llamarn

ai llamacpp react-native

Last synced: 3 months ago
JSON representation

React Native Turbo Module for llama.cpp integration, optimized for the New Architecture

Awesome Lists containing this project

README 

          
# LlamaRN



A thin, reliable React Native Turbo Module wrapping [llama.cpp](https://github.com/ggml-org/llama.cpp) for on-device LLM inference on iOS and Android.



## Features



- Model loading, text completion, and chat completion

- Metal GPU on iOS; OpenCL / Vulkan / Hexagon NPU on Android

- Automatic CPU/GPU detection and optimal GPU layer estimation

- Chat completion with Jinja template support

- Multi-turn KV cache prefix reuse (pass a stable `id` per message)

- Embeddings generation

- Function / tool calling

- Thinking and reasoning model support (`reasoning_budget`, `reasoning_format`)

- **Multimodal / Vision** — image-in-prompt chat, CLIP-style embeddings, Whisper-style transcription, open-ended vision reasoning, and zero-copy camera frame pipeline



## Breaking changes



See **[BREAKING.md](./BREAKING.md)** for migration notes (thinking models / `reasoning_content`, KV cache behavior, GGUF + Jinja + tools pitfalls) and the **v0.7.0** release summary.



## Installation



```sh

npm install @novastera-oss/llamarn

```



## Developer Setup



### Prerequisites



1. Clone the repository

2. React Native development environment for your target platform(s)



### Initial Setup



```sh

npm install

npm run setup-llama-cpp        # init llama.cpp submodule

```



### Android



```sh

./scripts/build_android_external.sh   # build native libraries

cd example && npm run android

```



### iOS



```sh

cd example/ios && bundle exec pod install

cd .. && npm run ios

```



**Troubleshooting:**

- Android: `cd android && ./gradlew clean`

- iOS: `cd example/ios && rm -rf build Podfile.lock && pod install`



---



## Basic Usage



### Simple Completion



```js

import { initLlama } from '@novastera-oss/llamarn';



const context = await initLlama({

  model: '/path/to/model.gguf',

  n_ctx: 2048,

  n_batch: 512,

});



const result = await context.completion({

  prompt: 'What is artificial intelligence?',

  temperature: 0.7,

  top_p: 0.95,

});



console.log(result.text);

```



### Chat Completion



```js

const context = await initLlama({

  model: '/path/to/model.gguf',

  n_ctx: 4096,

  use_jinja: true,

});



const result = await context.completion({

  messages: [

    { role: 'system', content: 'You are a helpful assistant.' },

    { role: 'user',   content: 'Tell me about quantum computing.' },

  ],

  temperature: 0.7,

});



console.log(result.text);

// OpenAI-compatible: result.choices[0].message.content

```



### Multi-Turn Chat with KV Cache Reuse



Assign a stable `id` to each message. The native layer re-uses KV cache entries for messages whose ID matches the previous turn — only new tokens are encoded.



```js

const history = [

  { role: 'system',    content: 'You are a helpful assistant.', id: 'sys-1'    },

  { role: 'user',      content: 'Hi!',                          id: 'turn-1-u' },

];



const r1 = await context.completion({ messages: history });



history.push({ role: 'assistant', content: r1.text,         id: 'turn-1-a' });

history.push({ role: 'user',      content: 'Tell me more.', id: 'turn-2-u' });



// Only [turn-2-u] is encoded; everything before hits the cache

const r2 = await context.completion({ messages: history });

```



**Rules:**

- Messages without an `id` are always fully re-encoded.

- If you edit a message's content, change its `id` so the cache is invalidated.

- `resetKvCache: true` forces a full clear.



```js

await context.completion({ messages: history, resetKvCache: true });

```



### Thinking and Reasoning Models



```js

const context = await initLlama({

  model: '/path/to/reasoning-model.gguf',

  n_ctx: 4096,

  use_jinja: true,

  reasoning_budget: -1,          // -1 = unlimited, 0 = disabled, >0 = token limit

  reasoning_format: 'deepseek',  // 'none' | 'auto' | 'deepseek' | 'deepseek-legacy'

  thinking_forced_open: true,

});



const result = await context.completion({

  messages: [{ role: 'user', content: 'Solve: what is 15% of 240?' }],

});



// result.text may include … depending on the model

```



See **[BREAKING.md](./BREAKING.md)** for multi-turn thinking, KV cache, and tool-calling pitfalls.



### Tool Calling



```js

const context = await initLlama({

  model: '/path/to/model.gguf',

  n_ctx: 2048,

  use_jinja: true,          // parse_tool_calls enabled automatically

  parallel_tool_calls: false,

});



const response = await context.completion({

  messages: [

    { role: 'system', content: 'You are a helpful assistant.' },

    { role: 'user',   content: "What's the weather in Paris?" },

  ],

  tools: [{

    type: 'function',

    function: {

      name: 'get_weather',

      description: 'Get the current weather in a location',

      parameters: {

        type: 'object',

        properties: {

          location: { type: 'string' },

          unit:     { type: 'string', enum: ['celsius', 'fahrenheit'] },

        },

        required: ['location'],

      },

    },

  }],

  tool_choice: 'auto',

});



if (response.choices?.[0]?.finish_reason === 'tool_calls') {

  const call = response.choices[0].message.tool_calls[0];

  console.log(call.function.name, call.function.arguments);

}

```



### Embeddings



```js

const context = await initLlama({

  model: '/path/to/embedding-model.gguf',

  embedding: true,

  n_ctx: 2048,

});



const { data } = await context.embedding({ input: 'Text to embed' });

console.log(data[0].embedding); // Float32 array

```



---



## Multimodal / Vision



llamarn supports image and audio input for compatible models (LLaVA, Qwen-VL, Whisper) via a separate **multimodal projection model** (`mmproj` `.gguf`).



### Init with capabilities



```typescript

const model = await initLlama({

  model:        '/path/to/model.gguf',

  mmproj:       '/path/to/mmproj.gguf',

  capabilities: ['vision-chat'],  // 'image-encode' | 'audio-transcribe' | 'vision-reasoning'

  n_ctx: 4096,

  use_jinja: true,

});



const mods = await model.getSupportedModalities();

// { vision: true, audio: false }



const enabled = await model.isMultimodalEnabled();

// true

```



**Supported capabilities:**



| Value | Description |

|-------|-------------|

| `vision-chat` | Image-in-prompt chat completions (LLaVA, Qwen-VL) |

| `image-encode` | CLIP-style image embeddings |

| `audio-transcribe` | Whisper-style audio → text |

| `vision-reasoning` | Open-ended image analysis returning raw model text |



### Vision Chat



```typescript

const result = await model.completion({

  messages: [{

    role: 'user',

    content: [

      { type: 'text',      text: 'What is in this image?' },

      { type: 'image_url', image_url: { url: 'file:///path/to/image.jpg' } },

    ],

  }],

  n_predict: 256,

});



console.log(result.text);

```



Multiple images per message are supported. `file://` URIs and `data:image/...;base64,...` strings are both accepted. KV cache prefix reuse is automatically disabled when messages contain images.



### Image Embeddings



```typescript

const { embedding, n_tokens, n_embd } = await model.embedImage(

  'file:///path/to/image.jpg',

  { normalize: true },

);

// embedding: flat Float32 array, length = n_tokens * n_embd

```



### Audio Transcription



```typescript

const { text } = await model.transcribeAudio('file:///path/to/audio.wav');

```



### Open-ended Vision Reasoning



```typescript

const { raw_text } = await model.visionReasoning(

  'file:///path/to/image.jpg',

  { prompt: 'List all objects you see.' },

);

```



### Camera Frame (zero-copy)



Works with [VisionCamera](https://github.com/mrousavy/react-native-vision-camera) NativeBuffer frames — no disk I/O required.



```typescript

import { useFrameProcessor, runAsync } from 'react-native-vision-camera';



const frameProcessor = useFrameProcessor((frame) => {

  'worklet';

  runAsync(frame, async () => {

    // maxSize: 336 downsamples to ~330 KB during the pixel copy (recommended for LLaVA/CLIP)

    // Omit maxSize (or 0) for full-resolution analysis (OCR, detailed scenes)

    const result = await model.runOnFrame(

      frame, frame.width, frame.height, 'image-encode', { maxSize: 336 });



    if (result) {

      // result.embedding — Float32 array (n_tokens × n_embd)

      console.log('Tokens:', result.n_tokens);

    }

    // null means the encoder was busy with the previous frame — dropped automatically

  });

}, [model]);

```



Frame dropping is handled automatically in C++ via an atomic flag — no JS-side throttle needed.



**Supported model families:**



| Model | Capability |

|-------|-----------|

| LLaVA 1.5 / 1.6 | vision-chat |

| Qwen2-VL / Qwen3-VL | vision-chat |

| LLaMA 4 Scout | vision-chat |

| MiniCPM-V | vision-chat |

| Whisper | audio-transcribe |

| CLIP | image-encode |



---



## Model Info



Query a model's properties without fully loading it:



```typescript

import { loadLlamaModelInfo } from '@novastera-oss/llamarn';



const info = await loadLlamaModelInfo('/path/to/model.gguf');



console.log(info.description);       // human-readable model name + quant

console.log(info.n_params);          // parameter count

console.log(info.n_layers);          // total layer count

console.log(info.model_size_bytes);  // quantized on-disk size

console.log(info.optimalGpuLayers);  // recommended n_gpu_layers for this device

console.log(info.availableMemoryMB); // current free device RAM in MB

console.log(info.estimatedVramMB);   // estimated VRAM needed for optimalGpuLayers

console.log(info.architecture);      // e.g. "llama", "qwen2", "mistral"

console.log(info.gpuSupported);      // true if GPU offload is available

```



---



## Advanced Configuration



### GPU and Memory Parameters



| Parameter | Default | Description |

|-----------|---------|-------------|

| `n_gpu_layers` | `0` | Layers to offload to GPU (use `optimalGpuLayers` from `loadLlamaModelInfo`) |

| `n_ctx` | `2048` | Context window size |

| `n_batch` | `512` | Prompt processing batch size |

| `use_mmap` | `true` | Memory-mapped loading (faster startup) |

| `use_mlock` | `false` | Lock model in RAM (prevents swapping) |



### Thinking and Reasoning Parameters



| Parameter | Default | Description |

|-----------|---------|-------------|

| `reasoning_budget` | `-1` | Token budget for thinking: `-1` unlimited, `0` disabled, `>0` limited |

| `reasoning_format` | `'auto'` | `'none'` \| `'auto'` \| `'deepseek'` \| `'deepseek-legacy'` |

| `thinking_forced_open` | `false` | Force thinking tags in every response |

| `parse_tool_calls` | `true` | Parse tool call JSON from output (auto-enabled with `use_jinja`) |

| `parallel_tool_calls` | `false` | Allow multiple tool calls per response |



---



## Model Path Handling



### iOS



- Bundle path (Xcode resource): `${RNFS.MainBundlePath}/model.gguf`

- Absolute path: `/var/mobile/…/model.gguf`



### Android



- Cache directory: `${RNFS.CachesDirectoryPath}/model.gguf`

- App assets (copied to cache first): `RNFS.copyFileAssets('model.gguf', dest)`



---



## Documentation



- [Interface Documentation](INTERFACE.md) — Detailed API interfaces

- [Breaking Changes](BREAKING.md) — Migration notes

- [Example App](example/) — Working example with text chat and vision demo

- [Contributing Guide](CONTRIBUTING.md)



---



## About Novastera



**LlamaRN** is part of the **Novastera** open-source ecosystem. This library powers on-device LLM inference in [Novastera's](https://novastera.com) mobile applications — no data leaves the user's device.



Learn more: [https://novastera.com/resources](https://novastera.com/resources)



## License



Apache 2.0



## Acknowledgments



- **[mybigday/llama.rn](https://github.com/mybigday/llama.rn)** — foundational React Native binding for llama.cpp

- **[ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp)** — the core C++ inference library

- **[react-native-pure-cpp-turbo-module-library](https://github.com/Zach-Dean-Attractions-io/react-native-pure-cpp-turbo-module-library)** — reference for the Android C++ Turbo Module pattern