https://github.com/cap-go/capacitor-llm
Capacitor SDK to run LLM models locally in IOS and Android, also enable AppleInteligence
https://github.com/cap-go/capacitor-llm
capacitor capgo ionic
Last synced: about 1 month ago
JSON representation
Capacitor SDK to run LLM models locally in IOS and Android, also enable AppleInteligence
- Host: GitHub
- URL: https://github.com/cap-go/capacitor-llm
- Owner: Cap-go
- Created: 2025-06-12T03:23:26.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-09-12T10:50:42.000Z (9 months ago)
- Last Synced: 2025-09-12T12:35:08.156Z (9 months ago)
- Topics: capacitor, capgo, ionic
- Language: Vue
- Homepage: https://capgo.app
- Size: 938 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
Awesome Lists containing this project
README
# @capgo/capacitor-llm
➡️ Get Instant updates for your App with Capgo 🚀
Fix your annoying bug now, Hire a Capacitor expert 💪
Adds support for LLM locally run for Capacitor
It uses Apple Intelligence for the iOS system model, MediaPipe for existing `.task` models, and ExecuTorch `.pte` models on both iOS and Android.
**Mac Catalyst:** Native iOS functionality is disabled for Mac Catalyst builds. MediaPipe pods are skipped and native calls will return an unsupported response; use an iOS/iPadOS target for native features.
## Documentation
The most complete doc is available here: https://capgo.app/docs/plugins/llm/
## Compatibility
| Plugin version | Capacitor compatibility | Maintained |
| -------------- | ----------------------- | ---------- |
| v8.\*.\* | v8.\*.\* | ✅ |
| v7.\*.\* | v7.\*.\* | On demand |
| v6.\*.\* | v6.\*.\* | ❌ |
| v5.\*.\* | v5.\*.\* | ❌ |
> **Note:** The major version of this plugin follows the major version of Capacitor. Use the version that matches your Capacitor installation (e.g., plugin v8 for Capacitor 8). Only the latest major version is actively maintained.
## Installation
```bash
bun add @capgo/capacitor-llm
bunx cap sync
```
### iOS Additional Setup for Custom Models
Apple Intelligence works without bundled model files on supported iOS versions. For custom models, the plugin supports MediaPipe through CocoaPods and ExecuTorch when the host app links ExecuTorch with Swift Package Manager. The plugin package remains iOS 15 compatible; iOS ExecuTorch requires an iOS 17 or newer app target because ExecuTorch requires iOS 17.
**Using CocoaPods:**
The MediaPipe dependencies are already configured in the podspec. Make sure to run `pod install` after adding the plugin.
ExecuTorch is not provided by the CocoaPods integration. CocoaPods installs MediaPipe only. CocoaPods builds reject `engine: 'executorch'` with a clear runtime error unless the app also links ExecuTorch separately.
**Note about Static Framework Warning:**
When running `pod install`, you may see a warning about transitive dependencies with statically linked binaries. To fix this, update your Podfile:
```ruby
# Change this:
use_frameworks!
# To this:
use_frameworks! :linkage => :static
# And add this to your post_install hook:
post_install do |installer|
assertDeploymentTarget(installer)
# Fix for static framework dependencies
installer.pods_project.targets.each do |target|
target.build_configurations.each do |config|
config.build_settings['BUILD_LIBRARY_FOR_DISTRIBUTION'] = 'YES'
end
# Specifically for MediaPipe pods
if target.name.include?('MediaPipeTasksGenAI')
target.build_configurations.each do |config|
config.build_settings['ENABLE_BITCODE'] = 'NO'
end
end
end
end
```
**Using Swift Package Manager:**
The plugin's `Package.swift` does not link ExecuTorch directly because that would force every Swift Package Manager user onto iOS 17 or newer, even when they only use Apple Intelligence or MediaPipe.
To use ExecuTorch on iOS, add it to your app target:
1. Set the app target deployment version to iOS 17 or newer.
2. In Xcode, open the app project, select **Package Dependencies**, and add `https://github.com/pytorch/executorch.git`.
3. Use the `swiftpm-1.2.0` branch, or the ExecuTorch branch that matches the runtime version you want.
4. Add these products to the app target: `executorch_llm`, `backend_xnnpack`, `kernels_llm`, `kernels_optimized`, `kernels_quantized`, and `kernels_torchao`.
5. Add the `.pte` model and tokenizer file to the app target's Copy Bundle Resources, or download them at runtime.
If those ExecuTorch products are not linked into the app target, `setModel({ engine: 'executorch' })` rejects with a clear runtime error. MediaPipe GenAI still does not officially support SPM, so use CocoaPods for MediaPipe `.task` models.
## Adding a Model to Your App
The simplest cross-platform custom model path is ExecuTorch. It uses the same kind of `.pte` model file on iOS and Android, plus a tokenizer file.
### ExecuTorch Models (iOS and Android)
iOS uses ExecuTorch only when the host app links the ExecuTorch Swift Package products. It is not included by CocoaPods or by the plugin's `Package.swift`, so an app without those products can use Apple Intelligence or MediaPipe but will refuse `engine: 'executorch'`. Android uses the ExecuTorch Maven package.
Bundle the model files:
- iOS: add the `.pte` model and tokenizer file to your app target's Copy Bundle Resources.
- Android: place the `.pte` model and tokenizer file under `android/app/src/main/assets/`, then reference them with `/android_asset/...`.
```typescript
import { Capacitor } from '@capacitor/core';
import { CapgoLLM } from '@capgo/capacitor-llm';
const isAndroid = Capacitor.getPlatform() === 'android';
await CapgoLLM.setModel({
engine: 'executorch',
path: isAndroid ? '/android_asset/model.pte' : 'model.pte',
tokenizerPath: isAndroid ? '/android_asset/tokenizer.model' : 'tokenizer.model',
maxTokens: 2048,
sequenceLength: 2048,
temperature: 0.8,
});
const { id: chatId } = await CapgoLLM.createChat();
```
`engine: 'auto'` also selects ExecuTorch when the model path ends in `.pte` or when `tokenizerPath` is provided. Passing `engine: 'executorch'` is recommended when loading ExecuTorch models so failures are explicit.
### MediaPipe Models
MediaPipe remains available for existing `.task` models. Android models usually need both `.task` and `.litertlm` files. iOS MediaPipe support is available through CocoaPods and remains experimental for some `.task` files.
```typescript
await CapgoLLM.setModel({
path: '/android_asset/gemma-3-270m-it-int8.task',
modelType: 'task',
maxTokens: 2048,
topk: 40,
temperature: 0.8,
});
```
### Apple Intelligence
On supported iOS devices, Apple Intelligence can be used without bundling a model.
```typescript
await CapgoLLM.setModel({
path: 'Apple Intelligence',
engine: 'apple',
});
```
### Downloading Models at Runtime
Use `downloadModel` to keep large model files out of the app bundle. For ExecuTorch, `companionUrl` can point at the tokenizer file.
```typescript
const result = await CapgoLLM.downloadModel({
url: 'https://your-server.com/models/model.pte',
companionUrl: 'https://your-server.com/models/tokenizer.model',
filename: 'model.pte',
});
await CapgoLLM.setModel({
engine: 'executorch',
path: result.path,
tokenizerPath: result.companionPath,
sequenceLength: 2048,
});
```
## Usage Example
```typescript
import { CapgoLLM } from '@capgo/capacitor-llm';
const { readiness } = await CapgoLLM.getReadiness();
console.log('LLM readiness:', readiness);
const { id: chatId } = await CapgoLLM.createChat();
CapgoLLM.addListener('textFromAi', (event) => {
console.log('AI:', event.text);
});
CapgoLLM.addListener('aiFinished', (event) => {
console.log('AI finished responding to chat:', event.chatId);
});
await CapgoLLM.sendMessage({
chatId,
message: 'Hello! How are you today?',
});
```
## API
* [`createChat()`](#createchat)
* [`sendMessage(...)`](#sendmessage)
* [`getReadiness()`](#getreadiness)
* [`setModel(...)`](#setmodel)
* [`downloadModel(...)`](#downloadmodel)
* [`addListener('textFromAi', ...)`](#addlistenertextfromai-)
* [`addListener('aiFinished', ...)`](#addlisteneraifinished-)
* [`addListener('downloadProgress', ...)`](#addlistenerdownloadprogress-)
* [`addListener('readinessChange', ...)`](#addlistenerreadinesschange-)
* [`getPluginVersion()`](#getpluginversion)
* [Interfaces](#interfaces)
LLM Plugin interface for interacting with on-device language models
### createChat()
```typescript
createChat() => Promise<{ id: string; instructions?: string; }>
```
Creates a new chat session
**Returns:** Promise<{ id: string; instructions?: string; }>
--------------------
### sendMessage(...)
```typescript
sendMessage(options: { chatId: string; message: string; }) => Promise
```
Sends a message to the AI in a specific chat session
| Param | Type | Description |
| ------------- | ------------------------------------------------- | --------------------------------- |
| **`options`** | { chatId: string; message: string; } | - The chat id and message to send |
--------------------
### getReadiness()
```typescript
getReadiness() => Promise<{ readiness: string; }>
```
Gets the readiness status of the LLM
**Returns:** Promise<{ readiness: string; }>
--------------------
### setModel(...)
```typescript
setModel(options: ModelOptions) => Promise
```
Sets the model configuration
- iOS: Use "Apple Intelligence" as path for system model, provide a MediaPipe model, or set engine to "executorch"
- Android: Path to a MediaPipe or ExecuTorch model file (in assets or files directory)
| Param | Type | Description |
| ------------- | ----------------------------------------------------- | ------------------------- |
| **`options`** | ModelOptions | - The model configuration |
--------------------
### downloadModel(...)
```typescript
downloadModel(options: DownloadModelOptions) => Promise
```
Downloads a model from a URL and saves it to the appropriate location
- iOS: Downloads to the app's documents directory
- Android: Downloads to the app's files directory
| Param | Type | Description |
| ------------- | --------------------------------------------------------------------- | ---------------------------- |
| **`options`** | DownloadModelOptions | - The download configuration |
**Returns:** Promise<DownloadModelResult>
--------------------
### addListener('textFromAi', ...)
```typescript
addListener(eventName: 'textFromAi', listenerFunc: (event: TextFromAiEvent) => void) => Promise<{ remove: () => Promise; }>
```
Adds a listener for text received from AI
| Param | Type | Description |
| ------------------ | ------------------------------------------------------------------------------- | ----------------------------------- |
| **`eventName`** | 'textFromAi' | - Event name 'textFromAi' |
| **`listenerFunc`** | (event: TextFromAiEvent) => void | - Callback function for text events |
**Returns:** Promise<{ remove: () => Promise<void>; }>
--------------------
### addListener('aiFinished', ...)
```typescript
addListener(eventName: 'aiFinished', listenerFunc: (event: AiFinishedEvent) => void) => Promise<{ remove: () => Promise; }>
```
Adds a listener for AI completion events
| Param | Type | Description |
| ------------------ | ------------------------------------------------------------------------------- | ------------------------------------- |
| **`eventName`** | 'aiFinished' | - Event name 'aiFinished' |
| **`listenerFunc`** | (event: AiFinishedEvent) => void | - Callback function for finish events |
**Returns:** Promise<{ remove: () => Promise<void>; }>
--------------------
### addListener('downloadProgress', ...)
```typescript
addListener(eventName: 'downloadProgress', listenerFunc: (event: DownloadProgressEvent) => void) => Promise<{ remove: () => Promise; }>
```
Adds a listener for model download progress events
| Param | Type | Description |
| ------------------ | ------------------------------------------------------------------------------------------- | --------------------------------------- |
| **`eventName`** | 'downloadProgress' | - Event name 'downloadProgress' |
| **`listenerFunc`** | (event: DownloadProgressEvent) => void | - Callback function for progress events |
**Returns:** Promise<{ remove: () => Promise<void>; }>
--------------------
### addListener('readinessChange', ...)
```typescript
addListener(eventName: 'readinessChange', listenerFunc: (event: ReadinessChangeEvent) => void) => Promise<{ remove: () => Promise; }>
```
Adds a listener for readiness status changes
| Param | Type | Description |
| ------------------ | ----------------------------------------------------------------------------------------- | ---------------------------------------- |
| **`eventName`** | 'readinessChange' | - Event name 'readinessChange' |
| **`listenerFunc`** | (event: ReadinessChangeEvent) => void | - Callback function for readiness events |
**Returns:** Promise<{ remove: () => Promise<void>; }>
--------------------
### getPluginVersion()
```typescript
getPluginVersion() => Promise<{ version: string; }>
```
Get the native Capacitor plugin version.
**Returns:** Promise<{ version: string; }>
**Since:** 1.0.0
--------------------
### Interfaces
#### ModelOptions
Model configuration options
| Prop | Type | Description |
| -------------------- | ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`path`** | string | Model path or "Apple Intelligence" for iOS system model |
| **`engine`** | 'auto' \| 'apple' \| 'mediapipe' \| 'executorch' | Runtime engine to use. - "auto": uses Apple Intelligence on iOS when path is "Apple Intelligence", ExecuTorch for `.pte` or tokenizerPath, otherwise MediaPipe - "apple": iOS Foundation Models / Apple Intelligence - "mediapipe": MediaPipe GenAI `.task` models - "executorch": ExecuTorch `.pte` models with a tokenizer file |
| **`modelType`** | string | Model file type/extension (e.g., "task", "bin", "litertlm"). If not provided, will be extracted from path. |
| **`tokenizerPath`** | string | Tokenizer path for ExecuTorch models. Required when engine is "executorch". |
| **`specialTokens`** | string[] | Optional special tokens passed to iOS ExecuTorch tokenizers. |
| **`maxTokens`** | number | Maximum number of tokens the model handles |
| **`sequenceLength`** | number | Sequence length for ExecuTorch generation. Defaults to maxTokens when omitted. |
| **`topk`** | number | Number of tokens the model considers at each step |
| **`temperature`** | number | Amount of randomness in generation (0.0-1.0) |
| **`randomSeed`** | number | Random seed for generation |
#### DownloadModelResult
Result of model download
| Prop | Type | Description |
| ------------------- | ------------------- | ------------------------------------------------------- |
| **`path`** | string | Path where the model was saved |
| **`companionPath`** | string | Path where the companion file was saved (if applicable) |
#### DownloadModelOptions
Options for downloading a model
| Prop | Type | Description |
| ------------------ | ------------------- | ------------------------------------------------------------- |
| **`url`** | string | URL of the model file to download |
| **`companionUrl`** | string | Optional: URL of companion file (e.g., .litertlm for Android) |
| **`filename`** | string | Optional: Custom filename (defaults to filename from URL) |
#### TextFromAiEvent
Event data for text received from AI
| Prop | Type | Description |
| ------------- | -------------------- | -------------------------------------------------------------------------- |
| **`text`** | string | The text content from AI - this is an incremental chunk, not the full text |
| **`chatId`** | string | The chat session ID |
| **`isChunk`** | boolean | Whether this is a complete chunk (true) or partial streaming data (false) |
#### AiFinishedEvent
Event data for AI completion
| Prop | Type | Description |
| ------------ | ------------------- | --------------------------------- |
| **`chatId`** | string | The chat session ID that finished |
#### DownloadProgressEvent
Event data for download progress
| Prop | Type | Description |
| --------------------- | ------------------- | ---------------------------------------- |
| **`progress`** | number | Percentage of download completed (0-100) |
| **`totalBytes`** | number | Total bytes to download |
| **`downloadedBytes`** | number | Bytes downloaded so far |
#### ReadinessChangeEvent
Event data for readiness status changes
| Prop | Type | Description |
| --------------- | ------------------- | -------------------- |
| **`readiness`** | string | The readiness status |
## Example App Model Setup
The example app can use either the older MediaPipe assets or the new ExecuTorch assets.
### Recommended Custom Path: ExecuTorch
1. Export or download an ExecuTorch `.pte` model and matching tokenizer file.
2. Add both files to the iOS app bundle, or place both files in `example-app/android/app/src/main/assets/` for Android.
3. Call `setModel` with `engine: 'executorch'`, `path`, and `tokenizerPath`.
### Legacy MediaPipe Path
Android still supports Gemma 3 LiteRT assets from Kaggle. Download both files and place them in `example-app/android/app/src/main/assets/`:
- `gemma-3-270m-it-int8.task`
- `gemma-3-270m-it-int8.litertlm`
iOS MediaPipe remains experimental because some `.task` models can fail during prefill. Apple Intelligence or ExecuTorch is preferred on iOS.
## Known Issues
- ExecuTorch is native-only and is not available on web.
- iOS ExecuTorch requires linking the ExecuTorch SwiftPM products in the app target, and that app target must support iOS 17 or newer.
- Apple Intelligence requires iOS 26.0 or later and a supported device.
- Android requires minSdkVersion 24 or higher.
- Model files are large, so production apps should usually download them after install.