https://github.com/crabnebula-dev/tauri-plugin-llm
Tauri plugin to load and interact with most LLMs
https://github.com/crabnebula-dev/tauri-plugin-llm
Last synced: 6 months ago
JSON representation
Tauri plugin to load and interact with most LLMs
- Host: GitHub
- URL: https://github.com/crabnebula-dev/tauri-plugin-llm
- Owner: crabnebula-dev
- License: other
- Created: 2025-12-02T10:27:56.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2026-01-27T10:31:56.000Z (6 months ago)
- Last Synced: 2026-01-27T22:38:34.477Z (6 months ago)
- Language: Rust
- Size: 532 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE_POLYFORM-NONCOM
- Security: SECURITY.md
Awesome Lists containing this project
README
# Tauri Plugin LLM
This Tauri plugin allows loading and running inference on various large language models.
| Platform | Supported |
| -------- | --------- |
| Linux | ✓ |
| Windows | ✓ |
| macOS | ✓ |
| Android | ? |
| iOS | ? |
# Requirements
- Rust >= 1.77
- Go >= 1.20
# Install
# Usage
The plugin is not bundled with any LLM. The LLM must be shipped separately. In order to load an LLM, a specific configuration must be provided for the plugin. The JSON example configuration sets `Qwen3-4B-GGUF` as model to be loaded.
File `tauri.conf.json` :
```json
{
"build": { ... },
"tauri": { ... },
"plugins": {
"tauri-plugin-llm": {
"tokenizer_file": "./models/Qwen3-4B-Instruct-2507-FP8/tokenizer.json",
"tokenizer_config_file": "./models/Qwen3-4B-Instruct-2507-FP8/tokenizer_config.json",
"model_config_file": "./models/Qwen3-4B-Instruct-2507-FP8/config.json",
"model_index_file": "",
"model_file": "./models/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf",
"model_dir": "./models/Qwen3-4B-GGUF/",
"model_config": {
"top_k": 20,
"top_p": 0.8,
"temperature": 0.4,
"name": "Qwen3-4B-GGUF",
"file_type": "GGUF",
"penalty": 1.0,
"seed": "Random",
"thinking": false,
"streaming": true
},
"verbose": true
}
}
}
```
> **Note**: _The example above shows a model present at some folder called `model`. The model itself will not be shipped with the plugin._
The `LLMRuntime` will try to load an LLM and defer the initialization process to the actual loader. The Tauri Plugin LLM does not support all available models, yet. Internally, the model will be run in a dedicated thread to avoid blocking the main thread.
```Rust
// For demonstration purposes we load the configuration from the tests.
// Normally you would load the tauri config
let config = LLMRuntimeConfig::from_path("tests/fixtures/test_runtime_qwen3.config.json")?;
// Initialize the Runtime and detect available specific runtimes
let mut runtime = LLMRuntime::from_config(config)?;
// Starting the runtime. This will internally initialize the model and
// then run inference on the model on the incoming message.
runtime.run();
if let Err(_) = runtime.send(
// We will send a simple text prompt and instruct the model to just return anything we wrote
Query::Prompt {
messages: vec![QueryMessage {
role: "user".to_string(),
content: "Hello, World".to_string() },
QueryMessage {
role: "system".to_string(),
content: "You are a helpful assistant. Your task is to echo the incoming message. Do not describe anything. ".to_string()},
],
tools: vec![],
config: Some(QueryConfig::default()),
}) {
loop {
// We can continously check, if a message from the model is available
if let Ok(message) = runtime.retry_recv() {
tracing::info!("Received Message : {:?}", message);
break;
}
}
}
Ok(())
```