Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/scientist-labs/red-candle

Ruby gem for running state-of-the-art language models locally. Access LLMs, embeddings, rerankers, and NER models directly from Ruby using Rust-powered Candle with Metal/CUDA acceleration.
https://github.com/scientist-labs/red-candle

huggingface-candle magnus ruby

Last synced: 3 months ago
JSON representation

Ruby gem for running state-of-the-art language models locally. Access LLMs, embeddings, rerankers, and NER models directly from Ruby using Rust-powered Candle with Metal/CUDA acceleration.

Awesome Lists containing this project

README 

          
red-candle



[![build](https://github.com/scientist-labs/red-candle/actions/workflows/build.yml/badge.svg)](https://github.com/scientist-labs/red-candle/actions/workflows/build.yml)

[![Gem Version](https://badge.fury.io/rb/red-candle.svg)](https://badge.fury.io/rb/red-candle)



Run state-of-the-art **language models directly from Ruby**. No Python, no APIs, no external services - just Ruby with blazing-fast Rust under the hood. Hardware accelerated with **Metal (Mac)** and **CUDA (NVIDIA).** Red candle leverages the Rust ecosystem, notably [Candle](https://github.com/huggingface/candle) and [Magnus](https://github.com/matsadler/magnus), to provide a fast and efficient way to run LLMs in Ruby. See [Dependencies](#dependencies) for more.



## Install & Chat in 30 Seconds



[![red-candle quickstart](https://img.youtube.com/vi/hbyFCyh8esk/0.jpg)](https://www.youtube.com/watch?v=hbyFCyh8esk)



```bash

# Install the gem

gem install red-candle

```



```ruby

require 'candle'



# Download a model (one-time, ~650MB) - Mistral, Llama3, Gemma all work!

llm = Candle::LLM.from_pretrained("TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF",

                                  gguf_file: "tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf")



# Chat with it - no API calls, running locally in your Ruby process!

messages = [

  { role: "user", content: "Explain Ruby in one sentence" }

]



puts llm.chat(messages)

# => "Ruby is a dynamic, object-oriented programming language known for its

#     simplicity, elegance, and productivity, often used for web development

#     with frameworks like Rails."

```



## What Just Happened?



You just ran a 1.1-billion parameter AI model inside Ruby. The model lives in your process memory, runs on your hardware (CPU/GPU), and responds instantly without network latency.



## Stream Responses Like a Pro



```ruby

# Watch the AI think in real-time

llm.chat_stream(messages) do |token|

  print token

end

```



## Why This Matters



- **Privacy**: Your data never leaves your machine

- **Speed**: No network overhead, direct memory access

- **Control**: Fine-tune generation parameters, access raw tokens

- **Integration**: It's just Ruby objects - use it anywhere Ruby runs



## Supports



- **Tokenizers**: Access the tokenizer directly

- **EmbeddingModel**: Generate embeddings for text

- **Reranker**: Rerank documents based on relevance

- **NER**: Named Entity Recognition directly from Ruby

- **LLM**: Chat with Large Language Models (e.g., Llama, Mistral, Gemma, Qwen, Phi)

- **Structured Generation**: Generate JSON from a schema or match a regular expression



## Model Storage



Models are automatically downloaded and cached when you first use them. They are stored in:

- **Location**: `~/.cache/huggingface/hub/`

- **Size**: Models range from ~100MB (embeddings) to several GB (LLMs)

- **Reuse**: Models are downloaded once and reused across sessions



To check your cache or manage storage:

```bash

# View cache contents

ls -la ~/.cache/huggingface/hub/



# Check total cache size

du -sh ~/.cache/huggingface/



# Clear cache if needed (removes all downloaded models)

rm -rf ~/.cache/huggingface/hub/

```



----



## Usage



```ruby

require "candle"



x = Candle::Tensor.new([1, 2, 3, 4, 5, 6], :i64)

x = x.reshape([3, 2])

# [[1., 2.],

#  [3., 4.],

#  [5., 6.]]

# Tensor[[3, 2], f32]

```



```ruby

require 'candle'



# Default model (JinaBERT) on CPU

model = Candle::EmbeddingModel.from_pretrained

embedding = model.embedding("Hi there!")



# Specify device (CPU, Metal, or CUDA)

device = Candle::Device.cpu     # or Candle::Device.metal, Candle::Device.cuda

model = Candle::EmbeddingModel.from_pretrained("jinaai/jina-embeddings-v2-base-en", device: device)

embedding = model.embedding("Hi there!")



# Reranker also supports device selection

reranker = Candle::Reranker.from_pretrained("cross-encoder/ms-marco-MiniLM-L-12-v2", device: device)

results = reranker.rerank("query", ["doc1", "doc2", "doc3"])

```



## LLM Support



Red-Candle now supports Large Language Models (LLMs) with GPU acceleration!



### Supported Models



- **Gemma**: Google's Gemma models (e.g., `google/gemma-2b`, `google/gemma-7b`, `google/gemma-2b-it`)

- **Llama**: Llama 2 and Llama 3 models (e.g., `TinyLlama/TinyLlama-1.1B-Chat-v1.0`, `meta-llama/Llama-2-7b-hf`, `NousResearch/Llama-2-7b-hf`)

- **Mistral**: All Mistral models (e.g., `mistralai/Mistral-7B-Instruct-v0.1`)

- **Qwen**: Qwen 2 and 2.5 models (e.g., `Qwen/Qwen2-1.5B`, `Qwen/Qwen2.5-7B-Instruct`)

- **Phi**: Microsoft's Phi-2, Phi-3, Phi-3.5, and Phi-4 models (e.g., `microsoft/phi-2`, `microsoft/Phi-3-mini-4k-instruct`, `microsoft/phi-4`)

  - ⚠️ ⚠️ ⚠️ Note: Phi-3 and Phi-4 GGUF models have a known issue with KV cache persistence between generations. The `reset_cache` parameter doesn't work for GGUF models. Recreate the model instance for each generation.

  - `candle` pull request about phi-3 gguf models: https://github.com/huggingface/candle/pull/2937



### Quantized Model Support (GGUF)



Red-Candle supports quantized models in GGUF format, offering 4-8x memory reduction:



> **Note on GGUF Support**: Red-Candle now uses a unified GGUF loader that automatically detects the model architecture from the GGUF file. This means all GGUF models (including Mistral models from TheBloke) should now work correctly! The loader automatically selects the appropriate tokenizer based on the model type to ensure proper text generation.



```ruby

# Load quantized models - always specify the GGUF filename

llm = Candle::LLM.from_pretrained("TheBloke/Llama-2-7B-Chat-GGUF",

                                  device: device,

                                  gguf_file: "llama-2-7b-chat.Q4_K_M.gguf")



# Register custom tokenizer mappings for your models

Candle::LLM.register_tokenizer("my-org/my-model-GGUF", "my-org/my-tokenizer")



# Popular quantized model sources:

# - TheBloke: Extensive collection of GGUF models

# - Search HuggingFace for "GGUF" models

```



**Memory usage comparison (7B models):**

- Full precision: ~28 GB

- Q8_0 (8-bit): ~7 GB - Best quality, larger size

- Q5_K_M (5-bit): ~4.5 GB - Very good quality

- Q4_K_M (4-bit): ~4 GB - Recommended default, best balance

- Q3_K_M (3-bit): ~3 GB - Good for memory-constrained systems



**Quantization levels explained:**

- **Q8_0**: Almost identical to full model, use when quality is paramount

- **Q5_K_M**: Excellent quality with good compression

- **Q4_K_M**: Best balance of quality/size/speed (recommended default)

- **Q3_K_M**: Noticeable quality reduction but very compact

- **Q2_K**: ⚠️ **Not recommended** - Can cause inference errors due to extreme quantization



> **Warning**: Q2_K quantization can lead to "weight is negative, too large or not a valid number" errors during inference. Use Q3_K_M or higher for stable operation.



> ### ⚠️ Huggingface login warning

>

> Many models, including the one below, require you to agree to the terms. You'll need to:

> 1. Login to [Huggingface](https://huggingface.co)

> 2. Agree to the terms. For example: [here](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)

> 3. Authenticate your session. Simplest way is with `huggingface-cli login`. Detail here: [Huggingface CLI](https://huggingface.co/docs/huggingface_hub/en/guides/cli)

>

> More details here: [Huggingface Authentication](docs/HUGGINGFACE.md)



```ruby

require 'candle'



# Choose your device

device = Candle::Device.cpu     # CPU (default)

device = Candle::Device.metal   # Apple GPU (Metal)

device = Candle::Device.cuda    # NVIDIA GPU (CUDA)



# Load a model

llm = Candle::LLM.from_pretrained("google/gemma-2b-it", device: device)  # Gemma

# llm = Candle::LLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0", device: device)  # Llama

# llm = Candle::LLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1", device: device)  # Mistral



# Generate text

response = llm.generate("What is Ruby?", config: Candle::GenerationConfig.balanced)



# Stream generation

llm.generate_stream("Tell me a story", config: Candle::GenerationConfig.balanced) do |token|

  print token

end



# Chat interface

messages = [

  { role: "system", content: "You are a helpful assistant." },

  { role: "user", content: "Explain Ruby in one sentence." }

]

response = llm.chat(messages)

```



### GPU Acceleration



We see an 18x speed up running LLMs under CUDA vs CPU and a >3x speed up running under Metal vs CPU. Details [here](docs/DEVICE_SUPPORT.md#performance-considerations).



```ruby

# CPU works for all models

device = Candle::Device.cpu

llm = Candle::LLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0", device: device)



# Metal

device = Candle::Device.metal



# CUDA support (for NVIDIA GPUs)

device = Candle::Device.cuda   # Linux/Windows with NVIDIA GPU

```



### Debugging Token Generation



For debugging purposes, you can enable raw token output to see both token IDs and their raw representations:



```ruby

# Enable debug mode to see raw tokens during generation

config = Candle::GenerationConfig.balanced(debug_tokens: true)



# Non-streaming generation with debug tokens

result = llm.generate("Hello, world!", config: config)

puts result

# Output: [15043:Hello][11:,][1917:world][0:!]



# Streaming generation with debug tokens

llm.generate_stream("Hello, world!", config: config) do |text|

  print text  # Will show each token as it's generated: [15043:Hello][11:,][1917:world][0:!]

end



# Works with all models (Llama, Mistral, Gemma, and quantized GGUF models)

```



This is particularly useful for:

- Debugging tokenization issues

- Understanding how the model processes text

- Troubleshooting generation problems

- Analyzing model behavior



## Structured Generation



Red Candle supports structured generation to constrain LLM outputs to follow specific patterns like JSON schemas or regular expressions:



```ruby

# Define a JSON schema

schema = {

  type: "object",

  properties: {

    answer: { type: "string", enum: ["yes", "no"] },

    confidence: { type: "number", minimum: 0, maximum: 1 }

  },

  required: ["answer"]

}



# Generate and parse in one step

result = llm.generate_structured("Is Ruby easy to learn?", schema: schema)

puts result["answer"]      # "yes"

puts result["confidence"]  # 0.9



# Or use regex patterns for non-JSON outputs

phone_constraint = llm.constraint_from_regex('\d{3}-\d{3}-\d{4}')

config = Candle::GenerationConfig.balanced(constraint: phone_constraint)

phone = llm.generate("Generate a phone number:", config: config)

```



See [STRUCTURED_GENERATION.md](docs/STRUCTURED_GENERATION.md) for detailed documentation.



**Note on Reliability**: Structured generation constrains the model's output tokens, but success rates vary by model size and schema complexity. Smaller models (< 7B parameters) may occasionally produce incomplete or invalid JSON, especially with complex schemas. Consider implementing retry logic or fallback strategies in production applications. Larger models generally perform much better with structured generation.



## Tool Calling



Red-candle supports tool/function calling, enabling models to invoke external functions during generation. This works best with models fine-tuned for tool calling, such as Qwen3.



### Defining Tools



```ruby

get_weather = Candle::Tool.new(

  name: "get_weather",

  description: "Get the current weather for a city",

  parameters: {

    type: "object",

    properties: { city: { type: "string", description: "City name" } },

    required: ["city"]

  }

) { |args| { city: args["city"], temperature: 72, condition: "sunny" } }

```



### Extracting Tool Calls



`chat_with_tools` injects tool definitions into the system prompt, generates a response, and parses any `` tags from the output. It does **not** feed results back to the model — it just tells you what the model wants to call. You decide what to do with it:



```ruby

llm = Candle::LLM.from_pretrained("Qwen/Qwen3-0.6B")



messages = [{ role: "user", content: "What's the weather in San Francisco?" }]

result = llm.chat_with_tools(messages, tools: [get_weather],

  config: Candle::GenerationConfig.deterministic(max_length: 500))



if result.has_tool_calls?

  result.tool_calls.each do |tc|

    puts "#{tc.name}(#{tc.arguments})"

    output = get_weather.call(tc.arguments)

    puts "=> #{output}"

  end

else

  puts result.text_response

end

```



Pass `execute: true` to automatically run the tools (but still no round-trip back to the model):



```ruby

result = llm.chat_with_tools(messages, tools: [get_weather], execute: true,

  config: Candle::GenerationConfig.deterministic(max_length: 500))



result.tool_results.each do |tr|

  puts "#{tr[:tool_call].name} => #{tr[:result]}"

end

```



### Agent (Multi-Turn Tool Loop)



`Candle::Agent` completes the round-trip: generate → parse tool calls → execute → feed results back to the model → repeat until the model produces a final text answer or hits `max_iterations`. This is a convenience wrapper for quick prototyping — for production use, frameworks like [RubyLLM](https://github.com/crmne/ruby_llm) manage this loop for you via the [ruby_llm-red_candle](https://github.com/scientist-labs/ruby_llm-red_candle) plugin:



```ruby

agent = Candle::Agent.new(llm, tools: [get_weather, lookup_price], max_iterations: 5)

result = agent.run("What's the weather in Paris, and how much does a widget cost?",

  config: Candle::GenerationConfig.deterministic(max_length: 1000))



puts result.response         # Final text answer from the model

puts result.iterations       # Number of generate cycles

puts result.tool_calls_made  # Number of tools invoked

```



### Model Recommendations



Tool calling quality depends heavily on model size:



| Model | Tool Calling Quality |

|-------|---------------------|

| **Qwen3-8B GGUF** (~5 GB) | Calls correct tools, self-corrects errors, but may hallucinate values from tool results |

| **Qwen3-4B GGUF** (~2.5 GB) | Calls correct tools, occasional reasoning errors |

| **Qwen3-0.6B** (~1.2 GB) | Single-turn works, needs `max_length: 500+` for thinking |

| SmolLM2-360M | Does not work |

| TinyLlama-1.1B | Does not work (not fine-tuned for tool calling) |



**Tip:** Qwen3 models use a `` reasoning block before producing tool calls. Set `max_length` high enough (500+ for 0.6B, 1000+ for larger models) to allow room for both thinking and the tool call.



## ⚠️ Model Format Requirements



### EmbeddingModels and Rerankers: Safetensors Only



Red-Candle **only supports embedding models and rerankers that provide their weights in the [safetensors](https://github.com/huggingface/safetensors) format** (i.e., the model repo must contain a `model.safetensors` file). If the model repo does not provide the required file, loading will fail with a clear error. Most official BERT and DistilBERT models do **not** provide safetensors; many Sentence Transformers and JinaBERT models do.



**If you encounter an error like:**



```

RuntimeError: model.safetensors not found after download. Only safetensors models are supported. Please ensure your model repo contains model.safetensors.

```



this means the selected model is not compatible. Please choose a model repo that provides the required file.



### LLMs: Safetensors and GGUF Support



LLM models support two formats:

1. **Safetensors format** - Standard HuggingFace models (e.g., `TinyLlama/TinyLlama-1.1B-Chat-v1.0`)

2. **GGUF quantized format** - Memory-efficient quantized models (e.g., `TheBloke/Llama-2-7B-Chat-GGUF`)



See the [Quantized Model Support](#quantized-model-support-gguf) section for details on using GGUF models.



## Supported Embedding Models



Red-Candle supports the following embedding model types from Hugging Face:



1. `Candle::EmbeddingModelType::JINA_BERT` - Jina BERT models (e.g., `jinaai/jina-embeddings-v2-base-en`) (**safetensors required**)

2. `Candle::EmbeddingModelType::MINILM` - MINILM models (e.g., `sentence-transformers/all-MiniLM-L6-v2`) (**safetensors required**)

3. `Candle::EmbeddingModelType::DISTILBERT` - DistilBERT models (e.g., `distilbert-base-uncased-finetuned-sst-2-english`) (**safetensors required**)

4. `Candle::EmbeddingModelType::STANDARD_BERT` - Standard BERT models (e.g., `scientistcom/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext`) (**safetensors required**)



> **Note:** Most official BERT and DistilBERT models do _not_ provide safetensors. Please check the model repo before use.



You can get a list of all supported model types and suggested models paths:



```ruby

Candle::EmbeddingModelType.all  # Returns all supported model types

Candle::EmbeddingModelType.suggested_model_paths  # Returns hash of suggested models for each type

```



## A note on memory usage

The default model (`jinaai/jina-embeddings-v2-base-en` with the `sentence-transformers/all-MiniLM-L6-v2` tokenizer, both from [HuggingFace](https://huggingface.co)) takes a little more than 3GB of memory running on a Mac. The memory stays with the instantiated `Candle::EmbeddingModel` class, if you instantiate more than one, you'll use more memory. Likewise, if you let it go out of scope and call the garbage collector, you'll free the memory. For example:



```ruby

> require 'candle'

# Ruby memory = 25.9 MB

> model = Candle::EmbeddingModel.from_pretrained

# Ruby memory = 3.50 GB

> model2 = Candle::EmbeddingModel.from_pretrained

# Ruby memory = 7.04 GB

> model2 = nil

> GC.start

# Ruby memory = 3.56 GB

> model = nil

> GC.start

# Ruby memory = 55.2 MB

```



## A note on returned embeddings



The code should match the same embeddings when generated from the python `transformers` library. For instance, locally I was able to generate the same embedding for the text "Hi there!" using the python code:



```python

from transformers import AutoModel

model = AutoModel.from_pretrained('jinaai/jina-embeddings-v2-base-en', trust_remote_code=True)

sentence = ['Hi there!']

embedding = model.encode(sentence)

print(embedding)

```



And the following ruby:



```ruby

require 'candle'

model = Candle::EmbeddingModel.from_pretrained

embedding = model.embedding("Hi there!")

```



## Document Reranking



Red-Candle includes support for cross-encoder reranking models, which can be used to reorder documents by relevance to a query. This is particularly useful for improving search results or implementing retrieval-augmented generation (RAG) systems.



### Supported Models



Red-Candle supports both **BERT** and **XLM-RoBERTa** reranker architectures. The model type is auto-detected from `config.json`.



| Model | Architecture | Params | Notes |

|-------|-------------|--------|-------|

| `BAAI/bge-reranker-base` | XLM-RoBERTa | 278M | Recommended — strong quality, multilingual |

| `BAAI/bge-reranker-large` | XLM-RoBERTa | 560M | Best quality, higher resource usage |

| `BAAI/bge-reranker-v2-m3` | XLM-RoBERTa | 278M | Multilingual, very strong |

| `cross-encoder/ms-marco-MiniLM-L-12-v2` | BERT | 33M | Lightweight, English only |



### Basic Usage



```ruby

require 'candle'



# Initialize the reranker (BGE reranker recommended for quality)

reranker = Candle::Reranker.from_pretrained("BAAI/bge-reranker-base")



# Or use the lighter BERT-based model

reranker = Candle::Reranker.from_pretrained("cross-encoder/ms-marco-MiniLM-L-12-v2")



# Or with custom max_length for truncation (default is 512)

reranker = Candle::Reranker.from_pretrained(

  "cross-encoder/ms-marco-MiniLM-L-12-v2",

  max_length: 256  # Faster processing with less context

)



# Define your query and candidate documents

query = "How many people live in London?"

documents = [

  "London is known for its financial district",

  "Around 9 Million people live in London",

  "The weather in London is often rainy",

  "London is the capital of England"

]



# Rerank documents by relevance to the query (raw logits)

ranked_results = reranker.rerank(query, documents, pooling_method: "pooler", apply_sigmoid: false)



# Or apply sigmoid activation to get scores between 0 and 1

sigmoid_results = reranker.rerank(query, documents, pooling_method: "pooler", apply_sigmoid: true)



# The pooler method is the default and is recommended for cross-encoders, as is apply_sigmoid, so the above is the same as:

ranked_results = reranker.rerank(query, documents)



# Results are returned as an array of hashes, sorted by relevance

e.g.

ranked_results.each do |result|

  puts "Score: #{result[:score].round(4)} - Doc ##{result[:doc_id]}: #{result[:text]}"

end

# Output:

# Score: 1.0 - Doc #1: Around 9 Million people live in London

# Score: 0.0438 - Doc #3: London is the capital of England

# Score: 0.0085 - Doc #0: London is known for its financial district

# Score: 0.0005 - Doc #2: The weather in London is often rainy

```



### Arguments & Activation Functions



By default, `apply_sigmoid` is `true` (scores between 0 and 1). Set it to `false` to get raw logits. You can also select the pooling method:



- `pooling_method: "pooler"` (default)

- `pooling_method: "cls"`

- `pooling_method: "mean"`



Example without sigmoid activation:



```ruby

# Get raw logits

ranked_results = reranker.rerank(query, documents, apply_sigmoid: false)



ranked_results.each do |result|

  puts "Score: #{result[:score].round(4)} - Doc ##{result[:doc_id]}: #{result[:text]}"

end

# Output:

# Score: 10.3918 - Doc #1: Around 9 Million people live in London

# Score: -3.0829 - Doc #3: London is the capital of England

# Score: -4.7619 - Doc #0: London is known for its financial district

# Score: -7.5251 - Doc #2: The weather in London is often rainy

```



### Output Format



The reranker returns an array of hashes, each with the following keys:

- `:text` – The original document text

- `:score` – The relevance score (raw logit or sigmoid-activated)

- `:doc_id` – The original 0-based index of the document in the input array



This format is compatible with the Informers gem, which returns results as hashes with `:doc_id` and `:score` keys. The `doc_id` allows you to map results back to your original data structure.



### Pooling Methods



The reranker supports different pooling strategies for aggregating BERT embeddings:



```ruby

# Use alternative pooling methods

# "pooler" (default) - Uses the pooler layer with tanh activation (most accurate for cross-encoders)

# "cls" - Uses raw [CLS] token embeddings without the pooler layer

# "mean" - Mean pooling across all tokens (not recommended for cross-encoders)



# With raw logits

results = reranker.rerank_with_pooling(query, documents, "cls")



# With sigmoid activation

results = reranker.rerank_sigmoid_with_pooling(query, documents, "cls")

```



Note: Pooling methods only apply to BERT-based models. XLM-RoBERTa models (e.g., BGE rerankers) have a built-in classification head and ignore the `pooling_method` parameter. For BERT models, the default "pooler" method is recommended as it matches how cross-encoder models are trained.



### CUDA Support



For faster inference on NVIDIA GPUs:



```ruby

# Initialize with CUDA if available (falls back to CPU if not)

reranker = Candle::Reranker.from_pretrained("cross-encoder/ms-marco-MiniLM-L-12-v2", cuda: true)

```



### How It Works



Cross-encoder reranking models differ from bi-encoder embedding models:



- **Bi-encoders** (like the embedding models above) encode queries and documents separately into dense vectors

- **Cross-encoders** process the query and document together, allowing for more nuanced relevance scoring



The reranker concatenates the query and document with special tokens and processes them jointly through transformer layers to produce a single relevance score.



**BERT models** (e.g., MiniLM): Use a pooler layer (dense + tanh) on the [CLS] token, then a classifier layer. Pooling method is configurable (`pooler`, `cls`, `mean`).



**XLM-RoBERTa models** (e.g., BGE rerankers): Use a built-in classification head that returns logits directly. The `pooling_method` parameter is ignored — the model handles its own pooling internally.



This joint processing allows cross-encoders to capture subtle semantic relationships between queries and documents, making them more accurate for reranking tasks, though at the cost of higher computational requirements.



### Performance Considerations



**Important**: The Reranker automatically truncates documents to ensure stable performance. The default maximum is 512 tokens, but this is configurable.



#### Configurable Truncation



You can adjust the `max_length` parameter to balance performance and context:



```ruby

# Default: 512 tokens (maximum context, ~300ms per doc on CPU)

reranker = Candle::Reranker.from_pretrained(model_id)



# Faster: 256 tokens (~60% faster, ~120ms per doc on CPU)

reranker = Candle::Reranker.from_pretrained(model_id, max_length: 256)



# Fastest: 128 tokens (~80% faster, ~60ms per doc on CPU)

reranker = Candle::Reranker.from_pretrained(model_id, max_length: 128)

```



Choose based on your needs:

- **512 tokens**: Maximum context for complex queries (default)

- **256 tokens**: Good balance of speed and context

- **128 tokens**: Fast processing for simple matching



#### Performance Guidelines



1. **Document Length**: Documents longer than ~400 words will be truncated

   - The first 512 tokens (roughly 300-400 words) are used

   - Consider splitting very long documents into chunks if full coverage is needed



2. **Batch Size**: Process multiple documents in one call for efficiency

   ```ruby

   # Good: Single call with multiple documents

   results = reranker.rerank(query, documents)



   # Less efficient: Multiple calls

   documents.map { |doc| reranker.rerank(query, [doc]) }

   ```



3. **Expected Performance**:

   - **CPU**: ~0.3-0.5s per query-document pair

   - **GPU (Metal/CUDA)**: ~0.05-0.1s per query-document pair

   - Performance is consistent regardless of document length due to truncation



4. **Chunking Strategy** for long documents:

   ```ruby

   def rerank_long_document(query, long_text, chunk_size: 300)

     # Split into overlapping chunks

     words = long_text.split

     chunks = []



     (0...words.length).step(chunk_size - 50) do |i|

       chunk = words[i...(i + chunk_size)].join(" ")

       chunks << chunk

     end



     # Rerank chunks

     results = reranker.rerank(query, chunks)



     # Return best chunk

     results.max_by { |r| r[:score] }

   end

   ```



5. **Memory Usage**:

   - Model size: ~125MB

   - Each batch processes all documents simultaneously

   - Consider batching if you have many documents



## Tokenizer



Red-Candle provides direct access to tokenizers for text preprocessing and analysis. This is useful for understanding how models process text, debugging issues, and building custom NLP pipelines.



### Basic Usage



```ruby

require 'candle'



# Load a tokenizer from HuggingFace

tokenizer = Candle::Tokenizer.from_pretrained("bert-base-uncased")



# Encode text to token IDs

token_ids = tokenizer.encode("Hello, world!")

# => [101, 7592, 1010, 2088, 999, 102]



# Decode token IDs back to text

text = tokenizer.decode(token_ids)

# => "hello, world!"



# Get token strings (subwords) - useful for visualization

tokens = tokenizer.encode_to_tokens("Hello, world!")

# => ["[CLS]", "hello", ",", "world", "!", "[SEP]"]



# Get both IDs and tokens together

result = tokenizer.encode_with_tokens("preprocessing")

# => {"ids" => [101, 3653, 22618, 2527, 102],

#     "tokens" => ["[CLS]", "prep", "##ro", "##ces", "##sing", "[SEP]"]}

```



### Batch Processing



```ruby

# Encode multiple texts at once

texts = ["Hello world", "How are you?", "Tokenizers are cool"]

batch_ids = tokenizer.encode_batch(texts)



# Get token strings for multiple texts

batch_tokens = tokenizer.encode_batch_to_tokens(texts)

```



### Vocabulary Access



```ruby

# Get vocabulary size

vocab_size = tokenizer.vocab_size

# => 30522



# Get full vocabulary as a hash

vocab = tokenizer.get_vocab

# vocab["hello"] => 7592



# Convert a specific token ID to its string

token_str = tokenizer.id_to_token(7592)

# => "hello"



# Get special tokens

special = tokenizer.get_special_tokens

# => {"cls_token" => 101, "sep_token" => 102, "pad_token" => 0, ...}

```



### Configuration



```ruby

# Create a tokenizer with padding enabled

padded_tokenizer = tokenizer.with_padding(length: 128)



# Create a tokenizer with truncation

truncated_tokenizer = tokenizer.with_truncation(512)



# Configure padding with more options

padded_tokenizer = tokenizer.with_padding(

  length: 128,          # Fixed length padding

  direction: "right",   # Pad on the right (default)

  pad_token: "[PAD]"    # Padding token

)

```



### Model Integration



All models expose their tokenizers:



```ruby

# From LLM

llm = Candle::LLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")

llm_tokenizer = llm.tokenizer



# From EmbeddingModel

embedding_model = Candle::EmbeddingModel.from_pretrained

emb_tokenizer = embedding_model.tokenizer



# From Reranker

reranker = Candle::Reranker.from_pretrained("cross-encoder/ms-marco-MiniLM-L-12-v2")

rank_tokenizer = reranker.tokenizer

```



### Understanding Subword Tokenization



Modern tokenizers split unknown or rare words into subword pieces:



```ruby

# See how words are split into subwords

result = tokenizer.encode_with_tokens("unbelievable")

# => {"ids" => [101, 4895, 6499, 102],

#     "tokens" => ["[CLS]", "un", "##believable", "[SEP]"]}



# The ## prefix indicates a continuation of the previous token

complex = tokenizer.encode_to_tokens("preprocessing tokenization")

# => ["[CLS]", "prep", "##ro", "##ces", "##sing", "token", "##ization", "[SEP]"]

```



### Use Cases



- **Token Analysis**: Understand how your text is being processed by models

- **Debugging**: See why certain inputs might cause unexpected model behavior

- **Custom Preprocessing**: Build your own text processing pipelines

- **Educational**: Teach how modern NLP models handle text

- **NER Preparation**: Get aligned tokens for named entity recognition tasks



## Named Entity Recognition (NER)



Red-Candle includes comprehensive Named Entity Recognition capabilities for extracting entities like people, organizations, locations, and custom entity types from text.



### Model-based NER



Load pre-trained NER models from HuggingFace:



```ruby

require 'candle'



# Load a pre-trained NER model

ner = Candle::NER.from_pretrained("Babelscape/wikineural-multilingual-ner")



# Or load a model with a specific tokenizer (for models without tokenizer.json)

ner = Candle::NER.from_pretrained("dslim/bert-base-NER", tokenizer: "bert-base-cased")



# Extract entities from text

text = "Apple Inc. was founded by Steve Jobs and Steve Wozniak in Cupertino, California."

entities = ner.extract_entities(text)



entities.each do |entity|

  puts "#{entity[:text]} (#{entity[:label]}) - confidence: #{entity[:confidence].round(2)}"

end

# Output:

# Apple Inc. (ORG) - confidence: 0.99

# Steve Jobs (PER) - confidence: 0.99

# Steve Wozniak (PER) - confidence: 0.98

# Cupertino (LOC) - confidence: 0.97

# California (LOC) - confidence: 0.98



# Adjust confidence threshold (default: 0.9)

entities = ner.extract_entities(text, confidence_threshold: 0.95)



# Get token-level predictions for detailed analysis

tokens = ner.predict_tokens(text)

```



### Pattern-based Recognition



For domain-specific entities, use regex patterns:



```ruby

# Create pattern-based recognizers

email_recognizer = Candle::PatternEntityRecognizer.new("EMAIL", [

  /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b/

])



phone_recognizer = Candle::PatternEntityRecognizer.new("PHONE", [

  /\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/,         # 555-123-4567

  /\b\(\d{3}\)\s*\d{3}[-.]?\d{4}\b/,      # (555) 123-4567

  /\b\+1\s*\d{3}[-.]?\d{3}[-.]?\d{4}\b/   # +1 555-123-4567

])



# Extract entities

text = "Contact us at info@example.com or call 555-123-4567"

email_entities = email_recognizer.recognize(text)

phone_entities = phone_recognizer.recognize(text)

```



### Gazetteer-based Recognition



Use dictionaries for known entities:



```ruby

# Create gazetteer recognizers

companies = ["Apple", "Google", "Microsoft", "Amazon", "Tesla"]

company_recognizer = Candle::GazetteerEntityRecognizer.new("COMPANY", companies)



# Load from file

drug_recognizer = Candle::GazetteerEntityRecognizer.new("DRUG")

drug_recognizer.load_from_file("drug_names.txt")



# Case-sensitive matching

product_recognizer = Candle::GazetteerEntityRecognizer.new("PRODUCT",

  ["iPhone", "iPad", "MacBook"],

  case_sensitive: true

)

```



### Hybrid NER



Combine ML models with rule-based approaches for best results:



```ruby

# Create hybrid NER system

hybrid = Candle::HybridNER.new("Babelscape/wikineural-multilingual-ner")



# Add pattern recognizers

hybrid.add_pattern_recognizer("EMAIL", [/\b[\w._%+-]+@[\w.-]+\.[A-Z|a-z]{2,}\b/])

hybrid.add_pattern_recognizer("PHONE", [/\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/])



# Add gazetteer recognizers

hybrid.add_gazetteer_recognizer("COMPANY", ["Apple", "Google", "Microsoft"])

hybrid.add_gazetteer_recognizer("PRODUCT", ["iPhone", "Android", "Windows"])



# Extract all entities

text = "John Smith (john@apple.com) from Apple called about the new iPhone. Reach him at 555-0123."

entities = hybrid.extract_entities(text)



# Results include entities from all recognizers

# Overlapping entities are automatically resolved (highest confidence wins)

```



### Custom Entity Types



Perfect for specialized domains:



```ruby

# Biomedical entities

gene_patterns = [

  /\b[A-Z][A-Z0-9]{2,10}\b/,    # TP53, BRCA1, EGFR (bounded for safety)

  /\bCD\d+\b/,                  # CD4, CD8, CD34

  /\b[A-Z]+\d[A-Z]\d*\b/        # RAD51C, PALB2

]

gene_recognizer = Candle::PatternEntityRecognizer.new("GENE", gene_patterns)



# Financial entities

ticker_patterns = [

  /\$[A-Z]{1,5}\b/,             # $AAPL, $GOOGL

  /\b[A-Z]{1,5}\.NYSE\b/,       # AAPL.NYSE

  /\b[A-Z]{1,5}\.NASDAQ\b/      # GOOGL.NASDAQ

]

ticker_recognizer = Candle::PatternEntityRecognizer.new("TICKER", ticker_patterns)



# Legal entities

case_patterns = [

  /\b\d+\s+F\.\d+\s+\d+\b/,     # 123 F.3d 456

  /\b\d+\s+U\.S\.\s+\d+\b/,     # 123 U.S. 456

  /\bNo\.\s+\d+-\d+\b/          # No. 20-1234

]

case_recognizer = Candle::PatternEntityRecognizer.new("CASE", case_patterns)

```



### Available Pre-trained Models



Popular NER models on HuggingFace:



```ruby

# General multilingual NER (4 entity types: PER, ORG, LOC, MISC)

ner = Candle::NER.from_pretrained("Babelscape/wikineural-multilingual-ner")



# English NER (requires separate tokenizer)

ner = Candle::NER.from_pretrained("dslim/bert-base-NER", tokenizer: "bert-base-cased")



# Multilingual NER

ner = Candle::NER.from_pretrained("Davlan/bert-base-multilingual-cased-ner-hrl")



# OntoNotes 5 (18 entity types including DATE, TIME, MONEY, etc.)

ner = Candle::NER.from_pretrained("flair/ner-english-ontonotes-large")



# Biomedical NER

ner = Candle::NER.from_pretrained("dmis-lab/biobert-base-cased-v1.2")

ner = Candle::NER.from_pretrained("allenai/scibert_scivocab_uncased")

```



### Performance Tips



1. **Device Selection**: Use GPU for faster inference

   ```ruby

   ner = Candle::NER.from_pretrained("Babelscape/wikineural-multilingual-ner", device: Candle::Device.metal)

   ```



2. **Batch Processing**: Process multiple texts together when possible



3. **Confidence Threshold**: Balance precision/recall with appropriate thresholds



4. **Entity Resolution**: The hybrid NER automatically handles overlapping entities



### Output Format



All NER methods return entities in a consistent format:



```ruby

{

  "text" => "Apple Inc.",          # The entity text

  "label" => "ORG",               # Entity type

  "start" => 0,                   # Character start position

  "end" => 10,                    # Character end position

  "confidence" => 0.99,           # Confidence score (0-1)

  "token_start" => 0,             # Token start index (model-based only)

  "token_end" => 2,               # Token end index (model-based only)

  "source" => "model"             # Source: "model", "pattern", or "gazetteer"

}

```



## Vision-Language Models (VLM)



Red-Candle supports vision-language models for understanding and describing images. The VLM module uses LLaVA (Large Language and Vision Assistant), which combines a CLIP vision encoder with a Llama language model.



### Basic Usage



```ruby

require 'candle'



# Load a LLaVA model (requires ~13GB download on first use)

vlm = Candle::VLM.from_pretrained("llava-hf/llava-v1.6-vicuna-7b-hf")



# Describe an image

description = vlm.describe("photo.jpg")



# Ask a question about an image

answer = vlm.ask("photo.jpg", "What animal is in this image?")

# => "The animal in the image is a cat."



# Control output length

vlm.describe("photo.jpg", max_length: 500)

vlm.ask("photo.jpg", "What colors do you see?", max_length: 50)

```



### How It Works



1. **CLIP Vision Encoder**: Converts the image into a sequence of visual feature tokens (576 patches from a 336x336 image)

2. **MM Projector**: Projects vision features into the language model's embedding space

3. **Llama LLM**: Processes the combined image+text embeddings and generates a text response



### Supported Models



| Model | LLM Backend | Size | Notes |

|:------|:-----------|:-----|:------|

| `llava-hf/llava-v1.6-vicuna-7b-hf` | Llama (Vicuna) | 13GB | Recommended, LLaVA-Next with Llama backend |



### Notes



- First load downloads ~13GB of model weights (cached for subsequent use)

- Image preprocessing is automatic (resize, normalize to CLIP format)

- Generation uses greedy decoding

- Multiple calls work correctly (KV cache is reset between queries)



## Common Runtime Errors



### Weight is negative, too large or not a valid number



**Error:**

```

/Users/cpetersen/src/scientist/red-candle/lib/candle/llm.rb:25:in `_generate_stream': Generation failed: A weight is negative, too large or not a valid number (RuntimeError)

    from /Users/cpetersen/src/scientist/red-candle/lib/candle/llm.rb:25:in `generate_stream'

    ...

```



**Cause:** This error occurs when using overly aggressive quantization levels (particularly Q2_K) that result in numerical instability during inference. The 2-bit quantization can cause weights to become corrupted or produce NaN/Inf values.



**Solution:** Use a higher quantization level. Recommended options:

- Q4_K_M (4-bit) - Best balance of quality and size

- Q5_K_M (5-bit) - Higher quality with slightly larger size

- Q3_K_M (3-bit) - Minimum recommended quantization



```ruby

llm = Candle::LLM.from_pretrained("TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF",

                                  device: device,

                                  gguf_file: "tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf")

```



### Cannot find tensor model.embed_tokens.weight



**Error:**

```

Failed to load quantized model: cannot find tensor model.embed_tokens.weight (RuntimeError)

```



**Cause:** This error was common in earlier versions when loading GGUF files with incompatible tensor naming conventions. The unified GGUF loader in version 1.0.0+ should handle most GGUF files correctly.



**If you still encounter this error:**

1. Ensure you're using the latest version of red-candle (1.0.0 or higher)

2. Make sure to specify the exact GGUF filename:

   ```ruby

   llm = Candle::LLM.from_pretrained("TheBloke/Mistral-7B-Instruct-v0.2-GGUF",

                                     device: device,

                                     gguf_file: "mistral-7b-instruct-v0.2.Q4_K_M.gguf")

   ```

3. If the error persists, the GGUF file may use an unsupported architecture or format



### No GGUF file found in repository



**Error:**

```

Failed to load quantized model: No GGUF file found in repository TheBloke/model-name-GGUF. Try specifying a quantization level like Q4_K_M, Q5_K_M, or Q8_0. (RuntimeError)

```



**Cause:** The automatic GGUF file detection couldn't find a matching file, often due to naming variations.



**Solution:** Specify the exact GGUF filename:

```ruby

# Visit the HuggingFace repository to find the exact filename

llm = Candle::LLM.from_pretrained("TheBloke/Llama-2-7B-Chat-GGUF",

                                  device: device,

                                  gguf_file: "llama-2-7b-chat.Q4_K_M.gguf")

```



### Failed to download tokenizer



**Error:**

```

Failed to load quantized model: Failed to download tokenizer: request error: HTTP status client error (404 Not Found)

```



**Cause:** GGUF repositories often don't include separate tokenizer files since they're embedded in the GGUF format.



**Solution:** The code now includes fallback tokenizer loading. If you still encounter this error, ensure you're using the latest version of red-candle.



### Missing metadata in GGUF file



**Error:**

```

Failed to load GGUF model: cannot find gemma3.attention.head_count in metadata (RuntimeError)

```

or

```

Failed to load GGUF model: cannot find llama.attention.head_count in metadata (RuntimeError)

```



**Cause:** Some GGUF files may have been created with older conversion tools that don't include all required metadata fields.



**Solution:**

- Try a different GGUF file from the same model

- Look for GGUF files from TheBloke or other reputable sources

- Check if a newer version of the GGUF file is available

- Some Gemma GGUF files may not be compatible with the current loader



**Known compatibility issues:**

- `lmstudio-ai/gemma-2b-it-GGUF` - Missing required metadata fields

- Gemma 3 GGUF files may require specific tokenizers that are not publicly available

- For best compatibility, use Llama or Mistral GGUF files from TheBloke



## Development



FORK IT!



```

git clone https://github.com/scientist-labs/red-candle

cd red-candle

bundle

bundle exec rake compile

```



Pull requests are welcome.



## Testing



Red Candle has comprehensive tests at both the Ruby and Rust levels:



### Ruby Tests

```bash

# Run all Ruby tests

bundle exec rake test



# Run specific test suites

bundle exec rake test:device         # Device compatibility tests

bundle exec rake test:benchmark      # Benchmark tests

bundle exec rake test:llm:mistral    # Model-specific tests

```



### Rust Tests

```bash

# Run Rust unit and integration tests

cd ext/candle && cargo test



# Or use the Rake task

bundle exec rake rust:test

```



The Rust tests include:

- Unit tests within source files (using `#[cfg(test)]` modules)

- Integration tests for external dependencies (candle_core operations)

- Tests for structured generation, tokenization, and text generation



### Code Coverage



#### Rust Code Coverage

Red Candle uses `cargo-llvm-cov` for Rust code coverage analysis:



```bash

# Generate HTML coverage report (opens in target/llvm-cov/html/index.html)

bundle exec rake rust:coverage:html



# Show coverage summary in terminal

bundle exec rake rust:coverage:summary



# Generate detailed coverage report

bundle exec rake rust:coverage:report



# Generate LCOV format for CI integration

bundle exec rake rust:coverage:lcov



# Clean coverage data

bundle exec rake rust:coverage:clean

```



**Note**: Overall Rust coverage shows ~17% because most code consists of Ruby FFI bindings that are tested through Ruby tests. The testable Rust components have high coverage:

- Constrained generation: 99.59%

- Schema processing: 90.99%

- Integration tests: 97.12%



#### Ruby Code Coverage

Ruby test coverage is generated automatically when running tests:

```bash

bundle exec rake test

# Coverage report generated in coverage/index.html

```



## Release



1. Update version number in `lib/candle/version.rb` and commit.

2. `bundle exec rake build`

3. `git tag VERSION_NUMBER`

4. `git push --follow-tags`

5. `gem push pkg/red-candle-VERSION_NUMBER.gem`



## Dependencies



- [Candle](https://github.com/huggingface/candle)

- [Magnus](https://github.com/matsadler/magnus)

- [Outlines-core](https://github.com/dottxt-ai/outlines-core)