https://github.com/barek2k2/local_llm
Lightweight Ruby gem for interacting with locally running Ollama LLMs with streaming, chat, and full offline privacy.
https://github.com/barek2k2/local_llm
ai artificial-intelligence chatbot data-privacy-compliance data-security llm local-ai local-ai-development local-ai-llm machine-learning offline-ai privacy private-chat ruby ruby-on-rails security
Last synced: about 2 months ago
JSON representation
Lightweight Ruby gem for interacting with locally running Ollama LLMs with streaming, chat, and full offline privacy.
- Host: GitHub
- URL: https://github.com/barek2k2/local_llm
- Owner: barek2k2
- License: mit
- Created: 2025-12-02T06:10:34.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2025-12-06T18:51:48.000Z (6 months ago)
- Last Synced: 2026-04-19T06:44:08.526Z (about 2 months ago)
- Topics: ai, artificial-intelligence, chatbot, data-privacy-compliance, data-security, llm, local-ai, local-ai-development, local-ai-llm, machine-learning, offline-ai, privacy, private-chat, ruby, ruby-on-rails, security
- Language: Ruby
- Homepage:
- Size: 8.85 MB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# Local Large Language Model(LLM) or Offline LLM for Ruby on Rails
**`local_llm`** is a lightweight Ruby gem that lets you talk to **locally installed LLMs via Ollama** — with **zero cloud dependency**, full **developer control**, and **configurable defaults**, including **real-time streaming support**. Instead of sending sensitive data to cloud APIs, this gem allows you to interact with powerful AI models directly from your local machine or server. It is built for privacy, control, and simplicity, making it ideal for developers who want fast AI features without internet dependency, usage limits, or data exposure. LocalLLM works seamlessly with both plain Ruby and Ruby on Rails applications.
It supports:
- Completely **OFFLINE!**
- Any Ollama model (LLaMA, Mistral, CodeLLaMA, Qwen, Phi, Gemma, etc.)
- Developer-configurable default models
- Developer-configurable Ollama API endpoint
- Developer-configurable **streaming or non-streaming**
- One-shot Q&A and multi-turn chat
- Works in plain Ruby & Rails
- Also safe for HIPAA, SOC2, and regulated workflows where data privacy is a big concern
### Privacy & Security
- **100% local inference**
- **No cloud calls**
- **No API keys**
- **No data leaves your machine**
---
## 🎬 Live Demo
Ruby on Rails Example at https://github.com/barek2k2/local_llm_demo
## 🚀 Features
- Use **any locally installed Ollama model**
- Change **default models at runtime**
- Enable or disable **real-time streaming**
- Works with:
- `llama2`
- `mistral`
- `codellama`
- `qwen`
- `phi`
- Anything supported by Ollama
- No API keys needed
- No cloud calls
- Full privacy
- Works completely offline
---
## 📦 Installation
### Install Ollama
Download from:
https://ollama.com
Then start it:
```bash
ollama serve
```
Or simply install it by running `brew install ollama` command in your mac machine.
### How to Install New LLMs
```
ollama pull llama2:13b
ollama pull mistral:7b-instruct
ollama pull codellama:13b-instruct
ollama pull qwen2:7b
```
### Verify Installed Models
```
ollama list
```
### Configuration
```
LocalLlm.configure do |c|
c.base_url = "http://localhost:11434"
c.default_general_model = "llama2:13b"
c.default_fast_model = "mistral:7b-instruct"
c.default_code_model = "codellama:13b-instruct"
c.default_stream = false # true = stream by default, false = return full text
end
```
### Basic Usage (Non-Streaming)
```
LocalLlm.ask("llama2:13b", "What is HIPAA?")
LocalLlm.ask("qwen2:7b", "Explain transformers in simple terms.")
LocalLlm.general("What is a Denial of Service attack?")
LocalLlm.fast("Summarize this paragraph in 3 bullet points.")
LocalLlm.code("Write a Ruby method that returns factorial of n.")
```
### Constant Alias (LocalLlm vs LocalLLM)
For convenience and readability, `LocalLLM` is provided as a direct alias of `LocalLlm`.
This means **both constants work identically**:
```
LocalLlm.fast("Tell me About Bangladesh")
LocalLLM.fast("Explain HIPAA in simple terms.") # alias of LocalLlm
```
### Streaming Usage (Live Output)
```
LocalLlm.configure do |c|
c.default_stream = true
end
LocalLlm.fast("Explain HIPAA in very simple words.") do |chunk|
print chunk
end
```
### Per-Call Streaming Override
```
LocalLlm.fast("Explain LLMs in one paragraph.", stream: true) do |chunk|
print chunk
end
full_text = LocalLlm.fast("Explain DoS attacks briefly.", stream: false)
puts full_text
```
### List Installed Ollama Models from Ruby
```
LocalLlm.models
```
### Switching to Qwen (or Any New Model)
```
ollama pull qwen2:7b
```
```
LocalLlm.ask("qwen2:7b", "Explain HIPAA in simple terms.")
```
### Full Chat API (Multi-Turn)
```
LocalLlm.chat("qwen2:7b", [
{ "role" => "system", "content" => "You are a helpful assistant." },
{"role" => "user", "content" => "Explain Ruby shortly in one sentence"},
])
```
`Ruby is a dynamic, open-source programming language known for its simplicity and readability, designed for building web applications with the Ruby on Rails framework.`
```
LocalLlm.chat("qwen2:7b", [
{ "role" => "system", "content" => "You are a helpful assistant." },
{"role" => "user", "content" => "Explain Ruby shortly in one sentence"},
{ "role" => "assistant", "content" => "Ruby is an open-source, dynamic, object-oriented programming language that emphasizes simplicity and readability, making it popular for web development with the Rails framework" },
{ "role" => "user", "content" => "Tell me the year in number when it was created?" }
])
```
`Ruby was created in the year 1995.`
```
LocalLlm.chat("qwen2:7b", [
{ "role" => "system", "content" => "You are a helpful assistant." },
{"role" => "user", "content" => "Explain Ruby shortly in one sentence"},
{ "role" => "assistant", "content" => "Ruby is an open-source, dynamic, object-oriented programming language that emphasizes simplicity and readability, making it popular for web development with the Rails framework" },
{ "role" => "user", "content" => "Tell me the year in number when it was created?" },
{ "role" => "assistant", "content" => "Ruby was created in the year 1995." },
{"role" => "user", "content" => "Thanks so much!"}
])
```
`You're welcome! If you have any other questions, feel free to ask. Happy coding!`
### Make Qwen the Default
```
LocalLlm.configure do |c|
c.default_general_model = "qwen2:7b"
end
LocalLlm.general("Explain transformers.")
```
### 🔌 Remote Ollama / Docker Support
```
LocalLlm.configure do |c|
c.base_url = "http://192.168.1.100:11434"
end
```
### Troubleshooting
`Connection refused - connect(2) for "localhost" port 11434 (Errno::ECONNREFUSED)`
##### This means ollama is not installed or not running in your machine. So run the following commands below
- `brew install ollama`
- `brew services start ollama`
After successfully starting this, it would run on port 11434 intom your machine. make sure it pulls installed LLM by running `ollama list`
### How to use it in Ruby On Rails
- In your Gemfile, add the gem as
`gem "local_llm", "~> 0.1.1"`
- Run `bundle install`
- Create an initializer at `config/initializers/local_llm.rb`
- Put the following code into it
```
LocalLlm.configure do |c|
# Default Ollama endpoint
c.base_url = "http://localhost:11434"
# Choose your default models (must be installed in Ollama)
c.default_general_model = "qwen2:7b"
c.default_fast_model = "mistral:7b-instruct"
c.default_code_model = "codellama:13b-instruct"
c.default_stream = true # stream support by default
end
```
- Then from your any controller or model, run this
```
question = "What is Ruby?"
LocalLlm.fast(question) do |chunk|
print chunk
end
```