Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/r-uby-dev/llm

Ruby's capable AI runtime
https://github.com/r-uby-dev/llm

a2a a2a-client agent agent2agent agents ai ai-runtime llm llm-agents llm-framework llm-frameworks llms mcp mcp-client rag ruby ruby-lib ruby-library

Last synced: about 7 hours ago
JSON representation

Ruby's capable AI runtime

Awesome Lists containing this project

README 

          


  

    a r.uby.dev project

  




> A [r.uby.dev](https://r.uby.dev/llm) project.



Welcome to the canonical llm.rb repository.



llm.rb is an advanced runtime for building capable AI applications

on CRuby. By default it has zero runtime dependencies although certain

functionality – such as ActiveRecord support – require

optional dependencies that are opt-in.



## Features



The runtime supports OpenAI, OpenAI-compatible endpoints, Anthropic, Google

Gemini, Mistral, DeepSeek, DeepInfra, xAI, Z.ai, AWS Bedrock, Ollama, and llama.cpp.

It has first-class support for streaming, tool calls,  MCP

and A2A, embeddings, vector stores and the RAG pattern.



There are multiple HTTP backends to choose from, tools can be run concurrently

or in parallel via threads, async tasks, fibers, ractors, and fork, and it is

also possible to make a tool call while the model is still streaming.



The runtime builds on top of three core concepts: providers, contexts, and agents,

so once you learn the fundamentals, everything else falls into place naturally. And once

you learn llm.rb, you will also be able to use mruby-llm and

wasm-llm because the API is pretty much identical.



## Install



```bash

gem install llm.rb

```



## Quick start



#### LLM::Agent



The [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html) class is the default high-level interface,

and it is recommended for most use-cases. It manages tool execution

automatically, guards against infinite loops, manages conversation

state, and much more.



```ruby

require "llm"



llm = LLM.deepseek(key: ENV["KEY"])

agent = LLM::Agent.new(llm, stream: $stdout)

agent.talk "Hello world"

```



#### LLM::Context



The [`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html) class is at the heart of the runtime

and it is what [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html) uses under the hood.

It requires that the tool call loop be managed manually -

sometimes that can be useful, but usually for advanced use-cases.

If you're new to llm.rb, try [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html) first.



```ruby

require "llm"



llm = LLM.deepseek(key: ENV["KEY"])

ctx = LLM::Context.new(llm, stream: $stdout)

ctx.talk "Hello world"

```



#### LLM::Tool



Subclasses of [`LLM::Tool`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool.html) are plain Ruby classes with

an optional set of typed parameters. 
 The model can choose to

call them on your behalf, and they're one of the most powerful features

for extending the feature set or abilities of a model.



```ruby

class ReadFile < LLM::Tool

  name "read-file"

  description "Read a file"

  parameter :path, String, "The filename or path"

  required %i[path]



  def call(path:)

    {contents: File.read(path)}

  end

end

```



#### LLM::Stream



Streams can be simple IO objects or subclasses of

[`LLM::Stream`](https://r.uby.dev/api-docs/llm.rb/LLM/Stream.html) with structured callbacks for content,

reasoning, tool calls, tool returns, and compaction.



```ruby

class MyStream < LLM::Stream

  def on_content(content)

    print content

  end



  def on_reasoning_content(content)

    warn content

  end

end



llm = LLM.deepseek(key: ENV["KEY"])

agent = LLM::Agent.new(llm, stream: MyStream.new)

agent.talk "Explain Ruby fibers."

```



#### LLM::REPL



The [LLM::Agent#repl](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#repl-instance_method)

method allows an agent to spawn a read-eval-print loop

that can be useful while developing or operating agents.

It can be used to debug tool calls, confirm an

agent has done what was expected, or improve an agent by

asking questions about what it has done up to that point.



This feature requires that the [curses](https://github.com/ruby/curses)

and [kramdown](https://github.com/gettalong/kramdown) libraries are

installed and available to require.



The TUI displays a status line with a context-usage bar and cost

counter, a scrollable transcript with markdown rendering, and a

multi-line input area. The UI stays responsive while the model

is generating a response.



##### REPL: Agent



A REPL session is started by calling `repl` on any agent

instance. The session inherits the agent's model, tools,

skills, and instructions.



```ruby

require "llm"



llm = LLM.deepseek(key: ENV["KEY"])

agent = LLM::Agent.new(llm)

agent.repl

```



##### REPL: State



The `path:` option accepts a file path where runtime state

is read from and written to. This lets you resume a

conversation across REPL sessions.



```ruby

require "llm"



llm = LLM.deepseek(key: ENV["KEY"])

agent = LLM::Agent.new(llm)

agent.repl(path: "session.json")

```



##### REPL: Tools



The `tools` option lets you attach additional tools

for the duration of the session. This is in addition to

any tools that might already be associated with an agent.



A number of optional tools are distributed as part of

llm.rb. They power the agents that can be found in the

[agents/](agents/) directory.



```ruby

require "llm"



llm = LLM.deepseek(key: ENV["KEY"])

agent = LLM::Agent.new(llm)

agent.repl(tools: [Debugger])

```



The following example starts a read-eval-print loop

with all of the builtin tools available.



```ruby

require "llm"

require "llm/tools"



llm = LLM.deepseek(key: ENV["KEY"])

agent = LLM::Agent.new(llm)

agent.repl(tools: LLM::Tool.subclasses)

```



##### REPL: Skills



The `skills` option lets you load extra skill directories

without attaching them to an agent permanently.



```ruby

require "llm"



llm = LLM.deepseek(key: ENV["KEY"])

agent = LLM::Agent.new(llm)

agent.repl(skills: [__dir__])

```



##### REPL: Tracer



By default the tracer is disabled for the duration of the

session. Setting `tracer: true` configures the REPL to use

the tracer associated with an instance of

[`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html).



```ruby

require "llm"



llm    = LLM.deepseek(key: ENV["KEY"])

tracer = LLM.logger(llm, path: "agent.log")

agent  = LLM::Agent.new(llm, tracer:)

agent.repl(tracer: true, tools: [Debugger])

```



##### REPL: Commands



Commands are recognized by a `/` prefix and are backed by the

[`LLM::Repl::Command`](https://r.uby.dev/api-docs/llm.rb/LLM/Repl/Command.html)

class, which can be subclassed to add custom commands. Once you

create a subclass, it is automatically added to the repl. A command

can have zero or more parameters, and all parameters are presumed

to be a String (at least for now).



```ruby

require "llm"

require "llm/repl"



class Greeter < LLM::Command

  name "greet"

  description "Greets the given name"

  parameter :name, String, "The person's name"

  required %i[name]



  def call(name:)

    write("Welcome #{name}!\n")

  end

end

```



##### REPL: Input



The input area supports several keyboard shortcuts:



| Key | Action |

|---|---|

| `Enter` | Submit the current prompt |

| `Ctrl+A` | Jump to the start of the line |

| `Ctrl+E` | Jump to the end of the line |

| `Ctrl+F` | Move the cursor forward |

| `Ctrl+K` | Erase from cursor to the end of the line |

| `Ctrl+Y` | Paste previously killed text |

| `Ctrl+D` | Delete the character at the cursor |

| `Left / Right` | Move the cursor |

| `Up / Down` | Scroll the transcript |

| `/exit` | Leave the REPL |



#### LLM::MCP



The Model Context Protocol (MCP) has first-class support

in llm.rb. The stdio and http transports work out of the

box. MCP tools are translated into subclasses of

[`LLM::Tool`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool.html) that can be used with [`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html)

or [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html).



```ruby

require "llm"



llm   = LLM.deepseek(key: ENV["KEY"])

mcp   = LLM::MCP.stdio(argv: ["ruby", "server.rb"])

agent = LLM::Agent.new(llm, stream: $stdout, tools: mcp.tools)

agent.talk "Run the tool"

```



#### LLM::A2A



The Agent 2 Agent (A2A) protocol has first-class support

in llm.rb. The http and jsonrpc transports work out of the

box. A2A skills are translated into subclasses of

[`LLM::Tool`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool.html) that can be used with [`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html)

or [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html).



```ruby

require "llm"



llm   = LLM.deepseek(key: ENV["KEY"])

a2a   = LLM::A2A.rest(url: "https://remote-agent.example.com")

agent = LLM::Agent.new(llm, stream: $stdout, tools: a2a.skills)

agent.talk "Run the skill"

```



#### RAG



Most providers offer an embedding model that can be

used for semantic search, or similarity search. An

embedding model can generate embeddings that can then

be stored in a database that is optimized for storing

and querying vectors, such as SQLite's [sqlite-vec](https://github.com/asg017/sqlite-vec)

or PostgreSQL's [pg-vector](https://github.com/pgvector/pgvector).



llm.rb also includes support for OpenAI's vector store API. It

provides a vector database as a HTTP service but we won't cover

that here.



```ruby

require "llm"



llm  = LLM.openai(key: ENV["KEY"])

body = "llm.rb is Ruby's capable AI runtime."

embedding = llm.embed([body]).embeddings.first



Document.create!(

  title: "llm.rb",

  body:,

  embedding:,

)

```



#### Concurrency



The runtime supports five different concurrency strategies that have

different attributes. The choice between all of them often depends

on the requirements of your application.



IO-bound tools are a good fit for the `:task`, `:thread`,

and `:fiber` strategies while true parallelism can be achieved

with the `:fork` and `:ractor` strategies. The

`:fork` strategy also provides a separate process that offers

isolation from its parent.



```ruby

require "llm"



llm   = LLM.deepseek(key: ENV["KEY"])

tools = [FetchNews, FetchStocks, FetchFeeds]

agent = LLM::Agent.new(llm, tools:, concurrency: :fork)

agent.talk "Run the tools in parallel"

```



#### ORM



Because both [`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html), and [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html)

can be serialized to JSON and stored in a simple string, both ActiveRecord

and Sequel support can be implemented within a single column on a single row.



The runtime includes first-class support for both ActiveRecord *and* Sequel, and

for both Rack-based applications *and* Rails-based applications. On databases

where it is supported, such as PostgreSQL, the column can be optimized by using

the `jsonb` type.



```ruby

require "active_record"

require "llm"

require "llm/active_record"



class Agent < ApplicationRecord

  acts_as_agent do |agent|

    agent.model "deepseek-v4-pro"

    agent.instructions "solve the user's query"

    agent.tools [Research, FinalizeResearch, ActOnResearch]

  end



  private



  # By convention, this method defines the provider for a model.

  # If necessary, it can be renamed with: provider: :your_method.

  def set_provider

    LLM.deepseek(key: ENV["KEY"])

  end



  # By convention, this method returns the context options given

  # to LLM::Context or LLM::Agent.

  def set_context

    {}

  end

end



agent = Agent.create!

agent.talk "perform research"

```



## FAQ



What providers does llm.rb support?







**Cloud**



The following cloud-based providers are available to choose from. 


In no particular order:



🇺🇸 OpenAI 


🇺🇸 DeepInfra 


🇺🇸 xAI 


🇺🇸 Google (Gemini) 


🇺🇸 AWS bedrock 


🇺🇸 Anthropic 


🇨🇳 DeepSeek 


🇨🇳 zAI 


🇪🇺 Mistral 



**Weights**



The following providers provide access to open-weight models. 


In no particular order:



🇺🇸 DeepInfra 


🇺🇸 AWS bedrock 


🇨🇳 DeepSeek 


🇨🇳 zAI 


🇪🇺 Mistral 



**Local**



The following providers can be run locally on your own hardware. 


In no particular order:



* Ollama

* Llamacpp




I have a limited budget. What should I do?






There a few options. The first option is to host

your own model, and use the ollama or llamacpp

providers. This can be diffilcult though because

a capable model requires hardware that can

match it. If you have the ability to self-host,

this would be my first option.





The second option is DeepSeek. 


The deepseek-v4-flash model costs pennies to use. 


And llm.rb has been optimized for deepseek. For example,

DeepSeek does not have image generation capabilities

but on the llm.rb runtime it does (vector graphics only,

though).





The same is true for structured outputs. DeepSeek does

not support structured outputs in the same way as OpenAI or

Google, but the llm.rb runtime makes it appear as

though it does, through the `json_object` response

type.



If you're on a budget, DeepSeek is hard to beat.



Can I download llm.rb via a decentralized network?




You can!




We are on the radicle.network




Every commit that lands on GitHub also lands on Radicle.




Our repository ID is z2PtfQ6dYwyYaW2aGrztG1sMyDmCE.




Browse on the web.



## Resources



If you like what you read so far, check out the [deepdive.md](https://r.uby.dev/llm/deepdive/)

to learn more. Unfortunately it

wasn't possible to cover every feature without the README becoming a small book.

The [r.uby.dev](https://r.uby.dev) homepage also includes more learning material

and resources.



## License



[Business Source License 1.1](./LICENSE)




Commercial production use requires a commercial license.




Each version converts to the [BSD Zero Clause](https://choosealicense.com/licenses/0bsd/)

four years after its first public release.




Contact [robert@r.uby.dev](mailto:robert@r.uby.dev) for a commercial license.



### Waivers



Waivers are automatically granted for: 



  * Personal use

  * Students

  * Teachers

  * Evaluation, development, and testing

  * Non-profits and charities

  * Companies with less than or equal to 50 employees