https://github.com/guywaldman/orch
Rust framework for LLM orchestration
https://github.com/guywaldman/orch
ai llm rust
Last synced: about 1 year ago
JSON representation
Rust framework for LLM orchestration
- Host: GitHub
- URL: https://github.com/guywaldman/orch
- Owner: guywaldman
- License: mit
- Created: 2023-05-29T22:37:55.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-07-27T21:01:25.000Z (almost 2 years ago)
- Last Synced: 2025-03-28T19:08:23.865Z (about 1 year ago)
- Topics: ai, llm, rust
- Language: Rust
- Homepage: https://crates.io/crates/orch
- Size: 184 KB
- Stars: 202
- Watchers: 3
- Forks: 4
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE.md
Awesome Lists containing this project
README
# orch
`orch` is a library for building language model powered applications and agents for the Rust programming language.
It was primarily built for usage in [magic-cli](https://github.com/guywaldman/magic-cli), but can be used in other contexts as well.
> [!NOTE]
>
> If the project gains traction, this can be compiled as an addon to other languages such as Python or a standalone WebAssembly module.
# Installation
```shell
cargo add orch
cargo add orch_response
```
Alternatively, add `orch` as a dependency to your `Cargo.toml` file:
```toml
[dependencies]
orch = "*" # Substitute with the latest version
orch_response = "*" # Substitute with the latest version
```
# Basic Usage
## Simple Text Generation
```rust no_run
use orch::execution::*;
use orch::lm::*;
#[tokio::main]
async fn main() {
let lm = OllamaBuilder::new().try_build().unwrap();
let executor = TextExecutorBuilder::new().with_lm(&lm).try_build().unwrap();
let response = executor.execute("What is 2+2?").await.expect("Execution failed");
println!("{}", response.content);
}
```
## Streaming Text Generation
```rust no_run
use orch::execution::*;
use orch::lm::*;
use tokio_stream::StreamExt;
#[tokio::main]
async fn main() {
let lm = OllamaBuilder::new().try_build().unwrap();
let executor = TextExecutorBuilder::new().with_lm(&lm).try_build().unwrap();
let mut response = executor.execute_stream("What is 2+2?").await.expect("Execution failed");
while let Some(chunk) = response.stream.next().await {
match chunk {
Ok(chunk) => print!("{chunk}"),
Err(e) => {
println!("Error: {e}");
break;
}
}
}
println!();
}
```
## Structured Data Generation
```rust no_run
use orch::execution::*;
use orch::lm::*;
use orch::response::*;
#[derive(Variants, serde::Deserialize)]
pub enum ResponseVariants {
Answer(AnswerResponseVariant),
Fail(FailResponseVariant),
}
#[derive(Variant, serde::Deserialize)]
#[variant(
variant = "Answer",
scenario = "You know the capital city of the country",
description = "Capital city of the country"
)]
pub struct AnswerResponseVariant {
#[schema(
description = "Capital city of the received country",
example = "London"
)]
pub capital: String,
}
#[derive(Variant, serde::Deserialize)]
#[variant(
variant = "Fail",
scenario = "You don't know the capital city of the country",
description = "Reason why the capital city is not known"
)]
pub struct FailResponseVariant {
#[schema(
description = "Reason why the capital city is not known",
example = "Country 'foobar' does not exist"
)]
pub reason: String,
}
#[tokio::main]
async fn main() {
let lm = OllamaBuilder::new().try_build().unwrap();
let executor = StructuredExecutorBuilder::new()
.with_lm(&lm)
.with_preamble("You are a geography expert who helps users understand the capital city of countries around the world.")
.with_options(Box::new(variants!(ResponseVariants)))
.try_build()
.unwrap();
let response = executor
.execute("What is the capital of Fooland?")
.await
.expect("Execution failed");
println!("Response:");
match response.content {
ResponseVariants::Answer(answer) => {
println!("Capital city: {}", answer.capital);
}
ResponseVariants::Fail(fail) => {
println!("Model failed to generate a response: {}", fail.reason);
}
}
}
```
## Embedding Generation
```rust no_run
use orch::execution::*;
use orch::lm::*;
#[tokio::main]
async fn main() {
let lm = OllamaBuilder::new().try_build().unwrap();
let executor = TextExecutorBuilder::new()
.with_lm(&lm)
.try_build()
.unwrap();
let embedding = executor
.generate_embedding("Phrase to generate an embedding for")
.await
.expect("Execution failed");
println!("Embedding:");
println!("{:?}", embedding);
}
```
## More Examples
See the [examples](https://github.com/guywaldman/orch/tree/main/core/examples) directory for usage examples.