{"id":50399875,"url":"https://github.com/floneum/kalosm","last_synced_at":"2026-05-30T23:00:25.926Z","repository":{"id":176447404,"uuid":"644995806","full_name":"floneum/kalosm","owner":"floneum","description":"Instant, controllable, local pre-trained AI models in Rust","archived":false,"fork":false,"pushed_at":"2026-05-23T16:26:50.000Z","size":296870,"stargazers_count":2193,"open_issues_count":51,"forks_count":130,"subscribers_count":26,"default_branch":"main","last_synced_at":"2026-05-23T17:05:37.903Z","etag":null,"topics":["ai","candle","constrained-generation","dioxus","floneum-v3","kalosm","llama","llamacpp","llm","mistral","rust","transcription","whisper"],"latest_commit_sha":null,"homepage":"http://floneum.com/kalosm","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/floneum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"ealmloff"}},"created_at":"2023-05-24T17:35:57.000Z","updated_at":"2026-05-23T15:14:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"903bb1d8-ba26-4600-81df-b7bb7849ad97","html_url":"https://github.com/floneum/kalosm","commit_stats":{"total_commits":1176,"total_committers":10,"mean_commits":117.6,"dds":"0.13435374149659862","last_synced_commit":"ce31f7c10f385ad5a0a21c64ebad591fa71daacc"},"previous_names":["floneum/floneum","floneum/kalosm"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/floneum/kalosm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/floneum%2Fkalosm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/floneum%2Fkalosm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/floneum%2Fkalosm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/floneum%2Fkalosm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/floneum","download_url":"https://codeload.github.com/floneum/kalosm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/floneum%2Fkalosm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33424573,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-23T22:14:44.296Z","status":"online","status_checked_at":"2026-05-24T02:00:06.296Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","candle","constrained-generation","dioxus","floneum-v3","kalosm","llama","llamacpp","llm","mistral","rust","transcription","whisper"],"created_at":"2026-05-30T23:00:15.677Z","updated_at":"2026-05-30T23:00:25.917Z","avatar_url":"https://github.com/floneum.png","language":"Rust","funding_links":["https://github.com/sponsors/ealmloff"],"categories":["Rust"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eKalosm\u003c/h1\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003c!-- Crates version --\u003e\n  \u003ca href=\"https://crates.io/crates/kalosm\"\u003e\n    \u003cimg src=\"https://img.shields.io/crates/v/kalosm.svg?style=flat-square\"\n    alt=\"Crates.io version\" /\u003e\n  \u003c/a\u003e\n  \u003c!-- Downloads --\u003e\n  \u003ca href=\"https://crates.io/crates/kalosm\"\u003e\n    \u003cimg src=\"https://img.shields.io/crates/d/kalosm.svg?style=flat-square\"\n      alt=\"Download\" /\u003e\n  \u003c/a\u003e\n  \u003c!-- docs --\u003e\n  \u003ca href=\"https://docs.rs/kalosm\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/docs-latest-blue.svg?style=flat-square\"\n      alt=\"docs.rs docs\" /\u003e\n  \u003c/a\u003e\n  \u003c!-- Discord --\u003e\n  \u003ca href=\"https://discord.gg/dQdmhuB8q5\"\u003e\n    \u003cimg src=\"https://img.shields.io/discord/1120130300236800062?logo=discord\u0026style=flat-square\" alt=\"Discord Link\" /\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\nKalosm is an ecosystem of crates that make it easy to develop applications that use local or remote AI models. There are try main projects in this repo:\n\n- [Kalosm](./interfaces/kalosm): A simple interface for pre-trained models in rust\n- [Fusor](./fusor-ml/core): A runtime for quantized ML inference. Fusor uses WGPU to run models on any accelerator natively or in the browser\n\n## Kalosm\n\n[Kalosm](./interfaces/kalosm/) is a simple interface for pre-trained models in Rust. It makes it easy to interact with pre-trained, language, audio, and image models.\n\n### Model Support\n\nKalosm supports a variety of models. Here is a list of the models that are currently supported:\n\n| Model            | Modality | Size       | Description                            | Quantized | CUDA + Metal Accelerated | Example                                                                      |\n| ---------------- | -------- | ---------- | -------------------------------------- | --------- | ------------------------ | ---------------------------------------------------------------------------- |\n| Llama            | Text     | 1b-70b     | General purpose language model         | ✅        | ✅                       | [llama 3 chat](interfaces/kalosm/examples/chat.rs)                           |\n| Mistral          | Text     | 7-13b      | General purpose language model         | ✅        | ✅                       | [mistral chat](interfaces/kalosm/examples/chat-mistral-2.rs)                 |\n| Phi              | Text     | 2b-4b      | Small reasoning focused language model | ✅        | ✅                       | [phi 3 chat](interfaces/kalosm/examples/chat-phi-3.rs)                       |\n| Whisper          | Audio    | 20MB-1GB   | Audio transcription model              | ✅        | ✅                       | [live whisper transcription](interfaces/kalosm/examples/transcribe.rs)       |\n| RWuerstchen      | Image    | 5gb        | Image generation model                 | ❌        | ✅                       | [rwuerstchen image generation](interfaces/kalosm/examples/generate-image.rs) |\n| TrOcr            | Image    | 3gb        | Optical character recognition model    | ❌        | ✅                       | [Text Recognition](interfaces/kalosm/examples/ocr.rs)                        |\n| Segment Anything | Image    | 50MB-400MB | Image segmentation model               | ❌        | ❌                       | [Image Segmentation](interfaces/kalosm/examples/segment-image.rs)            |\n| Bert             | Text     | 100MB-1GB  | Text embedding model                   | ❌        | ✅                       | [Semantic Search](interfaces/kalosm/examples/semantic-search.rs)             |\n\n### Utilities\n\nKalosm also supports a variety of utilities around pre-trained models. These include:\n\n- [Extracting, formatting and retrieving context for LLMs](./interfaces/kalosm/examples/context_extraction.rs): [Extract context from txt/html/docx/md/pdf](./interfaces/kalosm/examples/context_extraction.rs) [chunk that context](./interfaces/kalosm/examples/chunking.rs) [then search for relevant context with vector database integrations](./interfaces/kalosm/examples/semantic-search.rs)\n- [Transcribing audio from your microphone or file](./interfaces/kalosm/examples/transcribe.rs)\n- [Crawling and scraping content from web pages](./interfaces/kalosm/examples/crawl.rs)\n\n### Structured Generation\n\nKalosm supports structured generation with arbitrary parsers. It uses a custom parser engine and sampler and structure-aware acceleration to make structure generation even faster than uncontrolled text generation. You can take any rust type and add `#[derive(Parse, Schema)]` to make it usable with structured generation:\n\n```rust\nuse kalosm::language::*;\n\n/// A fictional character\n#[derive(Parse, Schema, Clone, Debug)]\nstruct Character {\n    /// The name of the character\n    #[parse(pattern = \"[A-Z][a-z]{2,10} [A-Z][a-z]{2,10}\")]\n    name: String,\n    /// The age of the character\n    #[parse(range = 1..=100)]\n    age: u8,\n    /// A description of the character\n    #[parse(pattern = \"[A-Za-z ]{40,200}\")]\n    description: String,\n}\n\n#[tokio::main]\nasync fn main() {\n    // First create a model. Chat models tend to work best with structured generation\n    let model = Llama::phi_3().await.unwrap();\n    // Then create a task with the parser as constraints\n    let task = model.task(\"You generate realistic JSON placeholders for characters\")\n        .typed();\n    // Finally, run the task\n    let mut stream = task(\u0026\"Create a list of random characters\", \u0026model);\n    stream.to_std_out().await.unwrap();\n    let characters: [Character; 10] = stream.await.unwrap();\n    println!(\"{characters:?}\");\n}\n```\n\nhttps://github.com/user-attachments/assets/8900f57d-55c8-4d4a-a67b-73beab1e5155\n\nIn addition to regex, you can provide your own grammar to generate structured data. This lets you constrain the response to any structure you want including complex data structures like JSON, HTML, and XML.\n\n### Kalosm Quickstart!\n\nThis quickstart will get you up and running with a simple chatbot. Let's get started!\n\n\u003e A more complete guide for Kalosm is available on the [Kalosm website](https://floneum.com/kalosm/), and examples are available in the [examples folder](https://github.com/floneum/floneum/tree/main/interfaces/kalosm/examples).\n\n1. Install [rust](https://rustup.rs/)\n2. Create a new project:\n\n```sh\ncargo new kalosm-hello-world\ncd ./kalosm-hello-world\n```\n\n3. Add Kalosm as a dependency\n\n```sh\n# You can use `--features llama,metal`, `--features llama,cuda`, or `--features llama,mkl` if your machine supports an accelerator\ncargo add kalosm --features llama\ncargo add tokio --features full\n```\n\n4. Add this code to your `main.rs` file\n\n```rust, no_run\nuse kalosm::language::*;\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error\u003e\u003e {\n  let model = Llama::phi_3().await?;\n  let mut chat = model.chat()\n    .with_system_prompt(\"You are a pirate called Blackbeard\");\n\n  loop {\n    chat(\u0026prompt_input(\"\\n\u003e \")?)\n      .to_std_out()\n      .await?;\n  }\n}\n```\n\n5. Run your application with:\n\n```sh\ncargo run --release\n```\n\n[chat bot demo](https://github.com/floneum/floneum/assets/66571940/e4e76efb-6387-4fcd-aa3c-aa556e840334)\n\n## Fusor\n\n⚠️ Fusor is still early in development and is not ready for production use. Fusor will serve as the backend for Kalosm in the 0.5 release to enable web and AMD support\n\n[Fusor](./fusor-ml/core) is a WGPU runtime for quantized ML inference. Fusor works with the gguf file format to load quantized models. It targets uses WebGpu to target many different accelerators including Nvidia GPUs, AMD GPUs, and Metal. Most ML frameworks contain hand optimized kernels that perform a series of operations together. Fusor uses a kernel fusion compiler to make merge custom operation chains into an optimized kernel without dropping down to the shader code. This compiles to a single kernel:\n```rust, ignore\nfn exp_add_one(tensor: Tensor\u003c2, f32\u003e) -\u003e Tensor\u003c2, f32\u003e {\n  1. + (-tensor).exp()\n}\n```\n\n## Community\n\nIf you are interested in either project, you can join the [discord](https://discord.gg/dQdmhuB8q5) to discuss the project and get help.\n\n## Contributing\n\n- Report issues on our [issue tracker](https://github.com/floneum/floneum/issues).\n- Help other users in the discord\n- If you are interested in contributing, feel free to reach out on discord\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffloneum%2Fkalosm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffloneum%2Fkalosm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffloneum%2Fkalosm/lists"}