{"id":50988372,"url":"https://github.com/maximecb/bebelm","last_synced_at":"2026-07-07T22:00:51.861Z","repository":{"id":362823857,"uuid":"1259398032","full_name":"maximecb/bebelm","owner":"maximecb","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-06T04:44:20.000Z","size":388,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-06T05:23:35.102Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maximecb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-04T13:19:28.000Z","updated_at":"2026-06-06T04:44:24.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/maximecb/bebelm","commit_stats":null,"previous_names":["maximecb/bebelm"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/maximecb/bebelm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maximecb%2Fbebelm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maximecb%2Fbebelm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maximecb%2Fbebelm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maximecb%2Fbebelm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maximecb","download_url":"https://codeload.github.com/maximecb/bebelm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maximecb%2Fbebelm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35243953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-19T23:00:32.390Z","updated_at":"2026-07-07T22:00:51.853Z","avatar_url":"https://github.com/maximecb.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# BebeLM\n\nPure-Rust, CPU-only implementation of [LFM2.5-8B-A1B Q4_K_M](https://www.liquid.ai/blog/lfm2-5-8b-a1b).\nThis model is very capable and has only 1B active parameters, making it possible for the\nmodel to run at interactive speeds without a GPU.\n\nThis package intentionally has very few dependencies and requires no extra system\npackages to compile, making it easy to build and run.\nThis is a library crate which can be imported into your Rust projects, and it's now available\nvia [crates.io](https://crates.io/crates/bebelm). There is also a basic command-line\ninterface that you can use.\n\nThe model needs about ~6-8GB of RAM to run (depending on context length).\nBebeLM was tested on an M5 CPU as well as Ryzen 7x and Threadripper CPUs. It should work\non Intel and on Raspberry Pi 4/5 as well, but this is untested.\n\n## Setup instructions\n\nInstall cargo or update your rust toolchain:\n```sh\n# Install Rust toolchain\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n\n# Update Rust toolchain\nrustup update\n```\n\nRunning also requires downloading the ~5.2 GB Q4_K_M model weights:\n\n```sh\ncurl -L -o LFM2.5-8B-A1B-Q4_K_M.gguf \\\n  \"https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-Q4_K_M.gguf\"\n```\n\nThe CLI reads the weights path from `BEBELM_WEIGHTS_FILE`, defaulting to\n`./LFM2.5-8B-A1B-Q4_K_M.gguf` (the current directory). Point it elsewhere with:\n\n```sh\nexport BEBELM_WEIGHTS_FILE=/path/to/LFM2.5-8B-A1B-Q4_K_M.gguf\n```\n\n### Installing via cargo\n\nInstall the CLI from crates.io — this puts a `bebelm` binary on your `PATH`:\n\n```sh\ncargo install bebelm\n```\n\n### Development setup\n\nClone the repo and build from source:\n\n```sh\ngit clone https://github.com/maximecb/bebelm\ncd bebelm\ncargo build --release\n```\n\n## Command-line interface\n\nBuild with `cargo build --release`, then run a subcommand on `./target/release/bebelm` (the\nexamples below use `cargo run --release --` for convenience). Every subcommand loads the\nweights from `BEBELM_WEIGHTS_FILE` (see above).\n\n- **`generate [options] \u003cprompt\u003e…`** — one-shot text completion of a prompt; streams tokens as\n  they are produced and reports prefill/decode throughput.\n- **`chat [options]`** — interactive multi-turn chat. Streams the model's full output, showing\n  the `\u003cthink\u003e...\u003c/think\u003e` reasoning and the final answer in different colors. The KV / conv\n  caches persist across turns, so each message only prefills its own new tokens. `Ctrl-D` or\n  `/exit` to quit.\n- **`ask [options] \u003cquestion\u003e…`** — one-shot single-turn chat. Encodes the question as a user\n  turn, streams the model's reply (including reasoning), and exits.\n\nAll commands take the same options (sampling defaults to the model's recommended settings):\n\n- `--greedy` — deterministic greedy decoding instead of sampling.\n- `--max-gen N` — cap tokens generated per turn (default 2048); counts every generated token,\n  reasoning included, so a long `\u003cthink\u003e` block eats into the budget left for the answer.\n- `--max-think N` — cap the `\u003cthink\u003e` reasoning block to N tokens (forces `\u003c/think\u003e`).\n- `--no-think` — disable reasoning (equivalent to `--max-think 0`).\n- `--hide-think` — generate reasoning but hide it from the output (streams only the answer).\n- `--num-threads N` — cap the rayon worker pool (default: one per available core).\n\n```sh\n# Interactive chat\ncargo run --release -- chat\n\n# One-shot single-turn chat\ncargo run --release -- ask \"How do I implement binary search in Rust?\"\n\n# One-shot completion\ncargo run --release -- generate --max-gen 64 \"The capital of France is\"\n```\n\n## Public crate API\n\n`bebelm` is a library first; the CLI is a thin wrapper over it. The high-level entry point is\n`bebelm::agent::Agent` — a conversation bound to a loaded model that owns the token transcript\nand the decode-time caches.\n\nLoad the model once, then back one or more agents with it:\n\n```rust\nuse bebelm::agent::Agent;\nuse bebelm::model::Model;\n\n// mmaps + validates the GGUF.\nlet model = Model::load(\"LFM2.5-8B-A1B-Q4_K_M.gguf\")?;\n\n// An agent borrows the model — the ~5.2 GB of weights are shared, so several agents are cheap.\nlet mut agent = Agent::new(\u0026model);\n\nagent.append_user(\"What is the capital of France?\");\nlet turn = agent.assistant_turn(|_, _| {});   // generate the whole reply at once\nprintln!(\"{}\", turn.text);\n\n// Keep chatting — the KV/conv caches persist, so only the new tokens are prefilled.\nagent.append_user(\"And of Italy?\");\nlet turn = agent.assistant_turn(|_, _| {});\nprintln!(\"{}\", turn.text);\n```\n\nHere `|_, _| {}` is a do-nothing token callback, so the whole reply is just collected into\n`turn.text`. To instead stream tokens as they are generated, pass a real callback — see\n**Generating** below.\n\n**Configuration** — builder methods chained after `Agent::new(..)` (sampling defaults to the\nmodel's recommended temperature 0.2 / top-k 80 / repeat-penalty 1.05):\n\n- `.greedy()` — deterministic argmax decoding.\n- `.temperature(f32)` / `.top_k(usize)` / `.repeat_penalty(f32)` — individual sampler knobs.\n- `.max_gen(usize)` — tokens generated per turn (default 2048); counts reasoning tokens too, so\n  the `\u003cthink\u003e` block and the answer share this budget.\n- `.max_context(usize)` — KV attention-window cap in tokens (default 32768); older context\n  slides out rather than stopping generation.\n- `.max_think(usize)` — cap the `\u003cthink\u003e` reasoning block (`0` ⇒ no reasoning block at all).\n\n**Building the prompt** — these only grow the transcript; nothing runs until you generate:\n\n- `append_user(\u0026str)` — wrap a ChatML user turn (`\u003c|im_start|\u003euser\\n…\u003c|im_end|\u003e\\n`).\n- `append(\u0026str)` — append raw text (BOS is added automatically on the first append).\n- `append_tokens(\u0026[u32])` — append already-tokenized ids (e.g. a tool result).\n\n**Generating** — `assistant_turn` and `generate` both return a `Turn` and take an `on_token`\ncallback:\n\n- `assistant_turn(on_token)` — open an assistant turn (ChatML), stream the reply, and close the\n  turn; pair it with `append_user` (as above).\n- `generate(on_token)` — the lower-level primitive: prefill pending tokens, then decode a raw\n  continuation (no ChatML framing) until EOS or `max_gen`; pair it with `append` for plain text\n  completion:\n\n```rust\nlet mut agent = Agent::new(\u0026model);\nagent.append(\"The capital of France is\");\nlet turn = agent.generate(|_, _| {});      // raw continuation; turn.text = \" the city of Paris…\"\nprintln!(\"The capital of France is{}\", turn.text);\n```\n\nThe returned `Turn`:\n\n```rust\npub struct Turn {\n    pub ids: Vec\u003cu32\u003e,    // generated ids (excludes the prompt and the terminating EOS)\n    pub text: String,     // the decoded reply\n    pub stats: GenStats,  // prompt_tokens, generated_tokens, prefill/decode Durations + *_tps()\n    pub stop: StopReason, // Eos, MaxNew, or ToolCall\n}\n```\n\nThe `on_token` callback is `impl FnMut(u32, \u0026str)`, called once per visible token as it is\ndecoded — its arguments are `(id, text)`:\n\n- `id: u32` — the token id; compare it against the `bebelm::tokenizer` constants below for\n  control-token logic (e.g. spotting `\u003cthink\u003e` / `\u003c/think\u003e` to colour the reasoning).\n- `text: \u0026str` — that same token decoded to a string, ready to print.\n\nThe terminating EOS is not passed to the callback, and the full reply is in `turn.text` either\nway. To stream tokens as they are produced:\n\n```rust\nuse bebelm::tokenizer;\n\nagent.append_user(\"Explain RoPE briefly.\");\nagent.assistant_turn(|id, text| {\n    if id == tokenizer::TOKEN_THINK_END {\n        println!();  // the \u003cthink\u003e reasoning block just ended\n    }\n    print!(\"{text}\");\n});\n```\n\n`agent.clear()` resets the conversation (keeping the weights); `agent.history()` returns the\nfull token transcript.\n\n**Prefilling** — `agent.prefill()` runs the model over the appended-but-unprocessed prompt to\nwarm the KV/conv caches *without* decoding. `generate` prefills lazily anyway, so this is purely\nan optimization for the fork-many pattern below: warm a shared prefix once, then `clone` it.\nPrefilling never changes what the model produces.\n\n**Cloning** — `Agent` implements `Clone`, so a shared prefix (e.g. a system prompt plus a few\nexample turns) can be built and prefilled once, then cheaply forked into several independent\ncontinuations — each clone keeps its own transcript and KV/conv caches, and generating on one\ndoesn't affect the others:\n\n```rust\nlet mut base = Agent::new(\u0026model).greedy();\nbase.append_system(\"You are a terse assistant. Answer in one word where possible.\");\nbase.prefill();   // warm the shared prefix into the caches once, without generating\n\nlet mut a = base.clone();\nlet mut b = base.clone();\na.append_user(\"What is the capital of France?\");\nb.append_user(\"What is the capital of Italy?\");\nprintln!(\"{}\", a.assistant_turn(|_, _| {}).text);\nprintln!(\"{}\", b.assistant_turn(|_, _| {}).text);\n```\n\n**Tool use (function calling)** — register tools with `add_tool`, advertise them in the system\nblock with `append_system`, then let `assistant_turn_with_tools` run the loop: it generates,\ndispatches each tool the model calls, feeds the results back as a `tool`-role message, and\nrepeats until the model produces a plain-text answer (bounded by `max_rounds` assistant turns).\nTool schemas and parsed arguments are plain strings — no `serde` dependency.\n\n```rust\nuse bebelm::agent::Agent;\nuse bebelm::model::Model;\nuse bebelm::tool::{Schema, Tool, Type};\n\nlet model = Model::load(\"LFM2.5-8B-A1B-Q4_K_M.gguf\")?;\n\n// Register tools before the system block. `Tool` is `Clone`, so `Agent` stays `Clone`.\nlet mut agent = Agent::new(\u0026model).add_tool(Tool::new(\n    \"add\",\n    \"Add two integers.\",\n    Schema::new()\n        .req(\"a\", Type::Int, \"First addend\")\n        .req(\"b\", Type::Int, \"Second addend\"),\n    |call| {\n        // Args arrive as raw text; `parse_arg` parses one into the receiver's type (`arg`\n        // gives the raw \u0026str). Both return `Option`, so the callback picks the fallback here.\n        let a: i64 = call.parse_arg(\"a\").unwrap_or(0);\n        let b: i64 = call.parse_arg(\"b\").unwrap_or(0);\n        (a + b).to_string()\n    },\n));\n\nagent.append_system(\"You are a helpful assistant.\");\nagent.append_user(\"What is 21 + 21?\");\n\n// Run the agentic loop: stream the reply, and observe each tool call + result.\nlet turn = agent.assistant_turn_with_tools(\n    8,                                            // max assistant turns\n    |_id, text| print!(\"{text}\"),\n    |call, result| eprintln!(\"[tool] {} -\u003e {result}\", call.name),\n);\nprintln!(\"\\n{}\", turn.text);\n```\n\n`Schema::req` / `Schema::opt` declare required / optional parameters (`Type` is `Str`, `Int`,\n`Num`, or `Bool`); `Tool::raw` is an escape hatch that takes the entire tool JSON verbatim. An\nunknown tool name is reported back to the model rather than aborting the loop.\n\n**Special tokens** live in `bebelm::tokenizer` as `u32` constants. The agent handles BOS, EOS,\nand the ChatML / `\u003cthink\u003e` framing for you — these are mostly for interpreting the `id` your\n`on_token` callback receives:\n\n- `TOKEN_BOS` — `\u003c|startoftext|\u003e`, start-of-sequence (auto-prepended on the first `append`).\n- `TOKEN_IM_START` / `TOKEN_IM_END` — `\u003c|im_start|\u003e` / `\u003c|im_end|\u003e`, ChatML turn delimiters.\n- `TOKEN_EOS` — alias of `TOKEN_IM_END`; ends a turn.\n- `TOKEN_THINK` / `TOKEN_THINK_END` — `\u003cthink\u003e` / `\u003c/think\u003e`, reasoning-block delimiters.\n- `TOKEN_ENDOFTEXT` / `TOKEN_PAD` — `\u003c|endoftext|\u003e` / `\u003c|pad|\u003e`, document/pad markers.\n- `TOKEN_TOOL_LIST_START` / `TOKEN_TOOL_LIST_END` / `TOKEN_TOOL_CALL_START` / `TOKEN_TOOL_CALL_END`\n  — `\u003c|tool_*|\u003e` delimiters.\n- `TOKEN_FIM_PRE` / `TOKEN_FIM_MID` / `TOKEN_FIM_SUF` — `\u003c|fim_*|\u003e` fill-in-the-middle markers.\n\nFor lower-level use, `Model::forward_step(token, \u0026mut Cache)` runs the cached forward pass\ndirectly, and `bebelm::tokenizer::Tokenizer` (`encode` / `decode`) and `bebelm::sampler::Sampler`\nare public if you want to drive decoding yourself.\n\n## CPU / SIMD build\n\nThe x86 SIMD kernels are tuned for the machine you build on: `.cargo/config.toml` sets\n`target-cpu=native`, so a build automatically uses **AVX2 + FMA** when the CPU has them\nand falls back to whatever it supports otherwise.\n\nBecause `native` targets the build host, a binary built on an AVX2 machine may fault on an\nolder CPU. To build a portable binary, override the CPU target via `RUSTFLAGS` (it takes\nprecedence over `.cargo/config.toml`):\n\n```sh\n# AVX2 baseline — runs on any Haswell (2013) or newer x86:\nRUSTFLAGS=\"-C target-cpu=x86-64-v3\" cargo build --release\n\n# Universal baseline — runs on any x86_64 (SSE2 only, slowest):\nRUSTFLAGS=\"-C target-cpu=x86-64\" cargo build --release\n```\n\nThe instruction set is chosen at build time; there is no single binary that switches at\nruntime.\n\n## Running the tests\n\nThe test suite has two layers:\n\n- **Fast unit tests** run with plain `cargo test` — they need no model file and finish in\n  seconds, so they are the default and what CI runs first.\n- **End-to-end tests** (`tests/end_to_end.rs`) load the full ~5.2 GB Q4_K_M GGUF and run real\n  generation against it. They are gated behind `#[ignore]` so `cargo test` stays model-free, and\n  they read the weights path from `BEBELM_WEIGHTS_FILE` (defaulting to the repo-root GGUF, same\n  resolution as the CLI — see **Setup instructions** for downloading it).\n\nRun the **full** end-to-end suite — every `#[ignore]`d test — with `--ignored`:\n\n```sh\ncargo test --release -- --ignored --test-threads=1\n```\n\nEach test loads the model independently and runs real decoding, so the full suite is slow. For\na quick **partial** run, append a test-name filter (a substring match) — e.g. the single\nParis-completion smoke test, the fastest one:\n\n```sh\n# one end-to-end test (fast smoke check)\ncargo test --release -- --ignored capital_of_france_is_paris\n```\n\nA broader substring targets a group, e.g. `cargo test --release -- --ignored multi_turn`. List\nthe available end-to-end tests without running them with\n`cargo test --release -- --ignored --list`. Always use `--release`: a debug build runs the\nnumeric kernels far slower.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaximecb%2Fbebelm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaximecb%2Fbebelm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaximecb%2Fbebelm/lists"}