{"id":41332850,"url":"https://github.com/lloyal-ai/liblloyal","last_synced_at":"2026-04-03T02:03:20.922Z","repository":{"id":334072428,"uuid":"1087547242","full_name":"lloyal-ai/liblloyal","owner":"lloyal-ai","description":"Covalent Inference for llama.cpp","archived":false,"fork":false,"pushed_at":"2026-03-13T06:38:47.000Z","size":64758,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-13T14:57:47.161Z","etag":null,"topics":["inference","llamacpp","research-framework"],"latest_commit_sha":null,"homepage":"https://lloyal-ai.github.io/liblloyal/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lloyal-ai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":null},"created_at":"2025-11-01T06:04:10.000Z","updated_at":"2026-03-13T06:12:46.000Z","dependencies_parsed_at":"2026-02-27T13:04:13.003Z","dependency_job_id":null,"html_url":"https://github.com/lloyal-ai/liblloyal","commit_stats":null,"previous_names":["lloyal-ai/liblloyal"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/lloyal-ai/liblloyal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lloyal-ai%2Fliblloyal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lloyal-ai%2Fliblloyal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lloyal-ai%2Fliblloyal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lloyal-ai%2Fliblloyal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lloyal-ai","download_url":"https://codeload.github.com/lloyal-ai/liblloyal/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lloyal-ai%2Fliblloyal/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31326881,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-03T01:42:14.489Z","status":"online","status_checked_at":"2026-04-03T02:00:06.642Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["inference","llamacpp","research-framework"],"created_at":"2026-01-23T06:19:27.656Z","updated_at":"2026-04-03T02:03:20.916Z","avatar_url":"https://github.com/lloyal-ai.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# liblloyal\n\n[![Tests](https://github.com/lloyal-ai/liblloyal/actions/workflows/tests.yml/badge.svg)](https://github.com/lloyal-ai/liblloyal/actions/workflows/tests.yml)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)\n[![C++](https://img.shields.io/badge/C++-20-blue.svg)](https://en.cppreference.com/w/cpp/20)\n[![llama.cpp](https://img.shields.io/badge/llama.cpp-b8087-green.svg)](https://github.com/ggml-org/llama.cpp/releases/tag/b8087)\n\n**Covalent Inference for llama.cpp**\n\nComposable C++ primitives for forkable decode state and shared-prefix (KV) branching in llama.cpp. Fork a generation into a tree — branches share a prefix while keeping independent machinery (sampler chain, seed, grammar, logits snapshot, perplexity tracker) for controlled divergence at decode time.\n\n## Continuous Tree Batching\n\nTree search with N branches means N calls to `llama_decode()` — each paying GPU dispatch overhead, memory barriers, and PCIe round-trips. Continuous tree batching eliminates this: BranchStore packs tokens from N branches — each at a different position, different seq_id, each needing independent logits captured — into a single `llama_batch` and dispatches once. N branches, 1 GPU call.\n\n```cpp\n// Tree search inner loop: all branches advance in one GPU dispatch\nstore.decode_each({{child1.handle(), tok1},\n                   {child2.handle(), tok2},\n                   {child3.handle(), tok3}});\n// 3 branches × 3 positions × 3 seq_ids → 1 llama_decode()\n// Per-branch logits captured, positions advanced\n```\n\nTwo packing strategies for different access patterns:\n\n```cpp\n// commit: 1 token per branch — synchronous tree expansion\nstore.decode_each(items);\n\n// prefill: variable tokens per branch — asymmetric injection\nstore.decode_scatter({\n    {branchA.handle(), system_tokens},  // 200 tokens\n    {branchB.handle(), query_tokens},   //  12 tokens\n    {branchC.handle(), doc_tokens},     // 800 tokens\n});\n// Greedy bin-packed into ceil(total / n_batch) dispatches\n// Oversized items auto-fallback to chunked single-sequence decode\n```\n\nThe underlying decode grid (`decode.hpp`):\n\n|                  | Single Sequence  | Multi Sequence   |\n|------------------|------------------|------------------|\n| **Single Token** | `decode::one`    | `decode::each`   |\n| **Multi Token**  | `decode::many`   | `decode::scatter` |\n\n## The Branch API\n\nEach branch owns a KV cache lease, sampler chain, grammar, metrics tracker, and logits snapshot — everything needed for independent generation. `fork()` deep-clones all of it. Branches compose into best-of-N, speculative decoding, tree search, beam search.\n\n```cpp\nusing namespace lloyal::branch;\n\nBranchStore store;\nstore.init_tenancy(ctx);\n\n// Shared prompt: \"Explain quantum entanglement\"\nauto root = Branch::create(ctx, model, store, 0, params);\nroot.prefill(prompt_tokens.data(), prompt_tokens.size());\n\n// Fork 4 branches — each will get a different reasoning prefix\nauto analogy  = root.fork();\nauto formal   = root.fork();\nauto socratic = root.fork();\nauto visual   = root.fork();\n\n// Scatter-prefill: inject divergent prefixes in one batched dispatch\n// 4 branches × variable lengths → auto bin-packed into minimal GPU calls\nstore.decode_scatter({\n    {analogy.handle(),  tokenize(\"Think of it like two coins...\")},   // 12 tokens\n    {formal.handle(),   tokenize(\"In quantum mechanics, the...\")},    // 8 tokens\n    {socratic.handle(), tokenize(\"What happens when you measure...\")},// 10 tokens\n    {visual.handle(),   tokenize(\"Imagine two particles...\")},        // 7 tokens\n});\n\n// Generate — all 4 in lockstep, 1 GPU call per step\nstd::vector\u003cBranch*\u003e branches = {\u0026analogy, \u0026formal, \u0026socratic, \u0026visual};\nwhile (!branches.empty()) {\n    std::vector\u003cdecode::EachItem\u003e items;\n    for (auto* b : branches) {\n        auto tok = b-\u003esample();\n        if (b-\u003eis_eog(tok)) { b-\u003eprune(); continue; }\n        b-\u003eaccept(tok);\n        items.push_back({b-\u003ehandle(), tok});\n    }\n    std::erase_if(branches, [](auto* b) { return !b-\u003evalid(); });\n    if (!items.empty()) store.decode_each(items);\n}\n\n// Winner takes all — one seq_keep pass, losers vaporized\nauto* winner = *std::min_element(branches.begin(), branches.end(),\n    [](auto* a, auto* b) { return a-\u003eperplexity() \u003c b-\u003eperplexity(); });\nstore.retainOnly(winner-\u003ehandle());\n```\n\n**What `fork()` clones:** KV cache sequence, sampler chain handle (penalties, PRNG, filters), grammar handle (GBNF parser state), metrics handle (model + sampling perplexity), logits snapshot, logit bias, cached sampler params.\n\n**What `fork()` does NOT clone:** steer callback (captures references, unsafe to copy).\n\n## Hot-Swap Sampler \u0026 Grammar\n\nSampler chains, grammars, and metrics live in handle-based registries on BranchStore — instance-scoped, no global state. `set_sampler_params()` rebuilds the sampler chain with memoization (no-op if params unchanged). `set_grammar()` hot-swaps the grammar constraint.\n\n```cpp\n// EDT (Entropy-based Dynamic Temperature): adapt temperature per token\nfor (int i = 0; i \u003c max_tokens; i++) {\n    float entropy = metrics::model_entropy(root.logits(), root.n_vocab());\n    float temp = T0 * std::pow(N, THETA / std::max(entropy, 0.1f));\n    root.setSamplerParams(MyParams{.temperature = temp});  // memoized — no-op if temp unchanged\n    auto tok = root.sample();\n    if (root.is_eog(tok)) break;\n    root.accept(tok);\n    root.step(tok);\n}\n\n// Hot-swap grammar mid-generation\nroot.setGrammar(json_gbnf);     // constrain to JSON\nauto tok = root.sample();       // grammar-legal token\nroot.setGrammar(nullptr);       // remove constraint\n```\n\nHandles are freed automatically on `prune()` — no manual cleanup. `fork()` deep-clones all registry entries.\n\n## KV Tenancy\n\nTwo resources, two scales. Slots (65K) are how many branches can *exist* — cheap CPU state. Leases (256) are how many can *decode* — scarce KV cache residency. `kv::tenancy` manages the scarce resource as leases, acquired on `create()`/`fork()`, evicted on `prune()`, rebuilt on `retainOnly()`. No manual seq_id tracking, ever.\n\n```cpp\nstore.available();              // leases remaining — use for width/depth budget\nstore.retainOnly(winner);       // nuclear: 1 seq_keep, rebuild vacancy\nstore.drain();                  // explicit teardown before llama_free(ctx)\n```\n\nThe turn lifecycle: search is surgical (N × `prune()`), promotion is nuclear (1 × `retainOnly()`). Per turn, fork → expand → evaluate → prune losers → repeat. Between turns, promote winner → tree is gone → next turn starts fresh.\n\n## Topology\n\nParent/child edges are always-on. Simple chat → best-of-N → deep search is one continuum — the library provides topology queries at every point on the spectrum.\n\n```cpp\nstore.parent(handle);       // INVALID_HANDLE if root\nstore.children(handle);     // child handles\nstore.isLeaf(handle);       // no children?\nstore.isActive(handle);     // holds a KV lease?\n```\n\n| Method | FK analogy | Behavior |\n|--------|-----------|----------|\n| `prune()` | RESTRICT | Throws if children exist |\n| `pruneSubtree()` | CASCADE | Iterative post-order traversal |\n\nRAII `~Branch()` uses CASCADE — cleanup always succeeds, even with deep trees. Multi-tag KV cells ensure pruning a parent doesn't corrupt children's cache — a cell is freed only when ALL tags are removed.\n\n## Primitives\n\nThe building blocks that compose into the above:\n\n- **Tokenization** — Two-pass safe buffer sizing, special token handling\n- **Decoding** — Continuous tree batching, cross-sequence dispatch packing\n- **KV Cache** — Tenancy (vacancy manager), sequence ops, state snapshots, long-context compression\n- **Sampling** — Grammar-constrained, persistent chains, hot-swap with memoization\n- **Metrics** — Dual-level entropy/surprisal, rolling perplexity, cloneable state (BranchStore-scoped)\n- **Embeddings** — Pooled extraction, L2 normalization, similarity\n- **Chat Templates** — Jinja2 formatting with fallbacks\n\nLower-level handles for fine-grained control:\n\n```cpp\nauto chain = lloyal::sampler::create_chain(params);\nauto grammar_handle = lloyal::grammar::init_sampler(model, schema);\n```\n\nShared model weights — multiple contexts, one model load:\n\n```cpp\nauto model1 = lloyal::ModelRegistry::acquire(path, params);\nauto model2 = lloyal::ModelRegistry::acquire(path, params);  // cache hit\n```\n\n## From Simple to Complex\n\n**Single-sequence streaming** — the baseline everyone has:\n\n```cpp\nlloyal::decode::many(ctx, prompt_tokens.data(), prompt_tokens.size(), 0, n_batch);\nwhile (!done) {\n    auto token = lloyal::sampler::sample_with_params(ctx, vocab, params);\n    lloyal::decode::one(ctx, token, n_past++);\n}\n```\n\n**Best-of-N** — fork once, diverge everywhere, keep the best:\n\n```cpp\nusing namespace lloyal::branch;\n\nBranchStore store;\nstore.init_tenancy(ctx);\n\nauto root = Branch::create(ctx, model, store, 0, params);\nroot.prefill(prompt_tokens.data(), prompt_tokens.size());\n\n// Fork 8 candidates — KV prefix shared, each gets unique PRNG\nstd::vector\u003cBranch\u003e candidates;\nfor (int i = 0; i \u003c 8; i++) {\n    candidates.push_back(root.fork());\n    reseed_chain(candidates.back().handle(), store, 1000 + i);\n}\n\n// Generate 64 tokens — all 8 branches batched into single GPU calls\nfor (int t = 0; t \u003c 64; t++) {\n    std::vector\u003cdecode::EachItem\u003e items;\n    for (auto\u0026 c : candidates) {\n        auto tok = c.sample();\n        c.accept(tok);\n        items.push_back({c.handle(), tok});\n    }\n    store.decode_each(items);  // 8 branches, 1 llama_decode()\n}\n\n// Winner takes all — one seq_keep pass, 7 branches vaporized\nauto\u0026 winner = *std::min_element(candidates.begin(), candidates.end(),\n    [](auto\u0026 a, auto\u0026 b) { return a.perplexity() \u003c b.perplexity(); });\nstore.retainOnly(winner.handle());\n// store.available() == n_seq_max - 1 — all leases recovered\n```\n\n**Tree search with continuous tree batching** — the full show:\n\n```cpp\nBranchStore store;\nstore.init_tenancy(ctx);\n\nauto root = Branch::create(ctx, model, store, 0, params);\nroot.prefill(prompt_tokens.data(), prompt_tokens.size());\n\nfor (int turn = 0; turn \u003c max_turns; turn++) {\n    // Expand: fork children, each samples a different continuation\n    std::vector\u003cBranch\u003e leaves;\n    int width = std::min((int)store.available(), max_width);\n    for (int i = 0; i \u003c width; i++) {\n        auto leaf = root.fork();\n        reseed_chain(leaf.handle(), store, turn * 1000 + i);\n        leaves.push_back(std::move(leaf));\n    }\n\n    // Evaluate: generate depth tokens per leaf, batched across all branches\n    for (int d = 0; d \u003c depth; d++) {\n        std::vector\u003cdecode::EachItem\u003e items;\n        for (auto\u0026 leaf : leaves) {\n            auto tok = leaf.sample();\n            leaf.accept(tok);\n            items.push_back({leaf.handle(), tok});\n        }\n        store.decode_each(items);  // width branches × 1 GPU dispatch\n    }\n\n    // Score + surgical prune: RESTRICT evicts one lease per loser\n    std::sort(leaves.begin(), leaves.end(),\n        [](auto\u0026 a, auto\u0026 b) { return a.perplexity() \u003c b.perplexity(); });\n    for (size_t i = 1; i \u003c leaves.size(); i++)\n        leaves[i].prune();\n\n    // Promote: nuclear — winner becomes the new trunk\n    store.retainOnly(leaves[0].handle());\n    root = std::move(leaves[0]);\n}\n```\n\n## Architecture\n\n- **Header-only** — All implementations inline in `include/lloyal/*.hpp`\n- **Managed KV residency** — `kv::tenancy` tracks seq_id leases; consumers never see raw seq_ids\n- **Handle-based APIs** — Generation counters prevent ABA bugs on slot reuse; sampler chains, grammars, and metrics live in BranchStore-scoped registries (no global state)\n- **Shared model weights** — Thread-safe registry enables multi-context with single model load\n- **Zero runtime dependencies** — Only requires C++20 standard library + llama.cpp\n- **Multi-binding** — C++20 concepts decouple from binding-specific types (Node.js, React Native, CLI)\n\n## Integration\n\n### Git Submodule\n\n```bash\ngit submodule add -b v0.1.0 https://github.com/lloyal-ai/liblloyal.git\n```\n\n### CMake\n\n```cmake\nadd_subdirectory(liblloyal)\ntarget_link_libraries(your_target PRIVATE lloyal llama)\n```\n\n### CocoaPods (iOS)\n\n```ruby\ns.header_dir = \"lloyal\"\ns.source_files = \"liblloyal/include/**/*.{hpp,h}\"\n```\n\n## Documentation\n\n**Usage Guide:** [`docs/guide.md`](docs/guide.md)\n\n**API Reference:** Auto-generated from Doxygen-annotated headers\n\n- **Online:** [lloyal-ai.github.io/liblloyal](https://lloyal-ai.github.io/liblloyal/)\n- **Local:** `./scripts/generate-docs.sh` → `docs/api/html/index.html`\n- **Headers:** `include/lloyal/*.hpp` — fully documented with Doxygen\n\n## Testing\n\n- 256 unit tests (tenancy, topology, continuous tree batching, RESTRICT/CASCADE, handle registries)\n- 128 integration tests with real llama.cpp (multi-step generation, ABA prevention, batch error paths, retainOnly, hot-swap sampler/grammar)\n- Sanitizer validation (ASan, UBSan, LeakSan)\n\n```bash\n# Unit tests (stub-based, no model required)\ncd tests \u0026\u0026 cmake -B build \u0026\u0026 cmake --build build \u0026\u0026 ./build/TestRunner\n\n# Integration tests (real llama.cpp)\ncd tests \u0026\u0026 cmake -B build -DLLOYAL_BUILD_INTEGRATION_TESTS=ON \\\n  -DLLAMA_CPP_DIR=../llama.cpp \u0026\u0026 cmake --build build\nLLAMA_TEST_MODEL=path/to/model.gguf ./build/IntegrationRunner\n```\n\n## Design Principles\n\n1. **Primitives, not opinions** — Build your patterns, we provide the tools\n2. **Managed scarcity** — KV leases are automatic; capacity is queryable\n3. **Explicit over implicit** — No hidden state, clear contracts\n4. **Testable** — No framework coupling, works standalone\n5. **Version-isolated** — Absorbs llama.cpp API changes\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for development guidelines.\n\n## License\n\nApache 2.0 — See LICENSE file for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flloyal-ai%2Fliblloyal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flloyal-ai%2Fliblloyal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flloyal-ai%2Fliblloyal/lists"}