{"id":51503264,"url":"https://github.com/shifulegend/project-zero","last_synced_at":"2026-07-26T18:00:38.129Z","repository":{"id":362984414,"uuid":"1261155086","full_name":"shifulegend/project-zero","owner":"shifulegend","description":"CPU-optimized LLM inference engine (C)","archived":false,"fork":false,"pushed_at":"2026-07-24T11:51:41.000Z","size":31406,"stargazers_count":13,"open_issues_count":3,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-07-24T12:04:41.886Z","etag":null,"topics":["avx512","bitnet","c99","cpu","cpu-inference","deepseek","gguf","inference","llm","moe","simd","ternary","zero-dependencies"],"latest_commit_sha":null,"homepage":"https://github.com/shifulegend/project-zero/discussions","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shifulegend.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":".github/ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-06-06T10:04:44.000Z","updated_at":"2026-07-23T13:55:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"f247a763-14c4-4dcd-888c-5724a78377c0","html_url":"https://github.com/shifulegend/project-zero","commit_stats":null,"previous_names":["shifulegend/project-zero"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/shifulegend/project-zero","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shifulegend%2Fproject-zero","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shifulegend%2Fproject-zero/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shifulegend%2Fproject-zero/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shifulegend%2Fproject-zero/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shifulegend","download_url":"https://codeload.github.com/shifulegend/project-zero/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shifulegend%2Fproject-zero/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35922387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-26T02:00:06.503Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avx512","bitnet","c99","cpu","cpu-inference","deepseek","gguf","inference","llm","moe","simd","ternary","zero-dependencies"],"created_at":"2026-07-07T22:00:26.753Z","updated_at":"2026-07-26T18:00:38.116Z","avatar_url":"https://github.com/shifulegend.png","language":"C","funding_links":[],"categories":["Runtimes and Inference Engines","🔓 Open Source Inference Engines"],"sub_categories":[],"readme":"# Project Zero — CPU LLM Inference Engine\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Language: C](https://img.shields.io/badge/language-C99-blue.svg)](src/)\n[![SIMD](https://img.shields.io/badge/SIMD-AVX--512%20%7C%20AVX2%20%7C%20NEON-green)](src/math/)\n[![Benchmarks](https://img.shields.io/badge/Benchmarks-OpenBenchmarking.org-orange)](https://openbenchmarking.org/result/2606207-SHIF-PROJECT42)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/tty_bitnet.png\" width=\"720\" alt=\"Project Zero: BitNet b1.58 running at 36.25 tok/s on Intel Xeon, no GPU\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/demo_bitnet.gif\" width=\"720\" alt=\"Project Zero: BitNet b1.58-2B-4T live demo on i5-11300H\"\u003e\n\u003c/p\u003e\n\n[Benchmarks](#benchmarks) · [Quick Start](#quick-start) · [UI/UX](#ui-ux) · [Help Wanted](#help-wanted) · [Docs](docs/)\n\n---\n\nPure C, single binary. Runs Microsoft's BitNet b1.58 **up to 5.4× faster than Microsoft's own `bitnet.cpp`**, PrismML's Bonsai-27B **4.2–4.8× faster than PrismML's own engine fork**, and dense GGUF models — no GPU, no Python, no ML framework.\n\n- **Pure C, zero runtime deps** — `make release`, one executable, nothing else required\n- **3.5–8.3× faster than bitnet.cpp on i5-11300H** (INT4, t=1..8) · **1.33–1.80× faster on 4-core Xeon** ([third-party verified on OpenBenchmarking.org ↓](#benchmarks))\n- **35.79 tok/s on i5-11300H (INT4, 500 tokens, best-of-3)** · **36.25 tok/s on Xeon (PGO+LTO)**\n- **One binary, two model families** — BitNet ternary and dense F16 GGUF, no per-model rebuild\n\n---\n\n### Bonsai-27B on ordinary x86 — faster than the vendor's own engine\n\nProject Zero runs [PrismML's Ternary-Bonsai-27B](https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf) (ternary Q2_0, 7.16 GB, Apache 2.0) **4.2–4.8× faster than PrismML's own llama.cpp fork at every thread count** on a plain 4-core AVX-512 Xeon VM — **2.97 vs 0.70 tok/s at t=4** (60-token greedy decode, identical file/prompt/session, drift-bracketed by sentinel runs; [18 raw terminal captures + methodology](benchmark_results/sweep3_2026-07-18/) · [full comparison ↓](#bonsai)).\n\nRun it yourself — one binary, no Python (model download ~7.2 GB):\n\n```bash\ngit clone https://github.com/shifulegend/project-zero \u0026\u0026 cd project-zero \u0026\u0026 make release\ncurl -fL -o models/Ternary-Bonsai-27B-Q2_0.gguf \\\n  https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf/resolve/main/Ternary-Bonsai-27B-Q2_0.gguf\n./adaptive_ai_engine --model models/Ternary-Bonsai-27B-Q2_0.gguf \\\n  --prompt \"What is the capital of France?\" --max-tokens 60 --temperature 0 --threads 4\n```\n\n---\n\n\u003ca id=\"benchmarks\"\u003e\u003c/a\u003e\n\n## Benchmarks\n\n### Intel i5-11300H @ 3.10 GHz · 16 GB DDR4 · AVX-512 VNNI (2026-06-21, fresh sweep)\n\n**BitNet b1.58-2B-4T — Project Zero vs. Microsoft `bitnet.cpp`** — same model, same machine, same prompt, 500 tokens, sequential, **best of 3 runs per thread count**:\n\n| Threads | PZ BF16 (tok/s) | PZ INT4 (tok/s) | MSFT bitnet.cpp | BF16 Gain | INT4 Gain |\n|---|---|---|---|---|---|\n| 1 | 11.64 | **16.98** | 2.04 | +471% | +732% |\n| 2 | 20.23 | **27.99** | 3.76 | +438% | +644% |\n| 3 | 22.31 | **33.82** | 5.17 | +332% | +554% |\n| 4 | 23.42 | **35.79** | 6.64 | +253% | +439% |\n| 5 | 21.62 | **31.80** | 5.15 | +320% | +517% |\n| 6 | 22.28 | **33.68** | 6.01 | +271% | +460% |\n| 7 | 21.50 | **32.50** | 6.61 | +225% | +392% |\n| 8 | 21.28 | **32.03** | 6.04 | +252% | +430% |\n\n**Peak: PZ INT4 = 35.79 tok/s (t=4) · PZ BF16 = 23.42 tok/s (t=4) · MSFT = 6.64 tok/s (t=4)**\n\n\u003e Project Zero INT4 is **3.9–8.3×** faster than bitnet.cpp across all thread counts. BF16 is **3.3–5.7×** faster. Prompt also faster: MSFT reports 7.00 tok/s prompt eval at t=4 vs ~110+ tok/s for the PZ tokenizer path.\n\n**Peak-run terminal screenshots — same machine, same model, same prompt (500 tokens, best-of-3):**\n\n| PZ INT4 · t=4 · **35.79 tok/s** | PZ BF16 · t=4 · **23.42 tok/s** | MSFT bitnet.cpp · t=4 · **6.64 tok/s** |\n|---|---|---|\n| ![PZ INT4 peak](benchmark_results/sweep_2026-06-21/screenshots/bitnet_pz_int4_t4.png) | ![PZ BF16 peak](benchmark_results/sweep_2026-06-21/screenshots/bitnet_pz_bf16_t4.png) | ![MSFT peak](benchmark_results/sweep_2026-06-21/screenshots/bitnet_msft_t4.png) |\n\nAll 24 screenshots (t=1..8 × 3 engines): [`benchmark_results/sweep_2026-06-21/screenshots/`](benchmark_results/sweep_2026-06-21/screenshots/)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/comparison_graph_i5.png\" width=\"720\" alt=\"Tok/s vs threads: PZ BF16, PZ INT4, Microsoft bitnet.cpp on i5-11300H\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/bar_chart_i5.png\" width=\"640\" alt=\"Peak throughput bar chart: PZ INT4 35.79, PZ BF16 23.42, MSFT 6.64 tok/s\"\u003e\n\u003c/p\u003e\n\n### Intel Xeon · AVX-512 VNNI (PGO+LTO, from earlier run)\n\n| Threads | Project Zero | bitnet.cpp (i2_s) | Gain |\n|---|---|---|---|\n| 1 | **5.91 tok/s** | 4.96 | +19% |\n| 2 | **12.78 tok/s** | 9.46 | +35% |\n| 3 | **18.61 tok/s** | 13.59 | +37% |\n| 4 | **21.45 tok/s** | 16.10 | +33% |\n\nOptimized (PGO+LTO, INT4 classifier): **36.25 tok/s = 95% of the analytical DRAM bandwidth ceiling** on a 4-core Xeon.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/benchmark_bitnet.png\" width=\"720\" alt=\"BitNet b1.58-2B-4T: Project Zero beats Microsoft bitnet.cpp at every thread count (Xeon)\"\u003e\n\u003c/p\u003e\n\n\u003e On dense models, Project Zero leads `llama.cpp` at 1–3 threads (+32%/+4%/+15%) and trails at peak 4-thread. On DeepSeek-V2 MoE it runs ~7× slower — this is the known open problem ([Help Wanted ↓](#help-wanted)).\n\n### SmolLM2-135M F16 — Project Zero vs. llama.cpp (i5-11300H, 2026-06-21, fresh sweep)\n\nSame model, same machine, same prompt, 500 tokens, sequential, **best of 3 runs per thread count**:\n\n| Threads | Project Zero BF16 (tok/s) | llama.cpp (tok/s) | PZ Gain | llama.cpp Prompt tok/s |\n|---|---|---|---|---|\n| 1 | 58.00 | 53.50 | +8.4% | 319.7 |\n| 2 | 86.03 | 84.20 | +2.2% | 588.7 |\n| 3 | **100.44** | 106.20 | −5.4% | 700.9 |\n| 4 | 98.28 | **106.20** | −3.7% | 1092.0 |\n| 5 | 95.93 | 93.90 | +2.2% | 1063.6 |\n| 6 | 89.63 | 94.90 | −5.6% | 1019.6 |\n| 7 | 90.57 | 95.30 | −5.0% | 1020.4 |\n| 8 | 83.32 | 86.30 | −3.5% | 958.1 |\n\n**Peak: PZ = 100.44 tok/s (t=3) · llama.cpp = 106.20 tok/s (t=3)**\n\n\u003e PZ leads at t=1 (+8.4%) and t=2 (+2.2%), trails by 3–6% at peak. No fused Q4K matmul yet — see [Help Wanted ↓](#help-wanted). llama.cpp prompt eval is faster (700–1092 tok/s) because it batches the prompt; PZ does not yet report prompt eval speed separately.\n\n**Peak-run screenshots — SmolLM2 (best-of-3):**\n\n| PZ BF16 · t=3 · **100.44 tok/s** | llama.cpp · t=3 · **106.20 tok/s** |\n|---|---|\n| ![PZ SmolLM2 peak](benchmark_results/sweep_2026-06-21/screenshots/smollm2_pz_t3.png) | ![llama.cpp SmolLM2 peak](benchmark_results/sweep_2026-06-21/screenshots/smollm2_llama_t3.png) |\n\nAll 16 screenshots (t=1..8 × 2 engines): [`benchmark_results/sweep_2026-06-21/screenshots/`](benchmark_results/sweep_2026-06-21/screenshots/)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/benchmark_smollm2.png\" width=\"720\" alt=\"SmolLM2-135M F16: Project Zero vs llama.cpp t=1..8\"\u003e\n\u003c/p\u003e\n\n**Live terminal runs — Xeon (BitNet b1.58-2B-4T) and i5-11300H (SmolLM2-135M F16):**\n\n| BitNet b1.58-2B-4T (ternary, Xeon) | SmolLM2-135M (F16 dense, i5) |\n|---|---|\n| ![BitNet live run](docs/tty_bitnet.png) | ![SmolLM2 live run](docs/tty_smollm2.png) |\n\n**Xeon demo — 31-second live recording:**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/demo_xeon.gif\" width=\"720\" alt=\"Project Zero BitNet b1.58-2B-4T live demo on Xeon (36.25 tok/s)\"\u003e\n\u003c/p\u003e\n\n**Optimization journey — throughput across all phases:**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/performance_chart.png\" width=\"720\" alt=\"Project Zero throughput vs bitnet.cpp / llama.cpp across optimization steps\"\u003e\n\u003c/p\u003e\n\n*Per-configuration throughput from the [optimization journal](docs/PERFORMANCE_CEILING_REPORT.md).*\n\n📊 **Third-party results on OpenBenchmarking.org** — not self-reported:\n\n| Xeon vs. bitnet.cpp | i5-11300H vs. llama.cpp |\n|---|---|\n| [![Xeon result](docs/openbenchmarking_xeon_vs_bitnetcpp.png)](https://openbenchmarking.org/result/2606207-SHIF-PROJECT42) | [![i5 result](docs/openbenchmarking_i5_vs_llamacpp.png)](https://openbenchmarking.org/result/2606208-SHIF-PROJECT03) |\n\n**Run it yourself and post your result:** [Discussion #3 — community benchmarks](https://github.com/shifulegend/project-zero/discussions/3)\n\n\u003ca id=\"bonsai\"\u003e\u003c/a\u003e\n\n### Qwen 3.5/3.6 (Ternary-Bonsai-27B, hybrid Gated-DeltaNet + GQA, Q2_0 ternary) — 4-core Xeon VM\n\n**What made this fast, specifically:**\n\n- **Format detection, not a documented spec.** Ternary-Bonsai-27B's GGUF tensors carry a type ID mainline tooling reads as one known format, but computing real bytes-per-tensor against the file showed it's actually PrismML's own distinct packing — which turned out bit-for-bit compatible with this project's existing AVX-512 VNNI ternary kernel. Connecting the two made Q2_0 matmul **~29x faster** end-to-end (0.11 → 3.24 tok/s, measured A/B on the same host; [`mistakes.md`](docs/ai/mistakes.md)).\n- **Activations quantized once per matmul call, shared across every worker thread** — not redundantly re-quantized by each of the T threads (`src/math/parallel_matmul.c`).\n- **A real ISA-dispatch bug, not just a fallback path.** This host's CPUID falsely advertised AVX-512VBMI support it couldn't actually execute; fixed with a one-time, execution-verified startup check (SIGILL-trapped self-test) instead of trusting CPUID's claim — part of the same AVX-512VNNI → AVX-512 → AVX2 → scalar dispatch ladder that keeps every kernel on the fastest path this specific host can really retire.\n- **One binary, two weight formats** — this same executable runs both native packed-ternary and dense/quantized GGUF models, including this one, without a per-model rebuild.\n- **The same VNNI ternary kernel beats Microsoft's own `bitnet.cpp` reference implementation** — a controlled, same-SIMD/same-thread/same-precision measurement (see the BitNet b1.58 table up top) shows Project Zero **+19-37% faster than `bitnet.cpp`** at every thread count, BF16 head-to-head. Full methodology in [`docs/reports/BENCHMARK_REPORT.md`](docs/reports/BENCHMARK_REPORT.md) Addendum AP.\n\n**Thread scaling, Project Zero vs. llama.cpp** — same prompt, same identical `Ternary-Bonsai-27B-Q2_0.gguf` file, greedy decoding, 60-token cap, run **strictly sequentially** (one process at a time, full exit before the next starts):\n\n| Threads | Project Zero (tok/s) | llama.cpp (tok/s) | PZ Gain |\n|---|---|---|---|\n| 1 | 0.86 | 0.2 | +330% |\n| 2 | 1.62 | 0.4 | +305% |\n| 3 | 2.31 | 0.6 | +285% |\n| 4 | **2.74** | 0.8 | +243% |\n\nProject Zero scales near-linearly across all 4 physical cores with no plateau — 4 threads is confirmed as the throughput-optimal setting on this host, not an unverified assumption. llama.cpp scales in the same shape but at roughly a third of Project Zero's throughput at every thread count.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/qwen35_thread_scaling.png\" width=\"720\" alt=\"Ternary-Bonsai-27B thread scaling: Project Zero vs llama.cpp, 1-4 threads\"\u003e\n\u003c/p\u003e\n\n**Peak-run terminal screenshots (t=4) and t=1 for comparison:**\n\n| PZ · t=4 · **2.74 tok/s** | llama.cpp · t=4 · **0.8 tok/s** | PZ · t=1 · **0.86 tok/s** | llama.cpp · t=1 · **0.2 tok/s** |\n|---|---|---|---|\n| ![PZ t4](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/pz_t4_peak.png) | ![llama.cpp t4](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/llamacpp_t4_peak.png) | ![PZ t1](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/pz_t1.png) | ![llama.cpp t1](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/llamacpp_t1.png) |\n\nAll 8 screenshots (t=1..4 × 2 engines): [`benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/`](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/)\n\n**Note on the PZ screenshots above:** the original captures (2026-07-16) were taken before a startup-banner display fix and two capture-tool bugs were found. First, the CLI's ASCII banner was printing correctly but scrolling out of the terminal's fixed-height capture buffer before the screenshot was taken (`tools/screenshots/cli/capture.mjs` fixed a hardcoded 70-row terminal against Ternary-Bonsai-27B's \u003e100-line startup output — widened to 170 rows). Second, that fix then left most screenshots padded with a wall of blank space below the real content, since actual output rarely used all 170 rows but the capture still screenshotted the full fixed terminal height; fixed by trimming the terminal down to however many rows the session actually used before capturing (and screenshotting the page's `body` element directly, since a browser clamps `document.documentElement.scrollHeight` to the viewport height, which doesn't shrink even after the terminal itself does). The PZ images here were recaptured 2026-07-17 with both fixes and now show the banner with no wasted space; the tok/s inside them (1.07 at t=4, 0.31 at t=1) is lower than the 2.74/0.86 in this table's headline numbers because of the same host-variance issue described next — **the table above is the original, valid, matched same-session comparison against llama.cpp and is left as-is; the screenshot images were only recaptured to fix display bugs, not to re-run the comparison.**\n\n**A caveat on absolute numbers, found while investigating a follow-up question:** re-running this exact same command later in the same overall effort (same file, same flags, same thread count) measured well below 2.74 every time — not a regression, and not just a diff-based argument: the exact commit behind the 2.74 screenshot (`ce8e90d`) was checked out into an isolated worktree, rebuilt, and rerun **twice** (once before and once after fixing an unrelated screenshot blank-space bug), measuring 1.40 then 1.02 tok/s — two different numbers, same conclusion. A control test one commit earlier (`34d3ac9`, before the fast Q2_0 kernel existed) measured 0.12 then 0.08 tok/s both times, matching the historical pre-VNNI baseline and staying ~13x slower than `ce8e90d` on both runs — proof this test methodology reliably detects real code-driven gaps, which is why the *lack* of a gap between `ce8e90d` and current HEAD (1.08 then 0.95 tok/s across the same two rounds) is meaningful rather than noise. Screenshots: [`commit_bisect_ce8e90d_1.02toks.png`](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/commit_bisect_ce8e90d_1.02toks.png) · [`commit_bisect_34d3ac9_0.08toks.png`](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/commit_bisect_34d3ac9_0.08toks.png) · [`commit_bisect_HEAD_0.95toks.png`](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/commit_bisect_HEAD_0.95toks.png). Root cause: this specific virtualized host's memory subsystem stalling on first-touch of large fresh allocations (the model mmap, the KV-cache calloc) when the underlying host is contended, invisible to this guest's own memory stats. Full evidence chain in [`docs/ai/mistakes.md`](docs/ai/mistakes.md). Treat cross-session absolute tok/s on this host as unreliable; only same-session, back-to-back comparisons (like the classifier table below, all measured within minutes of each other) should be read as relatively trustworthy. Consolidated root-cause analysis with the full evidence chain and timeline: [`docs/reports/RCA_QWEN_TOKS_DROP_2026-07.md`](docs/reports/RCA_QWEN_TOKS_DROP_2026-07.md). Two further corrections from that investigation, both now fixed in the engine: the \"Data/token\" and \"Ceiling\" figures visible in these screenshots were computed from hardcoded BitNet-2B constants (~6x too optimistic for this 27B model), **and** the \"DRAM bandwidth (measured)\" figures were ~3x too low (a probe accounting bug: three read passes timed, one pass of bytes counted). The engine now reports model-adjusted data/token after load and measures bandwidth correctly (same host: 12.0 → 41.2 GB/s). Honest ceiling for this model on these hosts: ~6-7 tok/s. Full spec and audit: [`docs/architecture/CEILING_CALCULATION.md`](docs/architecture/CEILING_CALCULATION.md).\n\n**Full 3-axis sweep vs the PrismML fork (2026-07-18):** 18 sequential same-session runs (threads × SIMD × classifier for project-zero; threads for the fork; 4 interleaved drift sentinels) — project-zero leads **4.2–4.8x at every thread count** (t4: 2.97 vs 0.70 tok/s).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"benchmark_results/sweep3_2026-07-18/comparison_infographic.png\" width=\"860\" alt=\"Benchmark telemetry infographic: project-zero vs PrismML llama.cpp fork on Ternary-Bonsai-27B — 4.2-4.8x faster at every thread count, plus project-zero-only SIMD and classifier axes, bracketed by drift sentinels\"\u003e\n\u003c/p\u003e\n\nRaw terminal screenshots + raw pty byte streams for every run: [`benchmark_results/sweep3_2026-07-18/`](benchmark_results/sweep3_2026-07-18/) · interactive version of this chart: [`comparison.html`](benchmark_results/sweep3_2026-07-18/comparison.html).\n\n**Kernel update (2026-07-17):** restructuring the Q2_0 VNNI row dot (per-row vector accumulator instead of a per-block horizontal reduction, F16C scale decode) lifted this model from 2.74/2.80 to **3.56/3.54 tok/s (+28%)** in an interleaved same-session A/B on a 4-core Xeon VM, with token-identical greedy output — the new number beats the original 2.74 headline above on a *weaker* host class. Micro-benchmark (`make bench-q2`): 1.32-1.68x per shape, largest on the 248320×5120 LM head.\n\n**Classifier precision (auto / BF16 / INT8 / INT4), at the confirmed-best 4 threads:**\n\n| Classifier | tok/s | Classifier storage | Notes |\n|---|---|---|---|\n| auto (default) | 0.59 | 322 MB, zero-copy raw Q2_0 | no materialization, no extra RAM |\n| BF16 (explicit) | 1.02 | 2.5 GB materialized | materialized, mid-pack this run |\n| INT8 (explicit) | 0.70 | 1.2 GB materialized | materialized, mid-pack this run |\n| INT4 (explicit) | 1.04 | 0.6 GB materialized | fastest this run |\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/qwen35_classifier_comparison.png\" width=\"640\" alt=\"Classifier precision comparison: auto vs BF16 vs INT8 vs INT4 tok/s on Ternary-Bonsai-27B\"\u003e\n\u003c/p\u003e\n\nScreenshots (with banner, 2026-07-17): [`classifier_auto.png`](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/classifier_auto.png) · [`classifier_bf16.png`](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/classifier_bf16.png) · [`classifier_int8.png`](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/classifier_int8.png) · [`classifier_int4.png`](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/classifier_int4.png)\n\n**Real bug, then a real fix:** the first pass at this data showed BF16/INT8/INT4 all measuring ~1.07 tok/s — identical, not just close. Root cause: `forward.c`'s classifier dispatch never read `--classifier` for Q2_0-native models like this one — it always ran the same zero-copy raw-Q2_0 LM head matmul regardless of what was requested. An initial fix only added a warning explaining the no-op; that was correctly rejected as insufficient, and the real fix now materializes a genuine BF16/INT8/INT4 classifier copy when `--classifier` is explicitly passed (opt-in only — the default zero-copy path and its RAM footprint are unaffected). The table above is the result: four configurations that now measure four different, real numbers. Two earlier back-to-back sweeps both ordered BF16 \u0026lt; auto \u0026asymp; INT8 \u0026lt; INT4 (1.13/1.19/2.60/2.62, then 1.21/1.27/1.30/1.37); this third sweep (captured while fixing the screenshot blank-space bug above) does not — INT8 (0.70) came in below BF16 (1.02) this time. Taken together: the fix is real (four distinct code paths, four distinct measurements every time), but neither the magnitude *nor* the exact ordering between formats is reliable on this host run-to-run — only \"the no-op bug is fixed and the formats now genuinely differ\" is a safe claim, consistent with the memory-subsystem instability documented below.\n\nThe result contains a genuine surprise: INT8/INT4 are **faster** than the zero-copy default, despite reading *more* bytes (raw Q2_0 at 2.125 bits/weight is the smallest of the four). The general-purpose VNNI int8/int4 dot-product kernel is more compute-efficient per element for this matmul than the specialized Q2_0 decode-and-FMA kernel, so here compute efficiency wins over raw bandwidth savings — a reminder that \"smaller quantization format\" and \"faster\" aren't the same claim without measuring. Full writeup in [`docs/ai/mistakes.md`](docs/ai/mistakes.md) and [`docs/ai/decision-log.md`](docs/ai/decision-log.md).\n\nScreenshots: [`classifier_bf16.png`](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/classifier_bf16.png) · [`classifier_int8.png`](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/classifier_int8.png) · [`classifier_int4.png`](benchmark_results/qwen35_ternary_bonsai_2026-07-16/screenshots/classifier_int4.png)\n\nA second real, previously-hidden bug was found and fixed while collecting the classifier data: every `--classifier` run initially crashed with `SIGILL` on this host. Root cause — this virtualized (Firecracker) host's CPUID advertises AVX-512VBMI support that the underlying execution unit cannot actually retire; both this build's compile-time detection and the engine's own runtime CPUID probe agreed VBMI was available, but executing a VBMI instruction faulted. Fixed with a one-time, execution-verified startup check (a SIGILL-trapped self-test) that replaces blind CPUID trust with real verification before any code path uses VBMI. Full writeup in [`docs/ai/mistakes.md`](docs/ai/mistakes.md).\n\nFull interactive write-up (live charts, hover tooltips, full input/output transcripts for every run): see the benchmark artifact linked from this repo's PR/session history.\n\n---\n\n\u003ca id=\"quick-start\"\u003e\u003c/a\u003e\n\n## Quick Start\n\n**Option A — pre-built binary (Linux x86-64, no compiler needed):**\n\n```bash\nwget https://github.com/shifulegend/project-zero/releases/download/v0.1.0/adaptive_ai_engine-0.1.0-x86_64-linux.tar.gz\ntar xf adaptive_ai_engine-0.1.0-x86_64-linux.tar.gz\n./adaptive_ai_engine --model models/bitnet-b1.58-2B-4T.bin \\\n  --tokenizer models/bitnet-b1.58-2B-4T_tokenizer_proper.bin \\\n  --prompt \"The capital of France is\"\n```\n\n**Option B — build from source (60 seconds):**\n\n```bash\ngit clone https://github.com/shifulegend/project-zero.git\ncd project-zero\nmake demo   # builds engine + downloads SmolLM2-135M + runs a test prompt\n```\n\nExpected output: `The capital of France is Paris.`\n\nNo GPU. No Python at runtime. No API key. GCC or Clang + `make` + `curl` — nothing else.\n\n---\n\n\u003ca id=\"help-wanted\"\u003e\u003c/a\u003e\n\n## Help Wanted\n\nTwo open problems where outside expertise would make a real difference:\n\n| Problem | Current state | Target |\n|---|---|---|\n| **MoE expert weight repacking** | DeepSeek-V2-Lite runs at 1.90 tok/s — 7× behind `llama.cpp`. Top-K expert weights sit at non-contiguous GGUF offsets: **~86% L3 cache miss rate per token**. Fix: repack selected expert weights into contiguous memory at load time, matching llama.cpp's interleaved layout. | ≥ 9 tok/s |\n| **Native Q4_K matmul kernel** | Current dense-model path dequants Q4_K → F32 before multiply. A fused mixed-precision kernel would close the remaining gap to `llama.cpp` on dense 4-bit GGUF models. | — |\n\nExisting SIMD work documented in [`docs/KERNEL_INTERNALS.md`](docs/KERNEL_INTERNALS.md).\nMoE repacking thread: [Discussion #1](https://github.com/shifulegend/project-zero/discussions/1)\n\n---\n\n## What It Does\n\nRuns [Microsoft's BitNet b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T) ternary weights and **dense GGUF transformers** (SmolLM2, DeepSeek-V2-Lite) on commodity CPUs — from scratch, in C.\n\nAlso included in the same binary: OpenAI-compatible HTTP API (`--server --port 8080`), persistent RAG memory (`--memory-db`), SigLIP vision pipeline (`--vision`), and an agentic tool-use loop (`/agent`).\n\nNo GPU required. Python is offline tooling only (model conversion, testing).\n\n\u003e ⚠️ **Before contributing:** read [`GOLDEN_RULES.md`](GOLDEN_RULES.md). No hardcoding. Test after every change.\n\n---\n\n## Hardware\n\nMemory bandwidth is the bottleneck — the engine reads 420–680 MB of weights per token. SIMD backend and thread count are auto-detected at startup.\n\n| RAM config | BitNet tok/s | Notes |\n|---|---|---|\n| 4 GB | ~8–10 | disable earlyoom |\n| 8 GB single-channel DDR4 | ~13 | bandwidth ceiling |\n| 16 GB dual-channel DDR4 | ~16 | measured (+24% over single-ch) |\n\nSIMD: AVX-512 VNNI → AVX2 → NEON → Scalar, selected at startup.\n\n---\n\n## Architecture\n\n```\nadaptive_ai_engine\n├── src/math/       AVX-512 VBMI ternary kernel, VNNI INT8/INT4, AVX2/NEON fallbacks\n├── src/core/       mmap weight loader (zero-copy), GGUF architecture-agnostic parser\n├── src/sampling/   top-p / temperature (static 200K buffer, no malloc per token)\n├── src/threading/  C11 atomic spinlock thread pool (no futex per dispatch)\n└── src/transformer/ forward pass, attention, FFN, RoPE, KV cache (int8-quantized)\n```\n\nKey design choices: `mmap` + `POSIX_MADV_WILLNEED` for weight loading, runtime SIMD dispatch via function pointers, sliding-window int8 KV cache for 131k context, BF16 embeddings (660 MB smaller vs F32, no precision loss).\n\n---\n\n## Build\n\n```bash\nmake release      # -O3 -march=native (default)\nmake debug        # ASan + UBSan\nmake test         # 3,367 assertions across all modules\nmake clean\n```\n\nRequirements: GCC or Clang, pthreads, libm. No other dependencies.\n\n---\n\n## CLI Reference\n\n```bash\n./adaptive_ai_engine \\\n  --model   models/bitnet-b1.58-2B-4T.bin \\\n  --tokenizer models/bitnet-b1.58-2B-4T_tokenizer_proper.bin \\\n  --prompt  \"Your prompt here\" \\\n  --threads 4 \\\n  --max-tokens 256\n```\n\nKey flags: `--temperature`, `--top-p`, `--seed`, `--classifier {bf16|int8|int4|auto}`, `--server --port 8080`, `--memory-db path.vrdb`, `--image photo.jpg --vision vision.bin --proj projector.bin`\n\nFull flag reference and REPL commands: run `./adaptive_ai_engine --help`\n\n---\n\n\u003ca id=\"ui-ux\"\u003e\u003c/a\u003e\n\n## UI/UX\n\n\u003e Full how-to (starting the server, every web UI control, REPL commands, CLI flags, API routes):\n\u003e [`docs/WEBUI_GUIDE.md`](docs/WEBUI_GUIDE.md).\n\n**Web chat UI** — a browser-based chat interface embedded directly in the binary (no separate\ninstall): streaming responses, adjustable sampling parameters, stop/cancel mid-generation, a\ndark/light theme, and image upload (when the server is started with `--vision`/`--proj`).\n\n```bash\n./adaptive_ai_engine --model models/smollm2.gguf --server --port 8080\n# open http://127.0.0.1:8080/ in a browser\n```\n\n| Light | Dark |\n|---|---|\n| ![Web UI — light theme](docs/design/screenshots/03-reply-light-2026-07-15T22-39-24-616Z.png) | ![Web UI — dark theme](docs/design/screenshots/05-dark-2026-07-15T22-39-24-616Z.png) |\n\n**CLI/REPL polish** — colored output (`--color auto\\|always\\|never`, respects `NO_COLOR`), a\nmodel-load progress indicator, a live tok/s status line during generation, and markdown/code\nrendering in the interactive REPL:\n\n![CLI REPL — color, markdown rendering, live tok/s](docs/design/screenshots/cli-repl.png)\n\n**Startup banner** — an animated ASCII-art \"PROJECT ZERO\" splash (bottom-up slide-in reveal, a\nhand-crafted 5-row block font, no external figlet dependency) that finishes with a brief\ndim/bold shimmer, shown for the REPL and `--server` mode and suppressed for scripted one-shot\n`--prompt` runs — TTY-gated, so no escape codes ever leak into piped/redirected output:\n\n![CLI startup banner — animated reveal and shimmer](docs/demo_banner_shimmer.gif)\n\nStatic frame, for reference:\n\n![CLI startup banner — animated ASCII-art \"PROJECT ZERO\"](docs/design/screenshots/06-cli-startup-banner-2026-07-16T01-03-52Z.png)\n\n**Live \"thinking\" spinner** — a continuously animated braille spinner (bold cyan), advancing\nonce per streamed token next to the live tok/s status line, the same idea as Claude Code's\nanimated indicator while it's actively working:\n\n![CLI live spinner during generation, plus the banner shimmer](docs/design/screenshots/07-cli-spinner-and-shimmer-2026-07-16T01-44-29Z.png)\n\n**HTTP API hardening** — CORS (`--cors`/`--cors-origin`), optional API-key auth (`--api-key`,\noff by default), Prometheus metrics (`--metrics` → `GET /metrics`), interactive docs\n(`GET /docs`, `GET /openapi.json`), and a cancel endpoint (`POST /v1/chat/completions/cancel`)\nthat actually stops an in-flight generation, backed by a concurrency rearchitecture (per-\nconnection threads + a generation mutex) so static/metrics/docs requests are never blocked\nbehind a running chat completion.\n\nDesign decisions are checked against a written [design-principles reference](docs/design/ui-ux-principles.md)\nbefore being accepted — see the [Phase 22.4 review](docs/design/review-2026-07-15.md) for the\nfull pass/fail breakdown (one real bug and one design-checklist violation were caught and fixed\nduring that review, not just cosmetic nits).\n\n---\n\n## Docs\n\n| Document | What it covers |\n|---|---|\n| [KERNEL_INTERNALS.md](docs/KERNEL_INTERNALS.md) | AVX-512 VBMI kernel, MoE scatter problem, thread pool design |\n| [PERFORMANCE_CEILING_REPORT.md](docs/PERFORMANCE_CEILING_REPORT.md) | Full optimization journal: 1.4 → 36.25 tok/s, bandwidth math |\n| [DEBUGGING_JOURNAL.md](docs/DEBUGGING_JOURNAL.md) | Root-cause log of every major perf regression and fix |\n| [ROADMAP.md](.github/ROADMAP.md) | Phase status (✅/🆘/❌), active blockers, planned phases |\n| [CONTRIBUTING.md](CONTRIBUTING.md) | Build, test, and contribution protocol |\n| [DEVELOPER_ONBOARDING.md](DEVELOPER_ONBOARDING.md) | Testing mandate, QA protocol, branching strategy |\n\n---\n\n*Phase 34+ · BitNet b1.58-2B-4T · DeepSeek-V2-Lite-Chat (GGUF) · SmolLM2-135M F16 · SigLIP vision*\n*Best: **35.79 tok/s** (BitNet INT4, i5-11300H, 500 tok, best-of-3) · **36.25 tok/s** (Xeon PGO+LTO) · **100.44 tok/s** (SmolLM2 F16) · **5.4× vs bitnet.cpp** (INT4 @ t=4) · 95% DRAM ceiling*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshifulegend%2Fproject-zero","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshifulegend%2Fproject-zero","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshifulegend%2Fproject-zero/lists"}