{"id":51184858,"url":"https://github.com/mlelarge/silica","last_synced_at":"2026-06-27T09:30:43.648Z","repository":{"id":365736134,"uuid":"1273185628","full_name":"mlelarge/silica","owner":"mlelarge","description":"A transparent single-stream LLM inference engine for Apple Silicon (MLX) — Qwen3 decode, quantization, custom-kernel roofline. Audited \u0026 benchmarked vs mlx-lm and llama.cpp.","archived":false,"fork":false,"pushed_at":"2026-06-18T17:07:12.000Z","size":261,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-18T17:29:20.617Z","etag":null,"topics":["apple-silicon","inference-engine","llm-inference","metal","mlx","mlx-lm","quantization","qwen"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlelarge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-18T09:31:51.000Z","updated_at":"2026-06-18T17:07:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mlelarge/silica","commit_stats":null,"previous_names":["mlelarge/silica"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/mlelarge/silica","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlelarge%2Fsilica","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlelarge%2Fsilica/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlelarge%2Fsilica/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlelarge%2Fsilica/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlelarge","download_url":"https://codeload.github.com/mlelarge/silica/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlelarge%2Fsilica/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34848932,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-27T02:00:06.362Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apple-silicon","inference-engine","llm-inference","metal","mlx","mlx-lm","quantization","qwen"],"created_at":"2026-06-27T09:30:41.153Z","updated_at":"2026-06-27T09:30:43.639Z","avatar_url":"https://github.com/mlelarge.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# silica\n\nA transparent, single-stream LLM inference engine for Apple Silicon, built on\nMLX. Inverted from `mini-sglang`: on a Mac the bottleneck is **memory bandwidth\nand quantization**, not GPU scheduling. ~835 lines of engine, audited and\nbenchmarked against `mlx-lm` and `llama.cpp`.\n\n\u003e **Status: v0 complete and audited.** M0–M2 validated on device, M3 (custom\n\u003e kernels) gated out by evidence, M4 written up; an 8-reviewer code audit is\n\u003e fully resolved; two architecture families (Qwen3 + Llama) pass exact parity.\n\n## Results\n\nOn Apple M3 Max (40-core, 400 GB/s), Qwen3-0.6B / Llama-3.2-1B:\n\n| Dimension | Result |\n|---|---|\n| **Correctness** | parity gate vs `mlx-lm` **+ an independent HuggingFace fp32 oracle**; 69-test suite |\n| **Models** | dense Qwen3 + Llama **and MoE (OLMoE, Qwen3-MoE)**, registry-dispatched; exact next-token parity vs `mlx-lm` |\n| **MoE roofline** | **Qwen3-30B-A3B 4-bit decodes 110 tok/s** at batch=1 (reads ~3B active/token) — ~7× a dense 30B; 51% of usable BW |\n| **Pre-quantized load** | loads mlx-community 4/8-bit checkpoints directly (e.g. 30B-A3B in ~17 GB, not ~60 GB fp) |\n| **Quantization** | 8-bit ~lossless; 4-bit/g64 +17.8% PPL, **4-bit/g32 +8.0%** (vs llama.cpp Q4_K_M +6.2%) |\n| **Decode perf** | silica **== `mlx-lm`** within ~1.5% (~70% of usable bandwidth); `async_eval` +50% at 4-bit; `mx.compile` neutral |\n| **vs `llama.cpp`** | silica ≈ **0.89×** decode speed — ~12% behind hand-tuned C++/Metal |\n| **Custom kernels (M3)** | **declined** — the ~30% gap to the ceiling is real but scale-independent and already defined by Apple's `mx.quantized_matmul` |\n\nThe headline lever is quantization (the bandwidth denominator); the headline\nnon-finding is that, once `async_eval` hides per-step dispatch, there's little\nleft for a transparent engine to win over Apple's tuned kernels.\n\n**Docs:** [PLAN.md](PLAN.md) (design + roadmap) · [docs/REPORT.md](docs/REPORT.md)\n(performance \u0026 correctness scoreboard) · [docs/READING_GUIDE.md](docs/READING_GUIDE.md)\n(annotated code tour + lessons) · [docs/AUDIT.md](docs/AUDIT.md) (pre-build plan audit) ·\n[docs/CODE_AUDIT.md](docs/CODE_AUDIT.md) (post-build code audit) · results:\n[m1 quant](docs/results-m1.md) · [m2 perf](docs/results-m2-baseline.md) ·\n[m4 cross-engine](docs/results-m4-cross-engine.md) · [generality](docs/results-generality.md) ·\n[MoE](docs/results-moe.md).\n\n## Layout\n\n```\nsilica/\n  config.py       typed config (ModelConfig from HF config.json, Quant/Gen/Bench)\n  weights.py      load safetensors (single + sharded), selective quantize, registry dispatch\n  models/         per-architecture model files + a registry (SGLang-style)\n    common.py     shared layers: MLP, SwitchGLU/MoEBlock, Decoder, CausalLM, build_rope\n    qwen3.py llama.py        dense (Qwen3 QK-Norm; Llama llama3-RoPE)\n    olmoe.py qwen3_moe.py    MoE (router + gathered experts via mx.gather_qmm)\n    __init__.py   REGISTRY: HF `architectures` field -\u003e model class\n  cache.py        growing + quantized KV cache; PrefixCache (single-stream prefix reuse)\n  attention.py    sdpa() — fp -\u003e mx.fast SDPA; quantized KV -\u003e quantized_matmul path\n  sample.py       greedy + temp/top-k/top-p/min-p (per-sampler RNG key, no global seed)\n  detokenize.py   incremental UTF-8-safe BPE detok + stop sequences + flush\n  generate.py     chat template -\u003e prefill -\u003e decode loop (async_eval) -\u003e streamed text\n  compiled.py     M2 mx.compile decode experiment (correct, perf-neutral)\n  kernels/        custom Metal fusions (M3, gated out by evidence — empty by design)\nbench/            decode tok/s + achieved-bandwidth %; quant-quality PPL; cross-engine vs llama.cpp\ntests/            pure-python (config, roofline, sampler, detok, cache, ppl) + device parity gates\n```\n\n**Supported models:** dense **Qwen3** and **Llama-3.x / SmolLM2**, plus\n**Mixture-of-Experts** (**OLMoE**, **Qwen3-MoE** like Qwen3-30B-A3B) — dispatched\nfrom the checkpoint's `architectures` field. Adding one is a small attention\nblock + registry entry; the entire runtime and bench harness are reused unchanged.\nQwen3, Llama, OLMoE, and Qwen3-30B-A3B each pass an exact next-token parity gate vs\n`mlx-lm` ([generality](docs/results-generality.md), [MoE](docs/results-moe.md)). MoE\nis the on-thesis case: Qwen3-30B-A3B holds 30B weights but reads only its ~3B active\nexperts — **110 tok/s at batch=1** (~7× a dense 30B), loaded pre-quantized in ~17 GB.\n\n## Setup (Apple Silicon, [uv](https://docs.astral.sh/uv/))\n\n```bash\ncd silica\nuv venv                                  # creates .venv (honors requires-python)\nuv pip install -e \".[reference,dev]\"     # mlx, transformers, ... + mlx-lm oracle\n```\n\n`uv run \u003ccmd\u003e` auto-uses the project venv, so you can skip `source .venv/bin/activate`.\n\n## Run (on device)\n\n```bash\n# Greedy generation — works for either architecture\nuv run silica-generate --model Qwen/Qwen3-0.6B            --prompt \"Explain RoPE in one sentence.\"\nuv run silica-generate --model unsloth/Llama-3.2-1B-Instruct --prompt \"Name two planets.\"\n\n# Decode benchmark — bandwidth-% needs the chip's RATED bandwidth (M3 Max = 300 OR 400)\nuv run silica-bench --model Qwen/Qwen3-0.6B --tokens 128 --context-len 4096 --bandwidth 400 --chip \"M3 Max 40c\"\n\n# Quantization-quality perplexity ablation (fp16 / 8-bit / 4-bit)\nuv run silica-ppl --ablate --model Qwen/Qwen3-0.6B\n```\n\n## Test\n\n```bash\n# Pure-python (no MLX needed): config, roofline byte model, sampler, detok, ppl windowing\nuv run pytest tests/test_config.py tests/test_roofline.py tests/test_detokenize.py\n\n# Full device gate (needs MLX + mlx-lm + a checkpoint) — Qwen3 by default\nSILICA_PARITY_MODEL=Qwen/Qwen3-0.6B uv run pytest -m device\n\n# Llama generality gate downloads the ungated unsloth/Llama-3.2-1B-Instruct mirror\n```\n\n## Design highlights (what the audits hardened)\n\nA pre-build **plan audit** and a post-build **code audit** (both 8-reviewer,\nfindings adversarially verified) shaped the engine:\n\n- **Architecture details that break parity** — per-head QK-Norm before RoPE, no\n  QKV bias, decoupled `head_dim`, tied `lm_head`, the right RoPE θ / llama3\n  scaling (`silica/models/`).\n- **A real output path** — incremental UTF-8-safe detokenizer with a `finalize()`\n  flush, EOS/EOT + string stop-sequences, chat template (`detokenize.py`, `generate.py`).\n- **Two oracles** — `mlx-lm` (same-backend regression guard) **and** an\n  independent HuggingFace fp32 CPU oracle (the non-circular correctness check).\n- **Honest figure of merit** — bytes/token = weights *(incl. per-group\n  scales+biases)* + KV(context) + lm_head, over the recorded chip SKU's bandwidth\n  (`bench/roofline.py`); a weight double-count and a context mismatch were caught\n  by *running* (achieved BW exceeded the chip's physical peak).\n- **Per-sampler RNG** — a fixed seed makes one run reproducible without clobbering\n  global state (`sample.py`).\n\n## Caveats\n\n- **MLX is Metal-only**, so the device test gate doesn't run on hosted CI —\n  run it locally or on a self-hosted Apple-Silicon runner.\n- Under unavoidable machine load, absolute bandwidth numbers are contention-\n  limited; the benches report **relative/interleaved ratios** (which cancel the\n  noise) and flag contaminated runs.\n- Quantized KV and rotating KV do **not** compose in mainline MLX.\n- The Llama parity is vs `mlx-lm` (an independent implementation, but same\n  `mx.fast` kernels); an HF fp32 oracle for Llama is left as future work.\n\n## License\n\n[MIT](LICENSE) (code). Model weights carry their own licenses — never committed here.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlelarge%2Fsilica","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlelarge%2Fsilica","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlelarge%2Fsilica/lists"}