{"id":51184866,"url":"https://github.com/mlelarge/quartz","last_synced_at":"2026-06-27T09:30:46.250Z","repository":{"id":366442225,"uuid":"1275932169","full_name":"mlelarge/quartz","owner":"mlelarge","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-21T20:16:39.000Z","size":150,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-21T22:08:23.544Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlelarge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-21T10:25:02.000Z","updated_at":"2026-06-21T20:16:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mlelarge/quartz","commit_stats":null,"previous_names":["mlelarge/quartz"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/mlelarge/quartz","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlelarge%2Fquartz","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlelarge%2Fquartz/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlelarge%2Fquartz/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlelarge%2Fquartz/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlelarge","download_url":"https://codeload.github.com/mlelarge/quartz/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlelarge%2Fquartz/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34848932,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-27T02:00:06.362Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-27T09:30:44.964Z","updated_at":"2026-06-27T09:30:46.240Z","avatar_url":"https://github.com/mlelarge.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# quartz\n\nA transparent, **portable CPU** LLM inference engine in **numpy + Numba** — the\nCPU sibling of [`silica`](../silica) (which targets Apple Silicon / MLX). Same\ntransparent, single-stream design; the bottleneck and the levers move.\n\n\u003e **Status: M0–M2 + M4 done.** Dense Qwen3 + Llama decode in fp32 (parity-exact vs an\n\u003e independent HuggingFace fp32 oracle, cosine 1.000000); the fused int8 kernel +\n\u003e size-adaptive hybrid dispatcher (**4.25×** on the lm_head, ~1.2× end-to-end, int8 PPL\n\u003e cost +0.1%); a measured figure-of-merit; an external yardstick vs `llama.cpp` CPU\n\u003e (quartz is **bandwidth-competitive (~74 GB/s, == llama.cpp)** but **~3× slower**, from\n\u003e moving more bytes); and a **fused int8-body kernel** making all-int8 the fastest config\n\u003e (**1.17× over hybrid, 1.43× over fp32**), to the numba ceiling. See\n\u003e **[results-fused.md](docs/results-fused.md)**, **[results-cross-engine.md](docs/results-cross-engine.md)**,\n\u003e **[results-cpu.md](docs/results-cpu.md)**, **[results-int8-quality.md](docs/results-int8-quality.md)**.\n\n## The thesis\n\nOn the GPU (`silica`), decode is memory-bandwidth-bound and **quantization is the\nlever**. On the CPU that inverts *naively* — but a measured spike shows it\nrecovers with the right kernel:\n\n- **fp32 is the compute dtype.** fp16 GEMM is ~32× slower (Accelerate doesn't\n  vectorize it); int8 is a *storage/bandwidth* lever, not a compute dtype.\n- **A *fused* int8 GEMV** (read int8 weights, dequant **in-register**, fp32 FMA —\n  never materialize an fp32 weight array) is **~4× faster** than fp32 BLAS on the\n  large, memory-bound lm_head (cos 0.99997), because it is DRAM-bandwidth-bound on\n  4× fewer bytes.\n- **But int8 loses on small cache-resident matmuls** (BLAS wins), so \"int8\n  everywhere\" is *slower* than fp32. The engine's core decision is a\n  **size-adaptive, per-matmul hybrid**: fp32 BLAS for the body, fused-int8 for the\n  lm_head — measured **~41 vs ~34 tok/s (1.21×)** end-to-end on an M3 Max.\n- **The win is capped by bytes, not Python overhead.** Decode is bandwidth-bound:\n  quartz fp32/hybrid hit the *same* **~74 GB/s as hand-tuned llama.cpp** (so the kernels\n  are competitive), and the ~1.2× ceiling is that the body stays fp32 (3× the bytes) — an\n  all-int8 body is *dispatch*-bound in numpy/numba. Net: quartz is **~0.32× llama.cpp Q8_0\n  on CPU (it's ~3.1× faster)**, a gap that is *entirely* bytes-moved. See\n  **[docs/results-cross-engine.md](docs/results-cross-engine.md)** and [results-cpu.md](docs/results-cpu.md).\n\n## Layout\n\n```\nquartz/\n  config.py       typed config (ModelConfig from HF config.json, Quant/Gen/Bench)\n  dtypes.py       safetensors load + bf16-\u003efp32 upcast (safetensors.numpy can't read bf16)\n  ops.py          fp32 primitives (rmsnorm/silu/softmax/linear) + leaf modules\n  rope.py         half-split RoPE (traditional=False) + llama3 freq scaling\n  attention.py    GQA einsum SDPA + offset-aware causal mask\n  cache.py        growing fp KV cache (preallocate in step chunks)\n  sample.py       greedy + temp/top-k/top-p/min-p (per-sampler numpy RNG)\n  generate.py     chat template -\u003e prefill -\u003e synchronous decode loop -\u003e streamed text\n  weights.py      safetensors load + dotted-key bind onto the model tree\n  models/         per-architecture files + registry (qwen3, llama)\n  kernels/        fused int8 GEMV/GEMM + size-adaptive dispatcher (M1)\nbench/            byte+FLOP roofline; decode tok/s \u0026 achieved-BW; crossover bench (M1/M2)\ntests/            pure-numpy unit tests + an HF-fp32 parity gate\n```\n\n## Setup ([uv](https://docs.astral.sh/uv/))\n\n```bash\ncd quartz\nuv venv\nuv pip install -e \".[reference,dev]\"     # numpy/numba/... + torch (HF fp32 oracle)\n```\n\n## Run\n\n```bash\nuv run python -m quartz.generate --model Qwen/Qwen3-0.6B --prompt \"Explain RoPE in one sentence.\"\n```\n\n## Test\n\n```bash\nuv run pytest -m \"not oracle\"                                       # pure numpy, no checkpoint\nQUARTZ_PARITY_MODEL=Qwen/Qwen3-0.6B uv run pytest tests/test_parity.py -m oracle\n```\n\n## License\n\n[MIT](LICENSE) (code). Model weights carry their own licenses — never committed here.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlelarge%2Fquartz","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlelarge%2Fquartz","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlelarge%2Fquartz/lists"}