{"id":51631353,"url":"https://github.com/orkait/orka.py","last_synced_at":"2026-07-13T07:32:30.727Z","repository":{"id":370621595,"uuid":"1224206105","full_name":"orkait/orka.py","owner":"orkait","description":"Vector-quantization compiler for LLM weights. Fits per-tensor RVQ codebooks, stores indices as bit-planes (~2 bits/weight), recovers quality with QAT, and exports to GGUF or vLLM.","archived":false,"fork":false,"pushed_at":"2026-07-09T21:29:44.000Z","size":2507,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-09T23:11:00.654Z","etag":null,"topics":["compression","gguf","inference","kmeans","large-language-models","llm","machine-learning","model-compression","python","pytorch","qat","quantization","rvq","vector-quantization","vllm"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orkait.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-29T04:00:43.000Z","updated_at":"2026-07-09T21:21:53.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/orkait/orka.py","commit_stats":null,"previous_names":["orkait/orka.py"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/orkait/orka.py","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orkait%2Forka.py","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orkait%2Forka.py/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orkait%2Forka.py/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orkait%2Forka.py/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orkait","download_url":"https://codeload.github.com/orkait/orka.py/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orkait%2Forka.py/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35414732,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-13T02:00:06.543Z","response_time":119,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","gguf","inference","kmeans","large-language-models","llm","machine-learning","model-compression","python","pytorch","qat","quantization","rvq","vector-quantization","vllm"],"created_at":"2026-07-13T07:32:30.625Z","updated_at":"2026-07-13T07:32:30.698Z","avatar_url":"https://github.com/orkait.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# 🐙 orka\n\n**Squish LLM weights down to ~2 bits each with vector quantization, then run them.**\n\nA residual-vector-quantization compressor for transformer weights: fit per-tensor codebooks, store indices as bit-planes, recover quality with QAT, and export to GGUF or vLLM. No magic, just codebooks and honest engineering.\n\n[![python](https://img.shields.io/badge/python-3.10+-3776AB?logo=python\u0026logoColor=white)](https://www.python.org)\n[![torch](https://img.shields.io/badge/PyTorch-optional-EE4C2C?logo=pytorch\u0026logoColor=white)](https://pytorch.org)\n[![format](https://img.shields.io/badge/format-RVQ_+_12--bit_bit--planes-4f8ff7)](#-how-it-works)\n[![bits](https://img.shields.io/badge/weights-~2_bits-7c3aed)](#-honest-benchmarks)\n[![ci](https://github.com/orkait/orka.py/actions/workflows/ci.yml/badge.svg)](https://github.com/orkait/orka.py/actions/workflows/ci.yml)\n[![license](https://img.shields.io/badge/license-Apache_2.0-green)](#-license)\n\n\u003c/div\u003e\n\n---\n\n## 🧭 What this actually is\n\norka takes a model's linear weights and replaces each little group of 8 numbers with **an index into a learned codebook**. Store the tiny indices instead of the fat weights, and the file shrinks to roughly **2 bits per weight** (8x smaller than fp16). Then, because raw 2-bit quantization is rough, it can **fine-tune the codebooks back toward the original** (QAT) so the model stays smart.\n\nThat's the whole trick. It is not a new number format, not a kernel-fusion miracle, not \"lossless.\" It's residual vector quantization (RVQ) applied carefully, with the boring-but-important parts (Hessian weighting, bit allocation, scale refinement, QAT) done properly.\n\n## ⚡ Quick start\n\n```bash\n# core install: no torch. The numpy backend is the deterministic reference path.\npip install orka-compiler\n\n# GPU packing and HF model loading are extras\npip install 'orka-compiler[torch,hf]'\n\n# 1. compress a model -\u003e a .orka artifact (~2 bits/weight)\norka pack  ./SmolLM-135M  --out model.orka \\\n    --quant-mode rvq-12-4 --normalization block-max --device cuda\n\n# 2. check what it cost you (perplexity vs the original)\norka verify  model.orka\n\n# 3. decode back to safetensors, or export to a runtime\norka reconstruct  model.orka --out recon.safetensors --format safetensors\norka export-vllm   model.orka --out ./model-vllm\n```\n\nWant orka to pick the settings for you?\n\n```bash\n# autoquant's --target is a KL budget against fp16, not a bit-width\norka autoquant ./model --objective min-bits --target 0.05 --out alloc.json\n\n# for an explicit bit-width target, allocate is the command with --target-bpw\norka allocate ./model --target-bpw 4.0 --out alloc.json\n```\n\n`python -m orka \u003ccommand\u003e` works identically if you prefer not to rely on the console script.\n\n## 🔬 How it works\n\n```\nweight group [8 values]\n  ──fit────▶  per-tensor codebook (e.g. 4096 entries)        ← k-means, Hessian-weighted\n  ──encode─▶  nearest-entry index per group                  ← ~2 bits/weight\n  ──store──▶  indices as 12-bit BIT-PLANES (lo byte + packed hi)\n  ──QAT────▶  fine-tune codebooks/weights back toward fp16    ← optional, recovers quality\n  ──decode─▶  scale x sum(codebook[index]) (+ sparse correction)\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e📐 The pieces, and where they live\u003c/summary\u003e\n\n| Subpackage | Job |\n|---|---|\n| `orka.core` | private primitives - format, tensor, checkpointing |\n| `orka.codebook` | k-means / k-means‖ codebook fitting |\n| `orka.quant` | quant specs, bit **allocation** (water-filling), activation calibration |\n| `orka.transforms` | normalization + rotations before VQ |\n| `orka.pipeline` | the **pack** orchestration |\n| `orka.inference` | the inference-time `VQLinear` + Triton/CUDA kernels |\n| `orka.qat` | quantization-aware training (QATVQLinear, distillation) |\n| `orka.autoquant` | probe, policy and escalation loop behind `orka autoquant` |\n| `orka.artifact` | `.orka` ops - reconstruct, export, gguf, merge, correct |\n| `orka.eval` | metrics, verify, sweeps, reports |\n| `orka.integrations` | HF quantizer + vLLM hooks |\n| `orka.deploy` | Kaggle / Modal bootstrap helpers |\n| `orka.config` | the environment knobs, resolved in one place |\n\nFull map: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).\n\u003c/details\u003e\n\n## 📊 Honest benchmarks\n\nReal numbers, real caveats. Perplexity on wikitext-2, SmolLM-135M. Lower is better. We benchmarked against llama.cpp's k-quants because pretending the competition doesn't exist is how you fool yourself.\n\n| method | bits/wt | ppl | vs fp16 | verdict |\n|---|--:|--:|--:|---|\n| fp16 | 16 | ~23.7 | - | the original |\n| orka strong PTQ (vq-16) | 2.0* | ~30 | +28% | near-fp16, but big codebooks eat the savings |\n| **orka planar + QAT** | 2.0 | ~42 | +79% | small + deployable, but not the quality leader |\n| **llama.cpp Q2_K** | ~2.6 | ~23 | +23% | mature k-quants are good; on tiny models they win |\n| llama.cpp Q4_K_M | ~4.5 | ~20 | +9% | more bits, better quality |\n\n`*` \"2-bit indices\" but per-tensor codebooks add overhead - on a 135M model a `vq-16` artifact is barely smaller than fp16. The honestly-compressed orka config is the **planar** one (~5x smaller on disk).\n\n\u003cdetails\u003e\n\u003csummary\u003eHow these numbers were measured, and what is missing\u003c/summary\u003e\n\nSliding-window (non-overlapping, 512-token context) perplexity over the wikitext-2 `test` split, fp32 forward, one 12GB consumer GPU. The method is the `_wikitext_ppl` helper in `deploy/kaggle/orka_qat_hi05b_kaggle.py`.\n\n**There is no single committed command that regenerates this table.** `orka eval` measures prompt-loss perplexity, not the wikitext-2 sweep. Treat the numbers as a record of what we measured, not as a reproducible benchmark, until a first-class harness lands. If you reproduce different numbers, we would rather hear it than not.\n\u003c/details\u003e\n\n### 🫡 The honest takeaways\n\n- **2 bits per weight, near-fp16 quality is real** with the full machinery (Hessian weighting + allocation + QAT) - on the right config.\n- **On a 135M model, mature k-quants (Q2_K) still beat orka's small-file config on quality-per-byte.** We are not ahead here, and we will not pretend we are.\n- **Codebook overhead amortizes with model size.** Tiny models are orka's worst case (the embedding table alone can be 20%+ of params). The interesting territory is larger models - that's where we expect VQ to pull ahead.\n- **More QAT levers is not always better.** Training the full weights on a small calibration set *overfit* and did worse than light codebook-only tuning. Measured, not guessed.\n\n## 🛠️ CLI\n\n```\npack          compress a model -\u003e .orka          allocate    water-fill bits per tensor\nverify        decode + measure error             reconstruct .orka -\u003e safetensors/json\ndistill       QAT fine-tune                       export-vllm .orka -\u003e vLLM dir\ncorrect       add low-rank/sparse correction     merge-orka  combine artifacts\nautoquant     auto-pick the config               eval/sweep  benchmark configs\n```\n\n`orka \u003ccommand\u003e --help` for the details.\n\n## ⚙️ Environment knobs\n\nAll of these resolve through [`orka/config.py`](orka/config.py).\n\n| Env var | Default | Effect |\n|---|---|---|\n| `ORKA_KMEANS_FAISS` | off | `=1` swaps the unweighted CUDA k-means for faiss GPU Lloyd (~2x faster at equal reconstruction MSE, byte-deterministic per seed). Requires `pip install faiss-gpu-cu12`; falls back to the built-in torch path if faiss is missing or the tensor uses Hessian/importance weighting. Off by default so packs stay reproducible regardless of whether faiss is installed. Truthy values: `1`, `true`, `yes`. |\n| `ORKA_ENABLE_AWQ` | off | `=1` enables the legacy AWQ path, which needs external calibration data. Truthy values: `1`, `true`, `yes`, `on`. |\n| `ORKA_HARD_CEILING_GB` | `25.0` | Upper bound on the RAM cap, whatever the CLI asks for. Raise it on machines larger than 32GB. |\n| `ORKA_PREFLIGHT_MIN_AVAIL_GB` | `5.0` | Refuse to start if `MemAvailable` is below `workload_budget + this`. |\n| `ORKA_PREFLIGHT_MAX_SWAP_GB` | `4.0` | Refuse to start if swap in use exceeds this (the system is already thrashing-prone). |\n| `ORKA_KMEANS_ITERS` | caller's value | Override the Lloyd iteration count, for quick validation runs. |\n| `ORKA_ZLIB_LEVEL` | `6` | zlib level for index/sidecar streams (clamped 0-9). Decode is level-agnostic, so `=1` trades a few percent of artifact size for ~4x faster compression on big packs. Recommended for giant-tensor (9B-class) packs. |\n| `ORKA_PREFETCH_BUDGET_GB` | `4.0` | Byte budget for producer read-ahead in streamed per-tensor packs. Caps the RAM the prefetched candidates retain; a candidate bigger than the budget is admitted alone, so vocab-width giants serialize instead of stacking. `\u003c=0` disables. |\n| `ORKA_ASSIGN_CHUNK_MB` | `128` | H2D transfer size per chunk in the tiled giant-tensor assign. Indices are identical for any chunk size; this only trades transfer count against per-chunk VRAM. |\n| `ORKA_LARGE_ASSIGN_ROWS` | `20000000` | Giant-tensor threshold, in vectors of width 8 (compared as elements, `rows*8`). Lower it only to exercise the giant code path on small models in tests. |\n\n## 🧪 Development\n\n```bash\npip install -e '.[dev,torch,hf]'\npre-commit install\n\npytest -q                            # full suite\npytest tests/test_golden_oracle.py   # the structural pack gate\n```\n\n`pack_checkpoint` output is **not byte-reproducible** (codebook bytes move under threaded BLAS), so the invariant we protect is structural: `tests/test_golden_oracle.py` packs a seeded model through 12 configurations and hashes a fingerprint of each manifest. Any change that moves the combined hash changed pack behaviour. See [CONTRIBUTING.md](CONTRIBUTING.md).\n\nThe package is organized by domain (see the table above). Old flat import paths (`orka.hf`, `orka.reconstruct`, ...) still work via compat shims while callers migrate to the canonical ones.\n\n## 📄 License\n\n[Apache License 2.0](LICENSE).\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\u003csub\u003eMade with codebooks, a constrained GPU, and a refusal to fudge the benchmark numbers.\u003c/sub\u003e\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forkait%2Forka.py","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forkait%2Forka.py","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forkait%2Forka.py/lists"}