{"id":50555013,"url":"https://github.com/asad-ismail/ternary-models","last_synced_at":"2026-06-04T06:02:51.391Z","repository":{"id":351408777,"uuid":"1210869576","full_name":"Asad-Ismail/ternary-models","owner":"Asad-Ismail","description":"Pre-quantized ternary models for VLMs, multimodal, and audio — the models GGUF can't touch","archived":false,"fork":false,"pushed_at":"2026-04-14T21:59:17.000Z","size":25,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-14T23:10:23.376Z","etag":null,"topics":["apple-silicon","edge-ai","gemma","huggingface","model-compression","multimodal","quantization","qwen","ternary-quantization","vision-language-model","vlm","whisper"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Asad-Ismail.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-14T20:53:52.000Z","updated_at":"2026-04-14T21:59:21.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Asad-Ismail/ternary-models","commit_stats":null,"previous_names":["asad-ismail/ternary-models"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/Asad-Ismail/ternary-models","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asad-Ismail%2Fternary-models","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asad-Ismail%2Fternary-models/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asad-Ismail%2Fternary-models/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asad-Ismail%2Fternary-models/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Asad-Ismail","download_url":"https://codeload.github.com/Asad-Ismail/ternary-models/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asad-Ismail%2Fternary-models/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33891733,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-04T02:00:06.755Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apple-silicon","edge-ai","gemma","huggingface","model-compression","multimodal","quantization","qwen","ternary-quantization","vision-language-model","vlm","whisper"],"created_at":"2026-06-04T06:02:48.474Z","updated_at":"2026-06-04T06:02:51.384Z","avatar_url":"https://github.com/Asad-Ismail.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ternary-models\n\nPre-quantized models produced with\n[ternary-quant](https://github.com/Asad-Ismail/ternary-quant) and published on\n[Hugging Face](https://huggingface.co/AsadIsmail).\n\nThis repo is the model-release companion to `ternary-quant`. The library handles\npost-training ternary quantization and inference; this repo tracks release\nmetadata, model cards, reproduction scripts, validation notes, and the Gradio\nplayground.\n\n## Positioning\n\nGGUF/llama.cpp and Unsloth are strong defaults for local text-LLM deployment,\nand llama.cpp supports selected multimodal models through `libmtmd`.\n\n`ternary-quant` targets a different workflow:\n\n- Hugging Face/Python-native post-training ternary quantization.\n- Component-aware quantization for mixed architectures such as VLMs, audio\n  models, seq2seq models, and diffusion/video transformers.\n- Explicit control over which components are quantized and which stay in\n  higher precision, for example keeping a vision encoder in FP16 while\n  quantizing a text backbone or decoder.\n- Packed ternary runtime modes for CUDA/Triton and Apple Silicon Metal, plus a\n  fast `cached` mode that dequantizes once at load.\n\nThe claim is not that this beats GGUF/Unsloth for standard decoder-only LLMs.\nFor plain local text generation, GGUF/Unsloth will usually be the more mature\ndeployment path. The value here is component-aware ternary PTQ for model\nfamilies that are awkward to handle with text-LLM-first quantization tooling.\n\n## Hub Collection\n\nAs of 2026-04-27, the Hugging Face collection contains these public\nternary/ternary-derived releases:\n\n| Model | Task | Base model | Notes |\n|---|---|---|---|\n| [Qwen3-1.7B-ternary](https://huggingface.co/AsadIsmail/Qwen3-1.7B-ternary) | Text generation | `Qwen/Qwen3-1.7B` | Early `ternary-quant` text model. |\n| [Qwen2-VL-2B-ternary](https://huggingface.co/AsadIsmail/Qwen2-VL-2B-ternary) | Image-text-to-text | `Qwen/Qwen2-VL-2B-Instruct` | VLM checkpoint. |\n| [Gemma4-E2B-ternary](https://huggingface.co/AsadIsmail/Gemma4-E2B-ternary) | Image-text-to-text | `google/gemma-4-E2B-it` | Early Gemma 4 multimodal checkpoint. |\n| [Qwen2.5-VL-7B-Instruct-ternary](https://huggingface.co/AsadIsmail/Qwen2.5-VL-7B-Instruct-ternary) | Image-text-to-text | `Qwen/Qwen2.5-VL-7B-Instruct` | VLM checkpoint. |\n| [gemma-4-E4B-it-ternary](https://huggingface.co/AsadIsmail/gemma-4-E4B-it-ternary) | Image-text-to-text | `google/gemma-4-E4B-it` | Gemma 4 multimodal checkpoint. |\n| [SmolVLM2-2.2B-Instruct-ternary](https://huggingface.co/AsadIsmail/SmolVLM2-2.2B-Instruct-ternary) | Image-text-to-text | `HuggingFaceTB/SmolVLM2-2.2B-Instruct` | Compact VLM checkpoint. |\n| [whisper-large-v3-ternary](https://huggingface.co/AsadIsmail/whisper-large-v3-ternary) | Automatic speech recognition | `openai/whisper-large-v3` | Decoder-focused Whisper quantization. |\n| [gemma-4-31B-it-ternary](https://huggingface.co/AsadIsmail/gemma-4-31B-it-ternary) | Image-text-to-text | `google/gemma-4-31B-it` | Large Gemma 4 multimodal checkpoint. |\n| [CogVideoX-2b-ternary](https://huggingface.co/AsadIsmail/CogVideoX-2b-ternary) | Text-to-video | `zai-org/CogVideoX-2b` | Diffusers-compatible video model artifact. |\n| [CogVideoX-5b-ternary](https://huggingface.co/AsadIsmail/CogVideoX-5b-ternary) | Text-to-video | `zai-org/CogVideoX-5b` | Diffusers-compatible video model artifact. |\n| [SmolVLM2-500M-Video-Instruct-ternary](https://huggingface.co/AsadIsmail/SmolVLM2-500M-Video-Instruct-ternary) | Image-text-to-text | `HuggingFaceTB/SmolVLM2-500M-Video-Instruct` | Small video-capable VLM. |\n| [SmolVLM2-256M-Video-Instruct-ternary](https://huggingface.co/AsadIsmail/SmolVLM2-256M-Video-Instruct-ternary) | Image-text-to-text | `HuggingFaceTB/SmolVLM2-256M-Video-Instruct` | Edge-oriented video-capable VLM. |\n| [whisper-medium-ternary](https://huggingface.co/AsadIsmail/whisper-medium-ternary) | Automatic speech recognition | `openai/whisper-medium` | Whisper ASR checkpoint. |\n| [whisper-small-ternary](https://huggingface.co/AsadIsmail/whisper-small-ternary) | Automatic speech recognition | `openai/whisper-small` | Smaller Whisper ASR checkpoint. |\n| [Wan2.1-T2V-1.3B-ternary](https://huggingface.co/AsadIsmail/Wan2.1-T2V-1.3B-ternary) | Text-to-video | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` | Diffusers-compatible video model artifact. |\n| [Wan2.2-TI2V-5B-ternary](https://huggingface.co/AsadIsmail/Wan2.2-TI2V-5B-ternary) | Text/image-to-video | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | Diffusers-compatible video model artifact. |\n\nThe local repo currently contains reproduction assets for a subset of the Hub\ncollection: Qwen2.5-VL-7B, Gemma 4 E4B, Gemma 4 31B, SmolVLM2-2.2B, and\nWhisper-large-v3. Some newer Hub releases may not yet have local model-card\nfolders or scripts in this repository.\n\n## What Is Actually Quantized?\n\n`ternary-quant` is component-first. Depending on the architecture and script,\nit may quantize only selected components rather than every parameter in the\nmodel.\n\nExamples:\n\n- VLMs: quantize the language backbone and/or multimodal connector while\n  preserving sensitive vision components when needed.\n- Whisper/audio: quantize selected encoder or decoder components.\n- Video diffusion transformers: publish Diffusers-compatible artifacts; check\n  the individual model card for whether packed ternary storage or dequantized\n  compatibility storage is used.\n\nDo not assume a table-level compression number applies uniformly to every\nparameter. For exact size, quantized components, validation notes, and runtime\nrequirements, use the individual Hugging Face model card.\n\n## Quick Start\n\n```bash\npip install ternary-quant\n```\n\n```python\nfrom ternary_quant.inference import load_ternary_model, generate_text\n\nmodel, tokenizer = load_ternary_model(\n    \"AsadIsmail/Qwen3-1.7B-ternary\",\n    device=\"auto\",\n    runtime_mode=\"cached\",\n)\n\noutput = generate_text(\n    model,\n    tokenizer,\n    \"Describe what component-aware quantization means.\",\n    max_new_tokens=100,\n)\nprint(output)\n```\n\nFor image, audio, and video models, follow the usage block on the specific\nmodel card. Processor/tokenizer behavior differs by architecture.\n\n## Reproducing Releases\n\nScripts in [`scripts/`](scripts/) capture the commands used for the local\nrelease subset. Example:\n\n```bash\npip install ternary-quant\n\nternary-quant quantize-broad google/gemma-4-E4B-it \\\n    --output ./gemma-4-E4B-it-ternary \\\n    --components text_backbone multimodal_connector \\\n    --scheme tritplane3 \\\n    --dtype float16 \\\n    --eval \\\n    --push-to-hub AsadIsmail/gemma-4-E4B-it-ternary\n```\n\nReproduction is model-dependent. Larger VLMs and video models need more RAM,\nmay require model-license access, and may need architecture-specific component\nselection.\n\n## Runtime Modes\n\n| Runtime | Hardware | Behavior |\n|---|---|---|\n| `cached` | CPU, NVIDIA, Apple Silicon | Dequantizes selected weights at load. Usually fastest and simplest. |\n| `metal` | Apple Silicon | Uses packed ternary Metal kernels where supported. |\n| `triton_memory` | NVIDIA GPU | Uses packed ternary CUDA/Triton path to reduce peak VRAM where supported. |\n| `gemlite` | NVIDIA GPU | Uses packed low-bit GEMM/GEMV path where supported by the installed stack. |\n\nRuntime support depends on model family, installed `ternary-quant` version,\nhardware, and optional dependencies.\n\n## Comparison With GGUF / Unsloth\n\n| Area | GGUF / Unsloth | `ternary-quant` / this repo |\n|---|---|---|\n| Standard text LLM deployment | Mature, fast, broad local ecosystem. | Supported, but not the main advantage. |\n| Selected VLM deployment | Supported in llama.cpp through `libmtmd` for supported models. | HF-native component-aware PTQ path for selected VLMs. |\n| Audio / ASR | llama.cpp has experimental multimodal audio support; coverage depends on model support. | Whisper checkpoints are published; component choices are model-specific. |\n| Quantization type | Mixed GGUF low-bit formats, including dynamic layer/model-specific schemes. | Ternary PTQ: grouped asymmetric ternary and TritPlane-style multi-plane ternary. |\n| Component control | Deployment-oriented; component control depends on converter/runtime support. | Explicit component selection such as text backbone, decoder, encoder, connector, or vision backbone. |\n| Best use case | Local deployment of supported models. | Research and release workflow for component-aware ternary quantization in the HF/Python stack. |\n\n## Validation Scope\n\n`ternary-quant` reports 95.4-99.2% FP16 quality retention on its measured\ndecoder-LLM lm-eval suite. That number should not be read as a universal claim\nfor every VLM, ASR, or video model in this repository.\n\nFor multimodal and audio releases, validation varies by model card and may\ninclude reconstruction error, smoke-test prompts, coherent-output checks, or\ntask-specific examples. Formal benchmark coverage is still model-dependent.\n\n## License\n\nRepository code and metadata are Apache 2.0. Each model artifact follows the\nlicense and terms of its base model.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasad-ismail%2Fternary-models","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasad-ismail%2Fternary-models","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasad-ismail%2Fternary-models/lists"}