{"id":52040229,"url":"https://github.com/Neroued/ninfer","last_synced_at":"2026-08-03T05:00:55.545Z","repository":{"id":372077294,"uuid":"1281510950","full_name":"Neroued/ninfer","owner":"Neroued","description":"High-performance single-GPU inference for selected model checkpoints and GPUs.","archived":false,"fork":false,"pushed_at":"2026-07-26T06:17:45.000Z","size":9815,"stargazers_count":89,"open_issues_count":0,"forks_count":13,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-07-26T08:10:57.103Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Neroued.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-06-26T16:20:43.000Z","updated_at":"2026-07-25T16:05:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Neroued/ninfer","commit_stats":null,"previous_names":["neroued/ninfer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Neroued/ninfer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neroued%2Fninfer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neroued%2Fninfer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neroued%2Fninfer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neroued%2Fninfer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Neroued","download_url":"https://codeload.github.com/Neroued/ninfer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neroued%2Fninfer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36218648,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-08-03T02:00:06.975Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-08-01T23:00:27.538Z","updated_at":"2026-08-03T05:00:55.538Z","avatar_url":"https://github.com/Neroued.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"# NInfer\n\n\u003e Selected checkpoints. Maximum single-GPU inference performance.\n\nNInfer is a from-scratch C++/CUDA inference engine for two exact Qwen3.6 checkpoints on a single\nNVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a local CLI or\nOpenAI-/Anthropic-compatible HTTP APIs.\n\nNInfer deliberately supports a closed set of model artifacts instead of acting as a general model\nruntime:\n\n| Model | Weights | NInfer artifact | Size | SHA-256 |\n|---|---|---|---:|---|\n| [Qwen3.6-27B](https://huggingface.co/neroued/Qwen3.6-27B-NInfer) | `groupwise-int` | `qwen3_6_27b.ninfer` | 17,495,365,888 bytes (16.29 GiB) | `7b51600ffd10632b9660f56085efdd9b751d79733ad32036a652234b64bebe7b` |\n| [Qwen3.6-27B NVFP4](https://huggingface.co/neroued/Qwen3.6-27B-nvfp4-NInfer) | `nvfp4` | `qwen3_6_27b_nvfp4.ninfer` | 18,324,064,000 bytes (17.07 GiB) | `bce5f00d066c0f20f1317bf1fdcb458264cf95837c3b1f3fbec163694627893a` |\n| [Qwen3.6-35B-A3B](https://huggingface.co/neroued/Qwen3.6-35B-A3B-NInfer) | `groupwise-int` | `qwen3_6_35b_a3b.ninfer` | 22,783,246,080 bytes (21.22 GiB) | `1fb9ea0b5b8561e49d9604115ec89e5d9f2b6f6434e32c37c57fffd480a325d2` |\n\nBoth 27B artifacts bind to the same registered `qwen3_6_27b` target; the version-2 artifact identity\nselects the weight profile without a separate target or runtime flag. The `nvfp4` profile uses W4A4\nTensor Core MMA for prefill and A16 NVFP4 kernels for decode while retaining the same Text, Vision,\nMTP, prefix-reuse, CLI, and serving paths.\n\n## Performance\n\nServing performance was measured on an RTX 5090 with INT8 group-64 KV cache, CUDA Graphs, a 1,024-\ntoken prefill chunk, and a maximum context of 262,144 tokens. Each reported fixture uses five fixed\nseeds after one warm-up. The two registered targets are reported independently and are not\ncross-target comparisons. The two 27B weight profiles are reported separately.\n\n**Qwen3.6-35B-A3B**\n\n- MTP0 at a 7,680-token prompt: **15,544.3 prefill tok/s** and **271.1 decode tok/s**.\n- MTP0 at a 260,096-token prompt: **5,157.1 prefill tok/s** and **188.2 decode tok/s**.\n- MTP3 long reasoning: **584.0–695.1 decode tok/s** with **72.4–83.3% acceptance**.\n- MTP3 structured output: **714.3 decode tok/s**, **87.7% acceptance**, and **3.63 tokens/round**.\n\n**Qwen3.6-27B (`groupwise-int`)**\n\n- MTP0 at a 7,680-token prompt: **3,218.1 prefill tok/s** and **77.6 decode tok/s**.\n- MTP0 at a 260,096-token prompt: **1,614.8 prefill tok/s** and **54.8 decode tok/s**.\n- MTP3 long reasoning: **161.9–175.4 decode tok/s** with **73.4–78.8% acceptance**.\n- MTP3 structured output: **193.0 decode tok/s**, **88.7% acceptance**, and **3.66 tokens/round**.\n\n**Qwen3.6-27B (`nvfp4`)**\n\n- MTP0 at a 7,680-token prompt: **11,191.5 prefill tok/s** and **86.4 decode tok/s**.\n- MTP0 at a 260,096-token prompt: **2,510.6 prefill tok/s** and **59.9 decode tok/s**.\n- MTP3 long reasoning: **201.6–222.7 decode tok/s** with **74.7–80.8% acceptance**.\n- MTP3 structured output: **243.1 decode tok/s**, **90.2% acceptance**, and **3.71 tokens/round**.\n- Against groupwise-int on the same corpus and runtime options: **3.48× the 7,680-token prefill\n  throughput**, **1.55× the 260,096-token prefill throughput**, and **25–28% higher MTP3 decode\n  throughput**.\n\nSee [Performance](docs/performance.md) for the full methodology, variability, reproduction command,\nand per-fixture results.\n\n## Evaluation\n\nCapability scores were measured through NInfer's OpenAI-compatible serving route with thinking\nenabled, MTP=3, and EvalScope 1.9.0 (0-shot, rule scoring, one sample per problem):\n\n| Model profile | AIME 2025 | AIME 2026 | GPQA-Diamond |\n|---|---:|---:|---:|\n| [Qwen3.6-27B groupwise-int](model-cards/Qwen3.6-27B-NInfer/README.md) | 86.67% | 93.33% | 86.87% |\n| [Qwen3.6-27B NVFP4](model-cards/Qwen3.6-27B-nvfp4-NInfer/README.md) | 93.33% | 93.33% | 84.34% |\n| [Qwen3.6-35B-A3B groupwise-int](model-cards/Qwen3.6-35B-A3B-NInfer/README.md) | 90.00% | 90.00% | 85.35% |\n\nThese are single-sample results under that NInfer evaluation profile, not pass@k. See the model\ncards and [full performance document](docs/performance.md) for correct/total counts and evaluation\nnotes.\n\n## Requirements\n\nNInfer currently requires:\n\n- 64-bit Linux;\n- NVIDIA GeForce RTX 5090 (`sm_120a`);\n- NVIDIA driver support for CUDA 13.1 and the CUDA Toolkit 13.1 or newer;\n- CMake 3.28 or newer and a C++20-capable host compiler;\n- `pkg-config`;\n- FFmpeg development libraries: `libavformat \u003e= 60`, `libavcodec \u003e= 60`,\n  `libavutil \u003e= 58`, and `libswscale \u003e= 7`;\n- `libcurl \u003e= 7.85`;\n- Ninja, when using the commands below.\n\nThe build rejects CUDA architectures other than `120a`. There is no install target or packaged\nbinary distribution; NInfer is run from its source build tree.\n\n## Build\n\n```bash\ngit clone https://github.com/Neroued/ninfer.git\ncd ninfer\n\ncmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release\ncmake --build build --parallel\n```\n\nThe default configuration builds:\n\n```text\nbuild/apps/ninfer\nbuild/apps/ninfer-serve\n```\n\nTests, benchmarks, and maintainer tools are excluded from the default build.\n\n## Docker\n\nBuild the runtime image on a 64-bit Linux host with an RTX 5090, a CUDA 13.1-compatible NVIDIA\ndriver, Docker, and the\n[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).\n\n```bash\ndocker build --tag ninfer:local .\n```\n\nDownload a model into `models/` as described below, then run the HTTP server:\n\n```bash\ndocker run --rm \\\n  --gpus '\"device=0\"' \\\n  --publish 8080:8080 \\\n  --volume \"$PWD/models:/models:ro\" \\\n  ninfer:local \\\n  ninfer-serve /models/qwen3_6_27b.ninfer \\\n  --host 0.0.0.0 \\\n  --model-id qwen3.6-27b\n```\n\nRun the CLI from the same image:\n\n```bash\ndocker run --rm \\\n  --gpus '\"device=0\"' \\\n  --volume \"$PWD/models:/models:ro\" \\\n  ninfer:local \\\n  ninfer /models/qwen3_6_27b.ninfer \\\n  --prompt \"Explain prefill and decode in three sentences.\" \\\n  --max-new 256\n```\n\n## Download a model\n\nUse the Hugging Face CLI to download one of the registered artifacts:\n\n```bash\nhf download neroued/Qwen3.6-27B-NInfer \\\n  qwen3_6_27b.ninfer \\\n  --local-dir models\n\n# Or the 27B NVFP4 weight variant:\nhf download neroued/Qwen3.6-27B-nvfp4-NInfer \\\n  qwen3_6_27b_nvfp4.ninfer \\\n  --local-dir models\n\n# Or:\nhf download neroued/Qwen3.6-35B-A3B-NInfer \\\n  qwen3_6_35b_a3b.ninfer \\\n  --local-dir models\n```\n\nCurrent NInfer builds accept only the version-2 artifact container, and all three downloads above\nare version 2. If an older 27B or 35B-A3B artifact was downloaded before this publication, migrate\nthat exact local file in place:\n\n```bash\npython3 -m tools.artifact.migrate_v1_to_v2 models/qwen3_6_27b.ninfer\n```\n\nUse the same command with `qwen3_6_35b_a3b.ninfer` for the 35B-A3B artifact. The migration updates\nonly container metadata; it does not rewrite the weight payload. Alternatively, download the\ncurrent version-2 file again from its Hugging Face repository.\n\nEach `.ninfer` file contains the weights and frontend resources needed by NInfer. It is not a\nTransformers checkpoint, Safetensors distribution, or GGUF file.\n\nEach artifact is complete, while GPU residency is fixed at process startup. Speculative decoding is\ndisabled by default, so MTP/DFlash state and the optimized proposal head are not uploaded.\nVision is also disabled by default, so its weights, Vision scratch phase, and frozen\nrequest-transient allocation are omitted. Add `--vision` to the CLI or server process that must\naccept image or video input. Disabled capabilities cannot be enabled by a later request. DFlash is\navailable only for the 35B-A3B target and is text-only.\n\n## Run the CLI\n\n```bash\n./build/apps/ninfer models/qwen3_6_27b.ninfer \\\n  --prompt \"Explain prefill and decode in three sentences.\" \\\n  --max-context 16384 \\\n  --max-new 256 \\\n  --spec mtp --draft-tokens 3 \\\n  --lm-head-draft\n```\n\nUse `--messages FILE` instead of `--prompt` for chat history, images, or videos:\n\n```bash\n./build/apps/ninfer models/qwen3_6_27b.ninfer \\\n  --messages examples/cli/messages/image_chart.json \\\n  --max-context 8192 \\\n  --max-new 128 \\\n  --vision\n```\n\nAnswer content is written to stdout. Loading progress, reasoning, timing, throughput, memory, and\nspeculative-decoding statistics are written to stderr. See the [CLI guide](docs/cli.md) and\n[committed examples](examples/cli/) for structured input and runtime options.\n\n## Run the HTTP server\n\n```bash\n./build/apps/ninfer-serve models/qwen3_6_27b.ninfer \\\n  --model-id qwen3.6-27b \\\n  --max-context 16384 \\\n  --spec mtp --draft-tokens 3 \\\n  --lm-head-draft\n```\n\nThen send an OpenAI-style request:\n\n```bash\ncurl http://127.0.0.1:8080/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"model\": \"qwen3.6-27b\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Reply with one short sentence.\"}],\n    \"max_tokens\": 64\n  }'\n```\n\nThe server also implements Anthropic Messages, streaming, token counting, multimodal input, and\nfunction-tool request/response translation. See [HTTP serving](docs/serving.md).\n\n## Capabilities\n\nBoth registered model targets support:\n\n- text generation with thinking and non-thinking prompt modes;\n- image, multi-image, video, and mixed multimodal messages;\n- chunked prefill and CUDA Graph decode;\n- MTP speculative decoding with draft windows from one to five;\n- BF16 and INT8 group-64 KV cache;\n- greedy, temperature, top-k, top-p, min-p, and presence/frequency-penalty sampling;\n- compatible-prefix reuse;\n- OpenAI Chat Completions and Anthropic Messages, including streaming and usage accounting;\n- prompt-rendered function tools and parsed tool calls.\n\n## Current limits\n\n- Only the two model targets listed above are accepted product targets.\n- Execution is specialized for one RTX 5090 and one CUDA device.\n- One Engine owns one resident sequence and runs one active request at a time.\n- Continuous batching, multi-GPU execution, CPU/GPU offload, and distributed serving are not\n  implemented.\n- Context capacity is configurable up to the registered models' native 262,144-token limit, subject\n  to GPU memory and KV-cache configuration.\n- Tool calls are parsed and returned to the client; NInfer does not execute tools.\n- The C++ headers are used by the in-tree applications and are not distributed as an installed SDK.\n\n## Documentation\n\n- [Contributing](CONTRIBUTING.md)\n- [Documentation index](docs/README.md)\n- [CLI](docs/cli.md)\n- [HTTP serving](docs/serving.md)\n- [Performance](docs/performance.md)\n- [CLI examples](examples/cli/)\n\n## License\n\nNInfer is licensed under the [Apache License 2.0](LICENSE).\n\nThe published artifacts are derived from\n[Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) and\n[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B). The 27B NVFP4 artifact also\nuses the fixed packed weights from\n[rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm](https://huggingface.co/rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm).\nThese source repositories are distributed under Apache-2.0. Vendored dependencies retain their own\nlicense files under `third_party/`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNeroued%2Fninfer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNeroued%2Fninfer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNeroued%2Fninfer/lists"}