{"id":50782848,"url":"https://github.com/jcartu/qwen-bench","last_synced_at":"2026-06-12T05:02:03.468Z","repository":{"id":357190110,"uuid":"1231575454","full_name":"jcartu/qwen-bench","owner":"jcartu","description":"Hub for ongoing Qwen inference benchmarks on NVIDIA Blackwell. Indexes all studies, hosts the rolling SOTA leaderboard, points to the toolchain.","archived":false,"fork":false,"pushed_at":"2026-05-11T17:38:25.000Z","size":2091,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-11T19:28:49.044Z","etag":null,"topics":["benchmark","blackwell","hub","inference","leaderboard","qwen","qwen3","rtx-pro-6000","speculative-decoding","vllm"],"latest_commit_sha":null,"homepage":"https://github.com/jcartu/llm-stress-harness","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jcartu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-07T04:58:19.000Z","updated_at":"2026-05-11T17:38:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jcartu/qwen-bench","commit_stats":null,"previous_names":["jcartu/qwen-bench"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jcartu/qwen-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcartu%2Fqwen-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcartu%2Fqwen-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcartu%2Fqwen-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcartu%2Fqwen-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jcartu","download_url":"https://codeload.github.com/jcartu/qwen-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcartu%2Fqwen-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34229624,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","blackwell","hub","inference","leaderboard","qwen","qwen3","rtx-pro-6000","speculative-decoding","vllm"],"created_at":"2026-06-12T05:02:00.133Z","updated_at":"2026-06-12T05:02:03.458Z","avatar_url":"https://github.com/jcartu.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"docs/images/hub_hero.png\" alt=\"qwen-bench — empirical inference characterization of Qwen models on Blackwell\" width=\"100%\" /\u003e\n\n# `qwen-bench`\n\n### Empirical inference characterization of Qwen models on NVIDIA Blackwell\n\n[![Studies](https://img.shields.io/badge/studies-11_published-success?style=for-the-badge)](STUDIES.md)\n[![SOTA](https://img.shields.io/badge/SOTA-tracker-blue?style=for-the-badge)](SOTA.md)\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg?style=for-the-badge)](LICENSE)\n[![Topic: qwen](https://img.shields.io/badge/topic-qwen-orange?style=for-the-badge)](https://github.com/topics/qwen)\n[![Topic: vllm](https://img.shields.io/badge/topic-vllm-orange?style=for-the-badge)](https://github.com/topics/vllm)\n[![Topic: blackwell](https://img.shields.io/badge/topic-blackwell-orange?style=for-the-badge)](https://github.com/topics/blackwell)\n\n\u003e **An ongoing public benchmark series. Updated whenever new measurements are taken.**\n\n[Current SOTA](#current-sota) ·\n[Studies](#studies-chronological) ·\n[Tools](#tools) ·\n[How to read this](#how-to-read-this) ·\n[Methodology](#methodology) ·\n[Reproduce](#reproduce-a-result)\n\n\u003c/div\u003e\n\n---\n\n## What this is\n\nThis is the **index repository** for an ongoing series of inference benchmarks\nmeasuring [Qwen](https://github.com/QwenLM) models on consumer- and\nworkstation-class NVIDIA Blackwell hardware (`RTX PRO 6000 Blackwell`,\n`RTX 5090`, etc.) under realistic agent-style production load.\n\nEach *study* is a separately-versioned satellite repo with raw data, configs,\nlogs, and a written report. This hub repo provides:\n\n- 📊 **The current SOTA leaderboard** across all studies → [SOTA.md](SOTA.md)\n- 📚 **A chronological index** of every study → [STUDIES.md](STUDIES.md)\n- 📈 **Merged result CSVs** combining numbers from all studies → [`data/`](data/)\n- 🛠️ **Pointers to the toolchain** that produced these results\n\nThe hub is the front door. The studies are the substance. The tools are reusable\nprimitives. None of these collapse into the others; each has its own permanent,\ncitable URL.\n\n---\n\n## Current SOTA\n\n\u003e Last updated: **2026-05-15** · Hardware: **2× NVIDIA RTX PRO 6000 Blackwell** (TP=2, SM120, 96 GB each, PCIe Gen5 x16)\n\n### 🏆 Production-deployed config (2026-05-12): **`repne/vllm:v3` + FP8 + MTP=3**\n\nThe configuration currently live on production (`vllm-qwen36-27b-sota.service`,\npromoted 2026-05-12 ~10:03 MSK). 88.4 % HE / 89.1 % MBPP / 369 tok/s peak /\n98 tok/s single-user / 0 length-truncated. Reasoning routed cleanly into the\nOpenAI `reasoning` field; `content` clean. For single-user OpenCode coding, the current deployed client profile now adds `thinking_token_budget=2048`, eliminating runaway thinking loops while preserving thinking mode.\n\n**Benchmark-only — NOT deployable:** FP8+MTP=5 scored 93.3 % HE / 402 tok/s peak\nin the offline harness but leaks raw `\u003cthink\u003e...\u003c/think\u003e` blocks into the OpenAI\n`content` field on production traffic. The deployed MTP=3 config was validated\nleak-free across **420 trials** (300 plain-chat @ T=0.7 + 120 tool/function-calling @ T=0.7,\nscanning `content`, `tool_calls[*].function.name`, and `tool_calls[*].function.arguments`)\nby the permanent dual-mode leak probe (`harness/leak_probe.py`). See\n[v3 suite Production Incident](https://github.com/jcartu/qwen-bench-2026-05-12-v3-suite/blob/main/FINAL_REPORT.md#production-incident-2026-05-12-mtp--use_local_argmax_reduction-incompatibility)\nand [LEAK_DETECTION.md](https://github.com/jcartu/qwen-bench-2026-05-12-v3-suite/blob/main/LEAK_DETECTION.md).\n\n### Throughput records (aggregate tok/s)\n\n| Concurrency × Context | Tok/s | Config | Source |\n|-----------------------|------:|--------|--------|\n| c=1, ctx=0            | 117.1 | FP8+MTP=3 | [day1-sprint § exp 06](https://github.com/jcartu/qwen36-27b-blackwell-inference-study) |\n| c=4, ctx=131k         | 350.5 | FP8+MTP=3 | [day1-sprint § exp 06](https://github.com/jcartu/qwen36-27b-blackwell-inference-study) |\n| c=8, ctx=0            | 875.0 | FP8+MTP=3 | [day1-sprint § exp 08 X1](https://github.com/jcartu/qwen36-27b-blackwell-inference-study) |\n| c=16, ctx=0           | 1,520.6 | FP8+MTP=3 | [day1-sprint § exp 08 X1](https://github.com/jcartu/qwen36-27b-blackwell-inference-study) |\n| **c=32, ctx=0** ⭐    | **2,083.7** | FP8+MTP=3 | [day1-sprint § exp 08 X1](https://github.com/jcartu/qwen36-27b-blackwell-inference-study) |\n| c=4, ctx=131k (long)  | best | FP8+DFlash=8 | [stress-validation § 13](https://github.com/jcartu/qwen36-27b-blackwell-stress-validation) |\n\n### Correctness records (Qwen3.6-27B)\n\n| Benchmark | Pass rate | Config | Source |\n|-----------|----------:|--------|--------|\n| HumanEval (164) — corrected ⭐ | **95.7 %** (157/164) | FP8+MTP=5 (offline rescore) | [v2-followup ADDENDUM](https://github.com/jcartu/qwen-bench-2026-05-11-v2-followup/blob/main/ADDENDUM.md) |\n| HumanEval pass@5 (any of 5) | **96.95 %** (159/164) | FP8+MTP=3 mt=8192 on `:latest`, temp=0.8 | [v2-followup](https://github.com/jcartu/qwen-bench-2026-05-11-v2-followup) |\n| HumanEval (164) — offline harness, **`:v3`** | 93.3 % (153/164) | FP8+MTP=5 on `repne/vllm:v3`, mt=16384 ⚠️ benchmark-only (`\u003cthink\u003e` leaks into `content` in production) | [v3suite](https://github.com/jcartu/qwen-bench-2026-05-12-v3-suite) |\n| HumanEval (164) — BF16 best | **95.1 %** (156/164) | BF16+DFlash N=8 mt=8192 (offline rescore) | [v2-followup ADDENDUM](https://github.com/jcartu/qwen-bench-2026-05-11-v2-followup/blob/main/ADDENDUM.md) |\n| **HumanEval (164) — production-deployed SOTA** ⭐ | **88.4 %** (145/164) | **FP8+MTP=3 on `repne/vllm:v3`** — currently live | [v3suite](https://github.com/jcartu/qwen-bench-2026-05-12-v3-suite) |\n| MBPP (257) ⭐ | **91.1 %** (234/257) | BF16+DFlash N=8 on `repne/vllm:v3` | [v3suite](https://github.com/jcartu/qwen-bench-2026-05-12-v3-suite) |\n| MBPP (257) | 90.3 % (232/257) | BF16+DFlash N=8 @ mt=8192 on `:v2` | [v2-followup](https://github.com/jcartu/qwen-bench-2026-05-11-v2-followup) |\n| MBPP (257) | 89.5 % (230/257) | BF16+DFlash=8 (mt=4096) | [stress-validation](https://github.com/jcartu/qwen36-27b-blackwell-stress-validation) |\n\n\u003e ⚠️ **HumanEval scores corrected 2026-05-11.** A bug in the harness's `extract_code()` deflated every prior HE result by 13–23 pp. Original (pre-correction) numbers are preserved with strikethrough in [SOTA.md § 2.1](SOTA.md#21-humaneval-164-problems--corrected-2026-05-11). See the [ADDENDUM](https://github.com/jcartu/qwen-bench-2026-05-11-v2-followup/blob/main/ADDENDUM.md) for the full bug analysis and fix.\n\n### Quality vs BF16 reference\n\n| Quantization | KL-divergence vs BF16 | Top-token agreement | Source |\n|--------------|----------------------:|--------------------:|--------|\n| Q8_0 GGUF    | 0.001828 nats         | 97.9 %              | [day1-sprint § exp 07](https://github.com/jcartu/qwen36-27b-blackwell-inference-study) |\n| FP8 W8A8     | (indistinguishable)   | (per Qwen card)     | [day1-sprint § exp 07](https://github.com/jcartu/qwen36-27b-blackwell-inference-study) |\n\n\u003e See [SOTA.md](SOTA.md) for the complete cross-study record book.\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"docs/images/sota_leaderboard.png\" alt=\"SOTA leaderboard visualization\" width=\"80%\" /\u003e\n\u003c/div\u003e\n\n---\n\n## Studies (chronological)\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"docs/images/studies_index.png\" alt=\"Studies timeline\" width=\"100%\" /\u003e\n\u003c/div\u003e\n\nEach study is a self-contained satellite repo or hub-native production addendum. Studies are listed newest-first.\n\n### 📖 2026-05 · Thinking-budget generalization study\n**[`studies/2026-05-15-thinking-budget-generalization`](studies/2026-05-15-thinking-budget-generalization/)** · *cross-domain validation · GPQA Diamond / GSM-Plus 2k / MMLU-Pro 1.4k · 3,598 trials · GPQA budget sweep*\n\nRepne challenge: prove `thinking_token_budget=2048` generalizes beyond the morning's coding probes. Result: strict Pareto improvement on every benchmark. **GPQA +33.8 pp accuracy at 0.24× wall** (z=6.81, p=1e-11; 57.1% of unbounded responses never emit an answer because they bump into `finish_reason=length` inside `\u003cthink\u003e`). **GSM-Plus accuracy parity at 0.53× wall, 2.5× tighter p95 token tail.** **MMLU-Pro Δ +9.5 pp accuracy at 0.39× wall.** GPQA budget sweep at tb ∈ {1024, 2048, 4096, 8192} shows peak at tb=4096 → sweet spot exists.\n\n### 📖 2026-05 · Single-user thinking-budget addendum\n**[`studies/2026-05-15-single-user-thinking-budget`](studies/2026-05-15-single-user-thinking-budget/)** · *client-side hard thinking budget · 5-problem coding probe · c=1/2/4/8 fanout sweep*\n\nFollow-up to the v3 production rollout: the engine was correct, but unbounded Qwen3 thinking caused coding prompts to spiral into 50k-character reasoning tails. `thinking_token_budget=2048` kept thinking enabled, improved pass rate 70%→80%, eliminated stuck responses 10%→0%, and cut c=1 p95 latency 130.8s→19.6s.\n\n### 📖 2026-05 · BF16+DFlash parameter sweep on `repne/vllm:v2`\n**[`qwen-bench-2026-05-dflash-v2-sweep`](https://github.com/jcartu/qwen-bench-2026-05-dflash-v2-sweep)** · *13 configs · 195 cells · 421 quality problems · 7h total wall time*\n\nTwo-stage sweep on `Qwen3.6-27B` BF16+DFlash, TP=2: 3×3 grid of\n`--max-num-batched-tokens` × `--max-cudagraph-capture-size` (Stage A), then\n`num_speculative_tokens ∈ {4, 8, 15, 16}` at the winner (Stage B), plus a\ntwo-config Quality comparison establishing the c=8 noise floor.\n\n**Headline:** Buffer/graph axis is flat (Δ=3.6 %). Speculative-tokens axis is\ndecisive: n=4 → −62 %, n=8 → winner, n=16 → −42.6 % cliff. Confirms Repne's\npublished defaults (`batched=32768 capture=256 num_spec=8`) are optimal.\n\n### 📖 2026-05 · Stress-validation suite\n**[`qwen36-27b-blackwell-stress-validation`](https://github.com/jcartu/qwen36-27b-blackwell-stress-validation)** · *5 configs × 4 phases × 2,105 hard problems · Zero crashes*\n\nA controlled head-to-head of 5 speculative-decoding configurations\n(`FP8+MTP={3,5}`, `BF16+DFlash={7,8,15}`) on `Qwen3.6-27B` across functional\ngates, throughput matrix, HumanEval, and MBPP. Adds two addenda re-running\nDFlash variants without the deprecated `gumbel` flag and characterizing the\npreviously-untested FP8+DFlash pairing across N ∈ {7, 8, 15}.\n\n**Headline:** FP8+MTP=3 holds HumanEval pass-rate SOTA on this study (corrected: 92.1 %, was 79.3 % pre-harness-bug-fix); BF16+DFlash variants\nhold MBPP SOTA (89.5 %); FP8+DFlash N=8 wins long-context.\n\n### 📖 2026-05 · 24-hour inference study (Day 1 sprint)\n**[`qwen36-27b-blackwell-inference-study`](https://github.com/jcartu/qwen36-27b-blackwell-inference-study)** · *8 experiments · 1,200+ benchmark runs · 102K KLD probe*\n\nA systematic 24-hour sprint comparing the [`repne/vllm`](https://hub.docker.com/r/repne/vllm)\nfork against upstream vLLM v0.19.1 / v0.20.1, plus llama.cpp Q8_0 GGUF, across\nfour quantization schemes (BF16, FP8 W8A8, NVFP4, GGUF Q8_0) and three\nspeculative-decoding methods.\n\n**Headline:** Peak 2,083 tok/s at c=32. Q8 GGUF KLD vs BF16 = 0.0018 (noise floor).\n\n#### Satellite repos absorbed by this study\nThe following pairwise-comparison repos contain the same data as subsections\nof `inference-study`. They remain published as independent citable artifacts\nbut the canonical home is the parent study:\n\n- [`qwen36-27b-bf16-dflash-repne-vs-upstream`](https://github.com/jcartu/qwen36-27b-bf16-dflash-repne-vs-upstream) → exp 05\n- [`qwen36-27b-fp8-repne-vs-upstream`](https://github.com/jcartu/qwen36-27b-fp8-repne-vs-upstream) → exp 04\n- [`qwen36-27b-nvfp4-mtp-experiment`](https://github.com/jcartu/qwen36-27b-nvfp4-mtp-experiment) → exp 03\n- [`repne-dflash-newimage`](https://github.com/jcartu/repne-dflash-newimage) → image-version regression sub-experiment\n\n### 📖 2026-03 · Closing the Opus Gap (tool-calling study)\n**[`closing-the-opus-gap`](https://github.com/jcartu/closing-the-opus-gap)** · *3,500 API calls · 10 phases · Qwen3 235B and GLM-4.7 on Cerebras*\n\nThe earlier study that kicked this whole line of work off. Tool-calling\noptimization on wafer-scale hardware. Different model and different hardware\nclass, but methodologically the same family of work. Included here for context.\n\n\u003e See [STUDIES.md](STUDIES.md) for the full chronological index with abstracts.\n\n---\n\n## Tools\n\nAll studies above were instrumented with:\n\n### 🛠️ [`llm-stress-harness`](https://github.com/jcartu/llm-stress-harness)\nThe diagnostic toolkit — generic, model-agnostic, OpenAI-compatible:\n\n- `harness/stress_harness.py` — 322-line failure-taxonomic correctness probe\n- `orchestrator/four_phase_harness.sh` — end-to-end 4-phase validation runner\n- `launchers/launch_*.sh` — parametric vLLM launcher templates (FP8+MTP, FP8+DFlash, BF16+DFlash)\n- `utils/wait_vllm_ready.sh` — readiness probe with post-ready settle window\n\nThe toolkit lives in its own repo so it can grow to other models and stay\nreusable. Studies pin a specific commit of the toolkit for reproducibility.\n\n---\n\n## How to read this\n\n```\n                        ┌───────────────────────────┐\n                        │   you are here            │\n                        │   jcartu/qwen-bench       │\n                        │   (the hub / index)       │\n                        └────────┬──────────────────┘\n                                 │\n      ┌──────────────────────────┼──────────────────────────┐\n      │                          │                          │\n      ▼                          ▼                          ▼\n┌──────────────┐        ┌────────────────┐         ┌────────────────┐\n│  STUDIES     │        │  SOTA          │         │  data/         │\n│  list of all │        │  rolling       │         │  merged CSVs   │\n│  studies     │        │  leaderboard   │         │  (machine      │\n│              │        │                │         │   readable)    │\n└──────────────┘        └────────────────┘         └────────────────┘\n      │\n      ├──────► jcartu/qwen36-27b-blackwell-stress-validation\n      ├──────► jcartu/qwen36-27b-blackwell-inference-study\n      ├──────► jcartu/closing-the-opus-gap\n      └──────► (future studies land here)\n                              │\n                              └──── instrumented with ────► jcartu/llm-stress-harness\n```\n\n**Three-question reader flow:**\n\n1. *\"What's the best config for X?\"* → start at [SOTA.md](SOTA.md)\n2. *\"How was that measured?\"* → click through to the linked study repo\n3. *\"Can I reproduce this?\"* → see [`Reproduce a result`](#reproduce-a-result) below\n\n---\n\n## Methodology\n\nAll studies in this hub follow shared conventions so numbers compare across studies:\n\n| Aspect | Convention |\n|--------|-----------|\n| **Determinism** | `temperature=0.0`, `top_p=1.0`, `seed=42` in every payload |\n| **Hardware logging** | GPU model, driver, CUDA version, NCCL version, PCIe gen + lane count documented per study |\n| **Engine version** | Specific vLLM image SHA + git rev recorded in every study |\n| **Concurrency notation** | `c=N` means N concurrent in-flight requests; `ctx=K` means K-token prefix (in tokens, not chars) |\n| **Throughput definition** | `aggregate_tps = Σ completion_tokens / wall_time` (the user-observed number) |\n| **Failure classification** | Every request labeled with one of 7 failure modes — see [llm-stress-harness](https://github.com/jcartu/llm-stress-harness#the-failure-taxonomy) |\n| **N (re-runs)** | Phase 2 throughput cells re-run with N≥2 reseeded runs unless noted |\n\n### What we deliberately do *not* measure\n\n- **Pass@k for k\u003e1.** All correctness scores are single-shot at `t=0`.\n- **Prefill-only / decode-only throughput.** Use `vllm bench` for that. These studies are end-to-end.\n- **Reasoning-content quality.** We log `reasoning_chars` for diagnostics, not for grading.\n- **Cross-model leaderboards.** This hub is *Qwen-only* by design. Cross-model is out of scope.\n\n---\n\n## Reproduce a result\n\n```bash\n# 1. Read the study's METHODOLOGY section\ngh repo view jcartu/qwen36-27b-blackwell-stress-validation --web\n\n# 2. Pull the toolkit at the version the study used\ngit clone https://github.com/jcartu/llm-stress-harness\ncd llm-stress-harness \u0026\u0026 git checkout \u003ccommit-sha-from-study\u003e\n\n# 3. Launch the documented config\nNUM_SPEC=3 ./launchers/launch_fp8_mtp.sh   # or whichever the study used\n\n# 4. Run the same orchestrator\n./orchestrator/four_phase_harness.sh \u003cconfig-label\u003e \u003ckv-budget\u003e ./out/\n\n# 5. Diff your numbers against the published CSV\npython3 -c \"import pandas as pd; print(pd.read_csv('your_results.csv').compare(pd.read_csv('published.csv')))\"\n```\n\nNumbers should land within **±2 %** at the same hardware tier. If not, file an\nissue against the study repo with both CSVs.\n\n---\n\n## Adding a new study\n\nWhen you publish a new benchmark study:\n\n1. **Name the new study repo** `qwen-bench-{YYYY-MM}-{slug}` (convention applies to studies created **2026-05 onward**; earlier study repos keep their original names — their URLs are stable forever)\n2. **Add a `← qwen-bench hub` badge** to the new study's README (line 1)\n3. **Set GitHub topics:** `qwen`, `qwen3`, `vllm`, `blackwell`, `benchmark`, plus study-specific tags\n4. **Open a PR against this hub** that:\n   - Adds an entry to [`STUDIES.md`](STUDIES.md) with abstract + headline\n   - Updates [`SOTA.md`](SOTA.md) if any record was broken\n   - Drops the study's `master.csv` or hub-native addendum CSV into [`data/{YYYY-MM}-{slug}.csv`](data/)\n   - Bumps the `studies-N_published` shield count at the top of this README\n\nFull per-study checklist, slug guidelines, and URL-stability rationale: **[CONTRIBUTING.md](CONTRIBUTING.md)**\n\nA future automated script (planned) will compute SOTA-record diffs from\nthe merged CSV and prevent regressions from being missed.\n\n---\n\n## Acknowledgments\n\nThe work indexed here builds on:\n\n- [Qwen team](https://github.com/QwenLM) — open-weight model lineage\n- [vLLM](https://github.com/vllm-project/vllm) — primary inference engine\n- [Repne](https://hub.docker.com/r/repne/vllm) — vLLM fork with DFlash + advanced spec-decode support\n- [HumanEval](https://github.com/openai/human-eval), [MBPP](https://github.com/google-research/google-research/tree/master/mbpp), [EvalPlus](https://github.com/evalplus/evalplus) — benchmark sources\n- [AesSedai](https://github.com/AesSedai/llama.cpp/tree/perplexity-sliding-window) — sliding-window perplexity branch used in KLD probe\n\nHub illustrations generated with **Gemini 3.1 nano banana** (`gemini-3.1-flash-image-preview`).\n\n---\n\n## License\n\nMIT. See [LICENSE](LICENSE).\n\nEach satellite study repo is independently licensed; check the study's own LICENSE\nfile.\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n**Maintained by [Josh Cartu](https://github.com/jcartu) · RASPUTIN AI Research Lab**\n\nIf you cite numbers from this hub, please link to the **specific study repo**\n(not just this hub) so readers can verify the methodology.\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcartu%2Fqwen-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjcartu%2Fqwen-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcartu%2Fqwen-bench/lists"}