{"id":50873689,"url":"https://github.com/hinanohart/nsa-eval","last_synced_at":"2026-06-15T07:31:22.928Z","repository":{"id":358506932,"uuid":"1241551362","full_name":"hinanohart/nsa-eval","owner":"hinanohart","description":"MPS-native Native Sparse Attention reference implementation + unified long-context evaluation suite (CUDA / MPS / CPU)","archived":false,"fork":false,"pushed_at":"2026-06-10T10:37:40.000Z","size":417,"stargazers_count":0,"open_issues_count":90,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-10T11:07:56.643Z","etag":null,"topics":["apple-silicon","benchmark","deepseek","long-context","mps","pytorch","ruler","sparse-attention"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hinanohart.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-17T14:31:25.000Z","updated_at":"2026-06-10T10:37:31.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hinanohart/nsa-eval","commit_stats":null,"previous_names":["hinanohart/nsa-eval"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/hinanohart/nsa-eval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hinanohart%2Fnsa-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hinanohart%2Fnsa-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hinanohart%2Fnsa-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hinanohart%2Fnsa-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hinanohart","download_url":"https://codeload.github.com/hinanohart/nsa-eval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hinanohart%2Fnsa-eval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34353189,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-15T02:00:07.085Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apple-silicon","benchmark","deepseek","long-context","mps","pytorch","ruler","sparse-attention"],"created_at":"2026-06-15T07:31:20.175Z","updated_at":"2026-06-15T07:31:22.920Z","avatar_url":"https://github.com/hinanohart.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nsa-eval\n\n[![CI](https://github.com/hinanohart/nsa-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/hinanohart/nsa-eval/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)\n\n**MPS-native Native Sparse Attention (NSA) reference implementation + unified long-context evaluation suite.**\n\nA vanilla-PyTorch NSA reference that dispatches onto Metal kernels on Apple Silicon (MPS), with a cross-device (CUDA / MPS / CPU) benchmark runner and reproducible Kaggle notebooks. To the maintainer's knowledge there is no other OSS NSA implementation that targets MPS as a first-class device — please open an issue if you find a counterexample. The project runs with minimum human intervention through v0.1: a weekly Claude Code Routine drives the issue board, and a Mac contributor is needed only for MPS validation.\n\n## Architecture\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"docs/architecture.png\" alt=\"nsa-eval architecture\" width=\"760\"\u003e\n\u003c/div\u003e\n\n## Why\n\n- DeepSeek NSA ([arXiv 2502.11089](https://arxiv.org/abs/2502.11089), ACL 2025 Best Paper) ships as a CUDA/Triton implementation. There is no published path for Apple Silicon users to run it locally.\n- Sparse attention research papers use inconsistent benchmark setups, which makes head-to-head comparison and reproduction hard.\n- This repo provides three things:\n  1. An MPS-native NSA reference implementation, with a Linux/CUDA path that mirrors it cell-for-cell.\n  2. A single runner for RULER / LongBench / Needle-in-a-Haystack / AgentBench against a fixed set of small open LLMs.\n  3. Kaggle notebooks (T4 / P100 / TPU v3-8) and Apple Silicon scripts that reproduce every reported number end-to-end.\n\n## Quickstart\n\n### Kaggle T4 (zero local setup)\n\nOpen `notebooks/_template_kaggle.ipynb` on Kaggle and click **Run All**. The notebook self-installs everything, downloads the configured model, and writes results to `benchmarks/results/\u003cdate\u003e/`.\n\n### Mac — Apple Silicon (MPS)\n\n```bash\nuv venv \u0026\u0026 source .venv/bin/activate\nuv pip install -e .\nnsa-eval run --config configs/config.yaml\n```\n\nTo change the attention backend, model, benchmark, or device, point `--config` at a different YAML file (compose from `configs/attention/`, `configs/model/`, etc.):\n\n```bash\nnsa-eval run --config configs/config.yaml   # uses YAML defaults\nnsa-eval list-backends                       # show registered attention backends + benchmarks\n```\n\n### Linux / CUDA\n\n```bash\nuv pip install -e \".[nsa_cuda]\"\n# vendor the fla-org/native-sparse-attention submodule first:\ngit submodule update --init --recursive\nnsa-eval run --config configs/config.yaml\n```\n\n## How it works\n\n1. **Config** — every run is fully described by a YAML file (`configs/config.yaml` composes model + benchmark + attention + device sub-configs from `configs/`).\n2. **Registry** — `BenchmarkRunner` looks up attention backends and benchmarks by name from registries populated at import time. Third-party plugins can register their own implementations without modifying core files.\n3. **EvalSpec** — a Pydantic model that captures the four-tuple `(model, benchmark, attention, device)` plus a random seed. Its `.slug()` method names the output JSON.\n4. **Results** — each run writes a JSON file to `benchmarks/results/\u003cdate\u003e/\u003cspec-slug\u003e.json`. Scaffold-only runs go to `benchmarks/results/_scaffold/` so they cannot pollute real result series used by the leaderboard and paper figures.\n5. **MPS backend** — ships as a complete interface + Linux-runnable stub. Apple Silicon hardware is required only for real MPS execution; the stub allows CI to run everywhere.\n\n## v0.1 (3-month) acceptance criteria\n\n| # | criterion |\n|---|---|\n| 1 | Evaluation matrix: 64 of 80 cells (80%) completed |\n| 2 | MPS NSA reference implementation runs Qwen2.5-1.5B + RULER 16k end-to-end on Mac M1/M2 (needs a Mac contributor) |\n| 3 | CUDA NSA path reproduces the same numbers on Kaggle T4 |\n| 4 | Tech report v0.1 published as a GitHub release (Markdown) |\n| 5 | Static benchmark dataset + reference script published on Hugging Face Datasets |\n| 6 | Kaggle notebook public and reproducible end-to-end |\n| 7 | README documents the MPS path honestly: \"minimum human intervention through v0.1\" + maintainer-known absence of another MPS NSA OSS (counterexamples invited) |\n| 8 | `experiments/_wip/` contains the failed cells, not deleted |\n\n## Supported models and backends\n\n**Models** (via `configs/model/`): `qwen25_0_5b`, `qwen25_1_5b`, `llama32_1b`, `llama32_3b`\n\n**Attention backends** (via `configs/attention/`): `nsa_mps`, `h2o`, `snapkv` (registered and callable); `nsa_cuda`, `full`, `morphkv` config files exist but are not yet registered in the runtime registry\n\n**Benchmarks** (via `configs/benchmark/`): `ruler`, `longbench`, `niah`, `agentbench`\n\n**Devices** (via `configs/device/`): `mps`, `cuda`, `cpu`\n\n## Mac contributor wanted\n\nThe maintainer's development machine is WSL2 Linux x86_64, so MPS execution cannot be validated locally. The MPS backend ships as a complete interface + Linux-runnable stub, and we need contributors with Apple Silicon hardware (M1 / M2 / M3) to validate, profile, and report regressions. See issue #1.\n\n## Layout\n\n```\nsrc/nsa_eval/\n  attention/    # protocol, NSA wrapper, MPS backend stub, MorphKV decode, baselines\n  eval/         # eval spec, runner, benchmarks (RULER / LongBench / NIAH / AgentBench)\n  models/       # Qwen2.5, Llama-3.2, HF adapter\n  config/       # pydantic schema\n  utils/        # seed, device select, run manifest\n  cli.py\nconfigs/        # YAML configs: model / benchmark / attention / device\nnotebooks/      # Kaggle + MPS templates\npapers/         # tech report v0.1 source\nexperiments/_wip/   # failed-experiment archive (rm-forbidden)\n.github/workflows/  # ci / bench-dispatch / bench-mps / leaderboard-update / preprint-rebuild / release / nightly-deps\n```\n\n## License\n\nMIT.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhinanohart%2Fnsa-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhinanohart%2Fnsa-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhinanohart%2Fnsa-eval/lists"}