{"id":48725650,"url":"https://github.com/rocm/aiter","last_synced_at":"2026-04-25T08:01:20.442Z","repository":{"id":275220112,"uuid":"886637961","full_name":"ROCm/aiter","owner":"ROCm","description":"AI Tensor Engine for ROCm","archived":false,"fork":false,"pushed_at":"2026-04-19T03:49:37.000Z","size":197595,"stargazers_count":407,"open_issues_count":382,"forks_count":286,"subscribers_count":11,"default_branch":"main","last_synced_at":"2026-04-19T05:30:53.340Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://rocm.github.io/aiter/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ROCm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-11T10:38:20.000Z","updated_at":"2026-04-18T09:49:59.000Z","dependencies_parsed_at":"2025-02-15T08:18:26.082Z","dependency_job_id":"82713cc3-2b6d-4d0a-b22c-3ab1942174c1","html_url":"https://github.com/ROCm/aiter","commit_stats":null,"previous_names":["rocm/aiter"],"tags_count":31,"template":false,"template_full_name":null,"purl":"pkg:github/ROCm/aiter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ROCm%2Faiter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ROCm%2Faiter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ROCm%2Faiter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ROCm%2Faiter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ROCm","download_url":"https://codeload.github.com/ROCm/aiter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ROCm%2Faiter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32254714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-25T04:23:17.126Z","status":"ssl_error","status_checked_at":"2026-04-25T04:21:53.360Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-11T22:11:32.685Z","updated_at":"2026-04-25T08:01:20.429Z","avatar_url":"https://github.com/ROCm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"docs/assets/aiter_logo.png\" alt=\"AITER\" width=\"400\"\u003e\n\u003cbr\u003e\u003cbr\u003e\n\n[![CI](https://github.com/ROCm/aiter/actions/workflows/aiter-test.yaml/badge.svg)](https://github.com/ROCm/aiter/actions/workflows/aiter-test.yaml)\n[![Release](https://img.shields.io/github/v/release/ROCm/aiter)](https://github.com/ROCm/aiter/releases)\n[![Docs](https://img.shields.io/badge/Docs-rocm.github.io%2Faiter-blue)](https://rocm.github.io/aiter)\n[![Last Commit](https://img.shields.io/github/last-commit/ROCm/aiter)](https://github.com/ROCm/aiter/commits)\n\n\u003c/div\u003e\n\n--------------------------------------------------------------------------------\n\n**AITER** (AI Tensor Engine for ROCm) is AMD's high-performance AI operator library, providing optimized GPU kernels for inference and training workloads on ROCm. It serves as a unified collection of production-ready operators that framework developers can integrate directly into their stacks.\n\n### Key Features\n\n- **C++ and Python APIs** — use operators from either level\n- **Multiple kernel backends** — Triton, Composable Kernel (CK), and hand-tuned ASM\n- **Inference and training** — not just serving kernels, but also training and GEMM+communication fused kernels\n- **Framework-agnostic** — integrate into vLLM, SGLang, or any custom framework\n\n## News\n\n- **[2026/04]** [AITER v0.1.12.post1 Released](https://github.com/ROCm/aiter/releases/tag/v0.1.12.post1) — patch on v0.1.12 with GEMM and scale masking accuracy fixes; v0.1.12 highlights include blockwise sparse Sage Attention, fused gated RMSNorm+group quantization, etc., plus MI355X tuned configs for Kimi-K2.5 and DeepSeek-V3\n- **[2026/02]** [JAX-AITER: Bringing AMD's Optimized AI Kernels to JAX on ROCm](https://rocm.blogs.amd.com/software-tools-optimization/jax-aiter/README.html)\n- **[2026/02]** [Beyond Porting: How vLLM Orchestrates High-Performance Inference on AMD ROCm](https://blog.vllm.ai/2026/02/27/rocm-attention-backend.html)\n- **[2026/01]** [Character.ai: 2x Production Inference Performance on AMD Instinct GPUs](https://blog.character.ai/technical-deep-dive-how-digitalocean-and-amd-delivered-a-2x-production-inference-performance-increase-for-character-ai/)\n- **[2026/01]** [ROCm Becomes a First-Class Platform in the vLLM Ecosystem](https://rocm.blogs.amd.com/software-tools-optimization/vllm-omni/README.html)\n- **[2025]** [Accelerated LLM Inference with vLLM 0.9.x and ROCm](https://rocm.blogs.amd.com/software-tools-optimization/vllm-0.9.x-rocm/README.html)\n- **[2025]** [Accelerate DeepSeek-R1 Inference: Integrate AITER into SGLang](https://rocm.blogs.amd.com/artificial-intelligence/aiter-intergration-s/README.html)\n- **[2025/08]** [AITER-Enabled MLA Layer Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/software-tools-optimization/aiter-mla/README.html)\n- **[2025/08]** [Tutorial: MLA Decoding Kernel of the AITER Library to Accelerate LLM Inference](https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/gpu_dev_optimize/aiter_mla_decode_kernel.html)\n- **[2025/03]** [Accelerating DeepSeek Inference with AMD MI300 — Microsoft](https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/accelerating-deepseek-inference-with-amd-mi300-a-collaborative-breakthrough/4407673)\n- **[2025/03]** [AITER: AI Tensor Engine For ROCm — Launch Announcement](https://rocm.blogs.amd.com/software-tools-optimization/aiter-ai-tensor-engine/README.html)\n\n## Ecosystem\n\nAITER is the **default kernel backend for LLM inference on AMD GPUs**, integrated into the major serving frameworks and powering production workloads at scale.\n\n### Framework Integration\n\n| Framework | Integration | Status | Operators Used |\n|---|---|---|---|\n| [**vLLM**](https://github.com/vllm-project/vllm) | Default attention backend on ROCm | Production | MHA, MLA, Paged Attention, Fused MoE, GEMM, RMSNorm, RoPE+KVCache |\n| [**SGLang**](https://github.com/sgl-project/sglang) | Default on ROCm Docker | Production | Attention, Fused MoE, Block-scale GEMM, All-reduce, RMSNorm |\n| [**ATOM**](https://github.com/ROCm/ATOM) | Built natively on AITER | Active development | All AITER operators (attention, MoE, sampling, communication) |\n| [**JAX**](https://github.com/ROCm/jax-aiter) | XLA FFI bridge, no PyTorch dependency | Experimental | MHA/FMHA, RMSNorm, BF16 GEMM |\n| Various customer proprietary inference engines | Kernel-level integration | Production | Attention, MoE, GEMM, quantization |\n\n### Performance Highlights\n\n| Operator | Speedup |\n|---|---|\n| MLA decode kernel | up to **17x** |\n| MHA prefill kernel | up to **14x** |\n| Block-scaled Fused MoE | up to **3x** |\n| Block-scaled GEMM | up to **2x** |\n| DeepSeek-R1 e2e (SGLang) | 6,484 → **13,704** tok/s (2.1x) |\n| JAX-AITER attention (MI350) | **4.39x** median |\n\n\u003e For detailed benchmarks, see the [ATOM Benchmark Dashboard](https://rocm.github.io/ATOM/benchmark-dashboard/).\n\n### Supported Hardware\n\n| GPU | Architecture | Status |\n|---|---|---|\n| AMD Instinct MI300X | gfx942 (CDNA3) | Fully supported |\n| AMD Instinct MI325X | gfx942 (CDNA3) | Fully supported |\n| AMD Instinct MI350 | gfx950 (CDNA4) | Supported |\n| AMD Instinct MI355X | gfx950 (CDNA4) | Supported |\n\n## Operators\n\nAITER provides optimized kernels for attention, MoE, GEMM, normalization, quantization, communication, and more. Each operator has unit tests under [`op_tests/`](op_tests/) that you can run directly:\n\n```bash\n# Example: run a single operator test\npython3 op_tests/test_mha.py\npython3 op_tests/test_mla.py\npython3 op_tests/test_moe.py\npython3 op_tests/test_gemm_a8w8.py\npython3 op_tests/test_rmsnorm2d.py\n\n# See all available operator tests\nls op_tests/test_*.py\n```\n\n## Installation\n\n```bash\ngit clone --recursive https://github.com/ROCm/aiter.git\ncd aiter\npython3 setup.py develop\n```\n\nIf you happen to forget the `--recursive` during `clone`, you can use the following command after `cd aiter`\n```bash\ngit submodule sync \u0026\u0026 git submodule update --init --recursive\n```\n\n### FlyDSL (Optional)\n\nAITER's FusedMoE supports [FlyDSL](https://pypi.org/project/flydsl/)-based kernels for mixed-precision MOE (e.g., A4W4). FlyDSL is optional — when not installed, AITER automatically falls back to CK kernels.\n\n```bash\npip install --pre flydsl\n```\n\nOr install all optional dependencies at once:\n\n```bash\npip install -r requirements.txt\n```\n\n### Opus — Lightweight C++ Template for Kernel Development\n\n[Opus](csrc/include/opus/) is a single-header C++ template library (`opus.hpp`) for writing HIP kernels on AMD GPUs — vectorized load/store, layout abstractions, and MFMA wrappers with a strong focus on **build time optimization** (up to 61x faster than standard torch extension builds). See the [Opus README](csrc/include/opus/README.md) and [`op_tests/opus/`](op_tests/opus/) for details.\n\n### Triton-based Communication (Iris)\n\nAITER supports GPU-initiated communication using the [Iris library](https://github.com/ROCm/iris). This enables high-performance Triton-based communication primitives like reduce-scatter and all-gather.\n\n```bash\npip install -e .\npip install -r requirements-triton-comms.txt\n```\n\nFor more details, see [docs/triton_comms.md](docs/triton_comms.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frocm%2Faiter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frocm%2Faiter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frocm%2Faiter/lists"}