{"id":49680765,"url":"https://github.com/manishklach/sram-inference-kernel-fastpath","last_synced_at":"2026-05-09T08:01:48.621Z","repository":{"id":355355491,"uuid":"1227476733","full_name":"manishklach/sram-inference-kernel-fastpath","owner":"manishklach","description":"Experimental Linux kernel fast-path patches for SRAM-based AI inference servers, targeting io_uring submission, registered buffers, CQ polling, wakeup attribution, and completion latency.","archived":false,"fork":false,"pushed_at":"2026-05-03T10:47:41.000Z","size":3754,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-05-08T07:03:38.217Z","etag":null,"topics":["ai-inference","ai-infrastructure","async-io","benchmarking","bpftrace","ebpf","ftrace","inference-systems","io-uring","kernel-development","kernel-fastpath","kernel-tracing","latency","linux-kernel","low-latency","operating-systems","performance-analysis","sram","systems-performance"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manishklach.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-02T18:30:58.000Z","updated_at":"2026-05-03T10:47:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"8b4388dc-e53e-4adc-b44a-4158e8818220","html_url":"https://github.com/manishklach/sram-inference-kernel-fastpath","commit_stats":null,"previous_names":["manishklach/sram-inference-kernel-fastpath"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/manishklach/sram-inference-kernel-fastpath","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manishklach%2Fsram-inference-kernel-fastpath","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manishklach%2Fsram-inference-kernel-fastpath/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manishklach%2Fsram-inference-kernel-fastpath/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manishklach%2Fsram-inference-kernel-fastpath/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manishklach","download_url":"https://codeload.github.com/manishklach/sram-inference-kernel-fastpath/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manishklach%2Fsram-inference-kernel-fastpath/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32811662,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-08T08:22:46.396Z","status":"online","status_checked_at":"2026-05-09T02:00:06.633Z","response_time":123,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-inference","ai-infrastructure","async-io","benchmarking","bpftrace","ebpf","ftrace","inference-systems","io-uring","kernel-development","kernel-fastpath","kernel-tracing","latency","linux-kernel","low-latency","operating-systems","performance-analysis","sram","systems-performance"],"created_at":"2026-05-07T06:44:21.741Z","updated_at":"2026-05-09T08:01:48.615Z","avatar_url":"https://github.com/manishklach.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SRAM Inference Kernel Fastpath\n\nExperimental Linux kernel fast-path research for SRAM-style AI inference servers, focused on io_uring submission latency, batching, registered buffers, tracing, and native attribution.\n\n## TL;DR\nIn deterministic AI inference (~20µs execution), Linux host overhead can match or exceed device \nlatency, effectively doubling end-to-end request time.\n\nIn our synthetic SRAM-style workload (~20µs compute), baseline p99 reaches ~40–50µs, indicating \nhost overhead comparable to device execution.\n\nThis repo isolates that overhead and prototypes the kernel fast paths required to close the gap.\n\n\u003e Once inference becomes deterministic, the Linux control plane—not the model—dominates latency.\n\n\nThis project targets the **post-compute bottleneck regime**, where hardware execution is no \nlonger the dominant source of latency.\n\n## What This Repo Demonstrates\n\n- **Deterministic compute does not eliminate latency variance**: Even with zero-variance \n  hardware execution, host-side effects drive significant jitter.\n- **Linux submission and completion paths remain significant**: System call overhead and \n  completion delivery pipelines contribute measurable microseconds.\n- **Tail latency (p99/p999) is driven by host-side effects**: Scheduling and interrupt \n  handling costs dominate the \"tail\" of the latency distribution.\n- **Existing io_uring fast paths reduce but do not eliminate this gap**: Existing io_uring fast paths reduce important parts of the path, while this repo measures what residual latency remains.\n\n## Benchmark Tracks\n\nTo provide a comprehensive evaluation, the validation harness supports two distinct tracks:\n- **NOP mode**: Measures raw `io_uring` ring overhead with minimal operations.\n- **SRAM20 mode**: Implements a deterministic AI inference model using a 20µs busy-wait \n  to simulate predictable hardware execution.\n\n## Takeaway\n\nSRAM-style AI inference does not eliminate latency—it exposes the Linux control plane as the \ndominant bottleneck. Closing this gap requires not faster accelerators, but faster kernel paths.\n\n## Important\n\nWSL results are used for **harness validation only**. They are NOT used to draw conclusions about:\n- SQPOLL effectiveness\n- Kernel scheduling behavior\n- Completion latency\n\nAll definitive research conclusions require native Linux validation.\n\n## Documentation \u0026 Methodology\n\n- [Quickstart Guide](docs/quickstart.md)\n- [Native Linux Validation Guide](docs/native-linux-validation.md)\n- [Existing io_uring Fast Paths and Remaining Gaps](docs/kernel-patches/existing-io_uring-fastpaths.md)\n- [Maintainer FAQ](docs/maintainer-faq.md)\n- [Project Roadmap](docs/roadmap.md)\n\n## Key Finding: Batching Dominates Submission Latency\n\nOur latest research indicates that for microsecond-scale inference, **Batching** is the most powerful optimization lever. It reduces per-request submission overhead by **~7×** in synthetic SRAM-style workloads, bringing the effective submission tax from ~600ns to **\u003c100ns per request**.\n\nSee [Submission Path Analysis](docs/kernel-patches/submission-path-analysis.md) for the full technical breakdown.\n\n## Optimal Batching Strategy\n\nThere exists a batch size range (8–16) that minimizes per-request overhead without significantly increasing base latency. Pushing beyond batch 16 yields diminishing returns and increases total end-to-end time.\n\nSee [Batch Sweep Results](docs/batch-sweep-results.md) for the optimization data.\n\n## Performance Visualization\n\nThe following plots summarize the batch sweep experiments:\n\n![Batch Latency](docs/assets/batch_latency.png)\n![Submission Cost](docs/assets/submission_cost.png)\n![Tail Spread](docs/assets/tail_spread.png)\n\n## Adaptive Batching Strategy\n\nReal-world inference systems must balance throughput and latency in the presence of host-side jitter. Our adaptive batching experiment demonstrates that a simple latency-based heuristic can outperform static strategies, particularly in reducing **p99 tail latency**.\n\nSee [Adaptive Batching Results](docs/adaptive-batching-results.md) for the performance breakdown.\n\n## Submission Path Bottleneck\n\nRecent native-like validation data shows that for deterministic workloads, **Submission-side latency** (`submit → issue`) is the primary bottleneck. Even after applying existing `io_uring` fast paths, the cost of the system call transition and request dispatch remains a significant contributor to tail latency.\n\nResearch has pivoted from completion-side polling to optimizing the submission plane to match the performance of microsecond-scale hardware.\n\n- [Submission Path Analysis](docs/kernel-patches/submission-path-analysis.md)\n- [Submission Experiment Plan](docs/submission-experiment-plan.md)\n\n## Preliminary Native Attribution Results\n\nInitial WSL-based validation indicates:\n- **p99 Dominance**: Latency is currently dominated by **Submission Path overhead** and **Hypervisor Jitter**.\n- **Completion Path**: Residual host-side completion latency is sub-microsecond in synchronous modes.\n- **Decision**: Experimental CQ polling is **NOT yet justified** on native hardware. Further bare-metal measurement is required to isolate kernel-specific completion bottlenecks.\n\nSee [Native Latency Breakdown](docs/native-latency-breakdown.md) for detailed attribution data.\n\n## What This Is / Is Not\n\n**This is:**\n- Experimental Linux kernel fast-path research and prototyping.\n- Reproducible latency modeling for deterministic AI workloads.\n- A measurement-first effort to justify new kernel APIs.\n\n**This is NOT:**\n- A production-ready kernel patch (yet).\n- A replacement for standard `io_uring` features.\n- Performance theater using non-deterministic hardware.\n\n## License\nGPL v2\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanishklach%2Fsram-inference-kernel-fastpath","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanishklach%2Fsram-inference-kernel-fastpath","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanishklach%2Fsram-inference-kernel-fastpath/lists"}