{"id":50082540,"url":"https://github.com/jeho-lee/Awesome-On-Device-AI-Systems","last_synced_at":"2026-06-08T07:01:45.131Z","repository":{"id":178194536,"uuid":"621585533","full_name":"jeho-lee/Awesome-On-Device-AI-Systems","owner":"jeho-lee","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-27T06:43:38.000Z","size":60,"stargazers_count":156,"open_issues_count":2,"forks_count":12,"subscribers_count":6,"default_branch":"main","last_synced_at":"2026-05-27T08:23:14.305Z","etag":null,"topics":["edge-computing","efficient-ai","machine-learning","mobile-systems","on-device-ai","resource-constrained-devices"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jeho-lee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-03-31T00:58:54.000Z","updated_at":"2026-05-27T06:43:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"6ad2a236-cf98-4b99-b3cf-43d6a84dfa39","html_url":"https://github.com/jeho-lee/Awesome-On-Device-AI-Systems","commit_stats":null,"previous_names":["jeho-lee/awesome-mobile-edge-ai-system-research","jeho-lee/awesome-on-device-ai","jeho-lee/awesome-efficient-ai","jeho-lee/awesome-on-device-ai-systems"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jeho-lee/Awesome-On-Device-AI-Systems","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeho-lee%2FAwesome-On-Device-AI-Systems","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeho-lee%2FAwesome-On-Device-AI-Systems/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeho-lee%2FAwesome-On-Device-AI-Systems/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeho-lee%2FAwesome-On-Device-AI-Systems/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jeho-lee","download_url":"https://codeload.github.com/jeho-lee/Awesome-On-Device-AI-Systems/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeho-lee%2FAwesome-On-Device-AI-Systems/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34051772,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["edge-computing","efficient-ai","machine-learning","mobile-systems","on-device-ai","resource-constrained-devices"],"created_at":"2026-05-22T16:04:04.263Z","updated_at":"2026-06-08T07:01:45.114Z","avatar_url":"https://github.com/jeho-lee.png","language":null,"funding_links":[],"categories":["Related Lists"],"sub_categories":["Speculative decoding"],"readme":"# Awesome On-Device AI Systems [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\r\n\r\nA curated list of **efficient on-device AI systems**, including practical inference engines, benchmarks, and state-of-the-art research papers for mobile and edge devices.\r\n\r\nThis repository bridges the gap between **Systems Research** (academic papers) and **Practical Deployment** (engineering frameworks), focusing on optimizing ML models (e.g., LLM/VLMs, ViTs, etc.) on resource-constrained hardware.\r\n\r\n\u003c!-- ############################################################################# --\u003e\r\n\r\n## 📂 Table of Contents\r\n\r\n- 🚀 Inference Engines\r\n  - [General ML Workloads](#general-ml-workloads)\r\n  - [LLM \u0026 GenAI Specialized](#llm--genai-specialized)\r\n  - [Vendor-Specific SDKs (NPU/DSP)](#vendor-specific-sdks-npudsp)\r\n\r\n\u003c!-- - [📊 Performance Benchmark](#-performance-benchmark) --\u003e\r\n\r\n- 📝 Research Papers\r\n  - [LLM Inference on Mobile SoCs](#llm-inference-on-mobile-socs)\r\n  - [Mobile Processor Characterization \u0026 Optimization](#mobile-processor-characterization--optimization)\r\n  - [Compiler-based ML Optimization](#compiler-based-ml-optimization)\r\n  - [Attention Acceleration](#attention-acceleration)\r\n  - [Quantization/Sparsity](#quantizationsparsity)\r\n  - [Application-centric On-device AI Systems](#application-centric-on-device-ai-systems)\r\n  - [Multi-DNN / Heterogeneous Runtime Scheduling](#multi-dnn--heterogeneous-runtime-scheduling)\r\n  - [On-device Training, Model Adaptation](#on-device-training-model-adaptation)\r\n  - [Profilers](#profilers)\r\n  \r\n\u003c!-- ############################################################################# --\u003e\r\n\r\n## 🚀 Inference Engines\r\n\r\nFrameworks and runtimes designed for deploying models on edge devices.\r\n\r\n### General ML Workloads\r\n* [LiteRT (formerly TensorFlow Lite)](https://ai.google.dev/edge/litert) - Google's framework for on-device inference.\r\n* [ExecuTorch](https://github.com/pytorch/executorch) - PyTorch’s end-to-end solution for enabling on-device AI.\r\n* [ONNX Runtime](https://onnxruntime.ai/) - Cross-platform inference engine for ONNX models.\r\n* [MNN](https://github.com/alibaba/MNN) - Lightweight deep learning framework by Alibaba.\r\n* [NCNN](https://github.com/Tencent/ncnn) - High-performance NN inference framework by Tencent.\r\n\r\n### Vendor-Specific SDKs\r\n* [Qualcomm QNN](https://www.qualcomm.com/developer/software/qualcomm-ai-engine-direct-sdk) - Qualcomm AI Stack for Snapdragon NPUs/DSPs.\r\n* [Apple Core ML](https://developer.apple.com/documentation/coreml) - Framework to integrate ML models into iOS/macOS apps.\r\n* [FluidAudio](https://github.com/FluidInference/FluidAudio) - Local audio AI SDK for Apple platforms with ASR, speaker diarization, VAD, and TTS optimized for Apple Neural Engine.\r\n* [NVIDIA TensorRT](https://developer.nvidia.com/tensorrt) - SDK for high-performance deep learning inference on NVIDIA GPUs (including Jetson).\r\n* [Intel OpenVINO](https://github.com/openvinotoolkit/openvino) - Toolkit for optimizing and deploying AI inference on Intel hardware (CPU/GPU/NPU).\r\n* [MediaTek NeuroPilot](https://neuropilot.mediatek.com/) - AI ecosystem and SDK for MediaTek NPUs.\r\n\r\n### LLM \u0026 GenAI Specialized\r\n* [llama.cpp](https://github.com/ggerganov/llama.cpp) - LLM inference in C/C++ with minimal dependencies.\r\n* [MLC LLM](https://github.com/mlc-ai/mlc-llm) - Universal solution for deploying LLMs on any hardware (based on TVM).\r\n* [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) - NVIDIA GPU-optimized LLM inference library, relevant for Jetson-class edge devices.\r\n* [mllm](https://github.com/UbiquitousLearning/mllm) - A fast and lightweight LLM inference engine for mobile and edge devices.\r\n* [MLX LM](https://github.com/ml-explore/mlx-lm) - LLM inference and fine-tuning toolkit built on MLX for Apple silicon.\r\n* [OmniInfer](https://github.com/omnimind-ai/OmniInfer-VLM) - High-performance, on-device VLM inference with hybrid NPU acceleration.\r\n* [RunAnywhere](https://github.com/RunanywhereAI/runanywhere-sdks) - Open-source SDK for running LLMs and multimodal models on-device across iOS, Android, and cross-platform apps.\r\n* [Off Grid](https://github.com/alichherawalla/off-grid-mobile-ai) - Open-source iOS/Android app running LLMs (Llama, Qwen, Gemma, Phi, DeepSeek) entirely on-device via llama.cpp. Includes voice (whisper.cpp), vision, on-device image generation, and tool calling.\r\n\r\n\u003c!-- ############################################################################# --\u003e\r\n\r\n## 📝 Research Papers\r\n\r\nNote: Some of the works are designed for inference acceleration on cloud/server infrastructure, which has much higher computational resources, but I also include them here if they can be potentially generalized to on-device inference use cases.\r\n\r\n#### LLM Inference on Mobile SoCs\r\n- [OSDI 2026] Inference in the Shadows: Taming Memory Bandwidth Contention in Mobile LLM Inference with Sereno\r\n- [MobiSys 2026] [Agent-X: Full Pipeline Acceleration of On-device AI Agents](https://arxiv.org/pdf/2605.10380)\r\n- [MLSys 2026] [Rethinking DVFS for Mobile LLMs: Unified Energy-Aware Scheduling with CORE](https://arxiv.org/abs/2507.02135)\r\n- [SenSys 2026] [LLM as a System Service on Mobile Devices](https://arxiv.org/pdf/2403.11805)\r\n- [EuroSys 2026] [Scaling LLM Test-Time Compute with Mobile NPU on Smartphones](https://arxiv.org/pdf/2509.23324v1)\r\n- [SOSP 2025] [Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference](https://arxiv.org/abs/2501.14794)\r\n- [ASPLOS 2025] [Neuralink: Fast on-Device LLM Inference with Neuron Co-Activation Linking](https://dl.acm.org/doi/10.1145/3676642.3736114)\r\n- [ASPLOS 2025] [Fast On-device LLM Inference with NPUs](https://arxiv.org/abs/2407.05858)\r\n- [arXiv 2024] [PowerInfer-2: Fast Large Language Model Inference on a Smartphone](https://arxiv.org/abs/2406.06282)\r\n\r\n#### Mobile Processor Characterization \u0026 Optimization\r\n- [EuroSys 2026] [viNPU: Optimizing Vision Transformer Inference on Mobile NPUs](https://dl.acm.org/doi/10.1145/3767295.3803619)\r\n- [ASPLOS 2026] [FlashMem: Supporting Modern DNN Workloads on Mobile with GPU Memory Hierarchy Optimizations](https://arxiv.org/abs/2602.15379)\r\n- [ICS 2025] [TMModel: Modeling Texture Memory and Mobile GPU Performance to Accelerate DNN Computations](https://doi.org/10.1145/3721145.3725774)\r\n\r\n#### Compiler-based ML Optimization\r\n- [ASPLOS 2024] [SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobile](https://dl.acm.org/doi/pdf/10.1145/3620666.3651384)\r\n- [ASPLOS 2024] [SoD\u003csup\u003e2\u003c/sup\u003e: Statically Optimizing Dynamic Deep Neural Network Execution](https://dl.acm.org/doi/pdf/10.1145/3617232.3624869)\r\n- [MICRO 2023] [Improving Data Reuse in NPU On-chip Memory with Interleaved Gradient Order for DNN Training](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=\u0026arnumber=10411391)\r\n- [MICRO 2022] [GCD\u003csup\u003e2\u003c/sup\u003e: A Globally Optimizing Compiler for Mapping DNNs to Mobile DSPs](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=\u0026arnumber=9923837)\r\n- [PLDI 2021] [DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusion](https://dl.acm.org/doi/pdf/10.1145/3453483.3454083)\r\n\r\n#### Attention Acceleration\r\n- [MLSys 2026] [IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference](https://arxiv.org/abs/2511.21513)\r\n- [MobiSys 2026] [ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inference](https://arxiv.org/abs/2508.16703)\r\n- [MLSys 2025] [MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devices](https://arxiv.org/pdf/2411.17720)\r\n- [MLSys 2025] [TurboAttention: Efficient attention approximation for High Throughputs LLMs](https://arxiv.org/pdf/2412.08585)\r\n- [ASPLOS 2023] [FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks](https://dl.acm.org/doi/10.1145/3575693.3575747)\r\n- [NeurIPS 2022] [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/pdf/2205.14135)\r\n\r\n#### Quantization/Sparsity\r\n- [ASPLOS 2026] [oFFN: Outlier and Neuron-aware Structured FFN for Fast yet Accurate LLM Inference](https://dl.acm.org/doi/pdf/10.1145/3779212.3790194)\r\n- [MLSys 2024] [AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration](https://arxiv.org/pdf/2306.00978)\r\n- [ISCA 2023] [OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization](https://arxiv.org/abs/2304.07493)\r\n\r\n#### Application-centric On-device AI Systems\r\n- [MobiSys 2025] [ARIA: Optimizing Vision Foundation Model Inference on Heterogeneous Mobile Processors for Augmented Reality](https://dl.acm.org/doi/10.1145/3711875.3729161)\r\n- [MobiCom 2024] [Panopticus: Omnidirectional 3D Object Detection on Resource-constrained Edge Devices](https://arxiv.org/pdf/2410.01270)\r\n- [MobiCom 2024] [Perceptual-Centric Image Super-Resolution using Heterogeneous Processors on Mobile Devices](https://dl.acm.org/doi/10.1145/3636534.3690698)\r\n- [IPSN 2023] [PointSplit: Towards On-device 3D Object Detection with Heterogeneous Low-power Accelerators](https://dl.acm.org/doi/pdf/10.1145/3583120.3587045)\r\n- [MobiSys 2023] [OmniLive: Super-Resolution Enhanced 360° Video Live Streaming for Mobile Devices](https://dl.acm.org/doi/pdf/10.1145/3581791.3596851)\r\n- [MobiCom 2022] [NeuLens: Spatial-based Dynamic Acceleration of Convolutional Neural Networks on Edge](https://dl.acm.org/doi/pdf/10.1145/3495243.3560528)\r\n- [MobiCom 2021] [Flexible high-resolution object detection on edge devices with tunable latency](https://dl.acm.org/doi/abs/10.1145/3447993.3483274)\r\n\r\n#### Multi-DNN / Heterogeneous Runtime Scheduling\r\n- [PPoPP 2024] [Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous SoCs](https://dl.acm.org/doi/pdf/10.1145/3627535.3638502)\r\n- [RTSS 2024] [FLEX: Adaptive Task Batch Scheduling with Elastic Fusion in Multi-Modal Multi-View Machine Perception](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10844787)\r\n- [MobiSys 2024] [Pantheon: Preemptible Multi-DNN Inference on Mobile Edge GPUs](https://dl.acm.org/doi/pdf/10.1145/3643832.3661878)\r\n- [Sensys 2023] [Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU](https://dl.acm.org/doi/10.1145/3625687.3625789)\r\n- [ATC 2023] [Decentralized Application-Level Adaptive Scheduling for Multi-Instance DNNs on Open Mobile Devices](https://www.usenix.org/system/files/atc23-sung.pdf)\r\n- [MobiSys 2022] [Band: Coordinated Multi-DNN Inference on Heterogeneous Mobile Processors](https://dl.acm.org/doi/pdf/10.1145/3498361.3538948)\r\n- [MobiSys 2022] [CoDL: efficient CPU-GPU co-execution for deep learning inference on mobile devices](https://dl.acm.org/doi/pdf/10.1145/3498361.3538932)\r\n\r\n#### On-device Training, Model Adaptation\r\n- [ASPLOS 2025] [Nazar: Monitoring and Adapting ML Models on Mobile Devices](https://dl.acm.org/doi/pdf/10.1145/3669940.3707246)\r\n- [SenSys 2024] [AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environments](https://arxiv.org/pdf/2410.08256)\r\n- [SenSys 2023] [EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge](https://dl.acm.org/doi/10.1145/3625687.3625793)\r\n- [MobiCom 2023] [Cost-effective On-device Continual Learning over Memory Hierarchy with Miro](https://dl.acm.org/doi/pdf/10.1145/3570361.3613297)\r\n- [MobiCom 2023] [AdaptiveNet: Post-deployment Neural Architecture Adaptation for Diverse Edge Environments](https://dl.acm.org/doi/pdf/10.1145/3570361.3592529)\r\n- [MobiSys 2023] [ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection](https://dl.acm.org/doi/pdf/10.1145/3581791.3596852)\r\n- [SenSys 2023] [On-NAS: On-Device Neural Architecture Search on Memory-Constrained Intelligent Embedded Systems](https://dl.acm.org/doi/10.1145/3625687.3625814)\r\n- [MobiCom 2022] [Mandheling: mixed-precision on-device DNN training with DSP offloading](https://dl.acm.org/doi/abs/10.1145/3495243.3560545)\r\n- [MobiSys 2022] [Memory-efficient DNN training on mobile devices](https://dl.acm.org/doi/abs/10.1145/3498361.3539765)\r\n\r\n#### Profilers\r\n- [MobiCom 2024] [MELTing point: Mobile Evaluation of Language Transformers](https://arxiv.org/abs/2403.12844) [[code]](https://github.com/brave-experiments/MELT-public)\r\n- [SenSys 2023] [nnPerf: Demystifying DNN Runtime Inference Latency on Mobile Platforms](https://dl.acm.org/doi/10.1145/3625687.3625797)\r\n- [MobiSys 2021] [nn-Meter: towards accurate latency prediction of deep-learning model inference on diverse edge devices](https://dl.acm.org/doi/10.1145/3458864.3467882)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeho-lee%2FAwesome-On-Device-AI-Systems","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjeho-lee%2FAwesome-On-Device-AI-Systems","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeho-lee%2FAwesome-On-Device-AI-Systems/lists"}