{"id":49224157,"url":"https://github.com/erectbranch/mit-efficient-ai","last_synced_at":"2026-04-24T06:04:36.521Z","repository":{"id":65280580,"uuid":"585158240","full_name":"erectbranch/MIT-Efficient-AI","owner":"erectbranch","description":"TinyML and Efficient Deep Learning Computing | MIT 6.S965/6.5940","archived":false,"fork":false,"pushed_at":"2026-03-26T04:45:28.000Z","size":105109,"stargazers_count":27,"open_issues_count":0,"forks_count":7,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-03-27T01:08:15.715Z","etag":null,"topics":["ai","deep-learning","efficient-ai","lecture-notes","mit-65940","mit-6s965","tinyml"],"latest_commit_sha":null,"homepage":"https://erectbranch.github.io/#/MIT-Efficient-AI/notes/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/erectbranch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-01-04T13:18:33.000Z","updated_at":"2026-03-26T04:45:33.000Z","dependencies_parsed_at":"2023-10-05T11:44:18.863Z","dependency_job_id":"7eb6c6d8-1529-4eb2-b851-8df1c827058a","html_url":"https://github.com/erectbranch/MIT-Efficient-AI","commit_stats":null,"previous_names":["erectbranch/mit-efficient-ai"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/erectbranch/MIT-Efficient-AI","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erectbranch%2FMIT-Efficient-AI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erectbranch%2FMIT-Efficient-AI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erectbranch%2FMIT-Efficient-AI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erectbranch%2FMIT-Efficient-AI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/erectbranch","download_url":"https://codeload.github.com/erectbranch/MIT-Efficient-AI/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erectbranch%2FMIT-Efficient-AI/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32211386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T03:15:14.334Z","status":"ssl_error","status_checked_at":"2026-04-24T03:15:11.608Z","response_time":64,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","deep-learning","efficient-ai","lecture-notes","mit-65940","mit-6s965","tinyml"],"created_at":"2026-04-24T06:04:31.773Z","updated_at":"2026-04-24T06:04:36.515Z","avatar_url":"https://github.com/erectbranch.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv width=\"100%\" height=\"100%\" align=\"center\"\u003e\n  \n\u003ch1 align=\"center\"\u003e\n  \u003cp align=\"center\"\u003eTinyML and Efficient Deep Learning Computing\u003c/p\u003e\n\u003c/h1\u003e\n\n\u003cb\u003eMIT 6.S965/6.5940 • Fall • 2022-2024\u003c/b\u003e\n\u003cbr\u003e\nInstructor : Song Han(Associate Professor, MIT EECS)\n\n\u003c/div\u003e\n\n\u003cbr\u003e\n\nLecture notes for courses [MIT 6.S965, Fall 2022 | MIT 6.5940, Fall 2023•2024](https://efficientml.ai)\n\n## Courses\n\n| Course | Video | Slide | Note | Homework |\n| --- | --- | --- | --- | --- |\n| MIT 6.5940 • 2024 • Fall | [Videos](https://youtube.com/playlist?list=PL80kAHvQbh-qGtNc54A6KW4i4bkTPjiRF) | [Slides](https://hanlab.mit.edu/courses/2024-fall-65940#schedule) | [Notes](2024/) | [Lab 1](https://colab.research.google.com/drive/1Fagq3JQBzCizodyxpHKvWDzfCC7F1RWN) / [Lab 2](https://colab.research.google.com/drive/11IBla1q1McoZ2oCANCGHns8VtzG5nCMP) / [Lab 3](https://colab.research.google.com/drive/1xKReLBHVS6bkFbYkfi-Ky3C4loQmG6Yc) / [Lab 4](https://colab.research.google.com/drive/16H9RvSg4XIF35X3fLGQUVwAE9ccvDj14) / [Lab 5](https://drive.google.com/drive/folders/1MhMvxvLsyYrN-4C6eQG8Zj2JeSuyAOf0) |\n| MIT 6.5940 • 2023 • Fall | [Videos](https://youtube.com/playlist?list=PL80kAHvQbh-pT4lCkDT53zT8DKmhE0idB) | [Slides](https://hanlab.mit.edu/courses/2023-fall-65940#schedule) | [Notes](2023/) | - |\n| MIT 6.S965 • 2022 • Fall | [Videos](https://youtube.com/playlist?list=PL80kAHvQbh-ocildRaxjjBy6MR1ZsNCU7) | [Slides](https://hanlab.mit.edu/courses/2022-fall-6s965#schedule) | [Notes](2022/) | [Lab 4: Deployment on MCU](https://github.com/Xuweijia-buaa/MIT-6.S965-TinyML-and-Efficient-Deep-Learning-Computing/blob/main/notebooks/mit_6s965_lab4_tinyml.ipynb) |\n\n## Lecture Notes\n\n### 📖 Basics of Deep Learning\n\n- [Basic Terminologies, Shape of Tensors](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec02/summary01)\n\n  \u003e Synapse(weight), Neuron(activation), Cell body\n\n  \u003e Fully-Connected layer, Convolution layer(padding, stride, receptive field, grouped convolution), Pooling layer\n\n- [Efficiency Metrics](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec02/summary02)\n\n  \u003e Metrics(latency, storage, energy)\n\n  \u003e Memory-Related(\\#parameters, model size, \\#activations), Computation(MACs, FLOP)\n\n### 📙 Efficient Inference\n\n- [Pruning Granularity, Pruning Critertion](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec03)\n\n  \u003e Unstructured/Structured pruning(Fine-grained/Pattern-based/Vector-level/Kernel-level/Channel-level)\n  \n  \u003e Pruning Criterion: Magnitude(L1-norm, L2-norm), Sensitivity and Saliency(SNIP), Loss Change(First-Order, Second-Order Taylor Expansion)\n  \n  \u003e Data-Aware Pruning Criterion: Average Percentage of Zero(APoZ), Reconstruction Error, Entropy\n\n- [Automatic Pruning, Lottery Ticket Hypothesis](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec04/summary01)\n\n  \u003e Finding Pruning Ratio: Reinforcement Learning based, Rule based, Regularization based, Meta-Learning based \n\n  \u003e Lottery Ticket Hypothesis(Winning Ticket, Iterative Magnitude Pruning, Scaling Limitation)\n\n  \u003e Pruning at Initialization(Connection Sensitivity, Gradient Flow)\n\n- [System \u0026 Hardware Support for Fine-grained Sparsity](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec04/summary02)\n\n  \u003e Efficient Inference Engine(EIE format: relative index, column pointer)\n\n- [Sparse Matrix-Matrix Multiplication, GPU Support for Sparsity](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec04/summary03)\n\n  \u003e Sparse Matrix-Matrix Multiplication(SpMM), CSR format\n\n  \u003e GPU Support for Sparsity: Hierarchical 1-Dimensional Tiling, Row Swizzle, M:N Sparsity, Block SpMM(Blocked-ELL format), PatDNN(FKW format)\n\n  ---\n\n- [Basic Concepts of Quantization](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec05/summary01)\n\n  \u003e Numeric Data Types: Integer, Fixed-Point, Floating-Point(IEEE FP32/FP16, BF16, NVIDIA FP8), INT4 and FP4\n\n  \u003e Uniform vs. Non-uniform quantization, Symmetric vs. Asymmetric quantization\n\n  \u003e Linear Quantization: Integer-Arithmetic-Only Quantization, Sources of Quantization Error(clipping, rounding, scaling factor, zero point)\n\n- [Vector Quantization](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec05/summary02)\n\n  \u003e Vector Quantization(Deep compression: iterative pruning, K-means based quantization, Huffman encoding), Product Quantization\n\n- [Post Training Quantization](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec06/summary01)\n\n  \u003e Weight Quantiztion: Per-Tensor Activation Per-Channel Activation, Group Quantization(Per-Vector, MX), Weight Equalization, Adative Rounding\n\n  \u003e Activation Quantization: During training(EMA), Calibration(Min-Max, KL-divergence, Mean Squared Error)\n\n  \u003e Bias Correction, Zero-Shot Quantization(ZeroQ)\n\n- [Quantization-Aware Training, Low bit-width quantization](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec06/summary02)\n\n  \u003e Fake quantization, Straight-Through Estimator\n\n  \u003e Binary Quantization(Deterministic, Stochastic, XNOR-Net), Ternary Quantization\n\n  ---\n\n- [Neural Architecture Search: basic concepts \u0026 manually-designed neural networks](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec07/summary01)\n\n  \u003e input stem, stage, head\n  \n  \u003e AlexNet, VGGNet, SqueezeNet(fire module), ResNet(bottleneck block, residual connection), ResNeXt(grouped convolution)\n  \n  \u003e MobileNet(depthwise-separable convolution, width/resolution multiplier), MobileNetV2(inverted bottleneck block), ShuffleNet(channel shuffle), SENet(squeeze-and-excitation block), MobileNetV3(h-swish)\n\n- [Neural Architecture Search: Search Space](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec07/summary02)\n\n  \u003e Search Space: Macro, Chain-Structured, Cell-based(NASNet), Hierarchical(Auto-DeepLab, NAS-FPN)\n\n  \u003e design search space: Cumulative Error Distribution, FLOPs distribution, zero-cost proxy\n\n- [Neural Architecture Search: Performance Estimation \u0026 Hardware-Aware NAS](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec08)\n\n  \u003e Weight Inheritance, HyperNetwork, Weight Sharing(super-network, sub-network)\n\n  \u003e Performance Estimation Heuristics: Zen-NAS, GradSign\n\n  ---\n\n- [Knowledge Distillation](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec10/summary01)\n\n  \u003e Knowledge Distillation(distillation loss, softmax temperature)\n  \n  \u003e What to Match?: intermediate weights, features(attention maps), sparsity pattern, relational information\n\n  \u003e Distillation Scheme: Offline Distillation, Online Distillation, Self-Distillation\n\n- [Distillation for Applications](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec10/summary02)\n\n  \u003e Applications: Object Detection, Semantic Segmentation, GAN, NLP\n\n  \u003e Tiny Neural Network: NetAug\n\n  ---\n\n- [MCUNet](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec11)\n\n  \u003e MCUNetV1: TinyNAS, TinyEngine\n\n  \u003e MCUNetV2: MCUNetV2 architecture(MobileNetV2-RD), patch-based inference, joint automated search\n\n### ⚙️ Efficient Training and System Support\n\n- [On-device Training, Transfer Learning](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec15)\n\n  \u003e Efficient Transfer Learning: TinyTL(Bias-only Fine-tuning, Lite residual module)\n\n  \u003e Sparse Layer/Tensor Update(Contribution Analysis), Real Quantized Training\n\n- [Tiny Training Engine](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec16/summary01/)\n\n  \u003e Intermediate Representation(IR), Compile-Time Autodiff\n\n  \u003e Graph-level Optimization(Sparse Update, Operation Reordering)\n\n- [Compilers, Graph-level Optimization](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec16/summary02/)\n\n  \u003e Compilers: Halide, TVM, AutoTVM\n\n  \u003e Graph-level Optimization Framework: MetaFlow, IOS\n\n- [Microcontroller, Loop Optimization](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec17/summary01)\n\n  \u003e Memory Hierarchy of Microcontroller, Primary Memory Format(NCHW, NHWC, CHWN)\n\n  \u003e Parallel Computing Techniques: Loop Optimization(Unrolling, Reordering, Tiling)\n\n- [Inference Optimization](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec17/summary02)\n\n  \u003e Parallel Computing Techniques: SIMD Programming(CMSIS-NN)\n  \n  \u003e Inference Optimization: Im2col, In-place, Choosing Data Layout(pointwise, depthwise), Winograd Convolution\n\n  ---\n\n### 🔧 Domain-Specific Optimizations\n\n- [Transformer](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2023/lec12/summary01)\n\n  \u003e NLP Task(Discriminative, Generative), Pre-Transformer Era(RNN/LSTM, CNN)\n\n  \u003e Transformer: Tokenizer, Embedding, Multi-Head Attention(self-attention), Feed-Forward Network, Layer Normalization(Pre-Norm, Post-Norm), Positional Encoding\n\n- [Transformer Design Variants](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2023/lec12/summary02)\n\n  \u003e Types of Transformer-based Models: Encoder-Decoder(T5), Encoder-only(BERT), Decoder-only(GPT)\n  \n  \u003e Relative Positional Encoding(ALiBi, RoPE, interpolating RoPE), KV cache optimization(Multi-query Attention, Grouped-query Attention), Gated Linear Unit\n\n  ---\n\n- [LLM Quantization](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2024/lec13/summary01)\n  \n  \u003e Quantization Difficulty of LLMs, Bottleneck of edge LLM Inference(Memory-bounded, Memory footprint of Weights)\n  \n  \u003e Weight-activation Quantization: SmoothQuant(Activation Smoothing)\n  \n  \u003e Weight-only Quantization: AWQ(1% Salient Weights, Activation-aware Scaling)\n\n- [Efficient System Support for LLM Quantization](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2024/lec13/summary02)\n  \n  \u003e System for Edge: TinyChat(Hardware-aware Weight Packing, Kernel Fusion)\n\n  \u003e System for Cloud: Overhead in Quantized GEMM, QServe(SmoothAttention, Dequantization with Reg-Level Parallelism)\n\n- [LLM Pruning \u0026 Sparsity](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2024/lec13/summary03)\n  \n  \u003e Weight Sparsity: Wanda\n\n  \u003e Contextual Sparsity: Deja Vu, Mixture-of-Experts\n\n  \u003e Attention Sparsity: SpAtten, H2O\n\n- [LLM Serving Systems](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2024/lec13/summary04)\n\n  \u003e Metrics for LLM Serving(TTFT, TPOT, Latency, Throughput),  Heuristics for Evaluation\n\n  \u003e PagedAttention(vLLM), FlashAttention, Speculative Decoding, Batching(Static, Dynamic, Continuous)\n\n  ---\n\n- [LLM Post Training](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2024/lec14/summary01)\n\n  \u003e Supervised Fine-Tuning, Reinforcement Learning from Human Feedback, Direct Preference Optimization\n\n  \u003e Parameter-Efficient Fine-Tuning: Additive(Adapter, Prompt/Prefix Tuning) Selective(BitFit), Reparameterized(LoRA)\n\n  \u003e PEFT Quantization: QLoRA, BitDelta\n\n- [Prompt Engineering](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2024/lec14/summary02)\n\n  \u003e In-Context Learning: Zero-Shot Prompting, Few-Shot Prompting\n\n  \u003e Chain-of-Thought, Retrieval Augmented Generation\n\n- [Long Context LLM](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2024/lec15/summary01)\n\n  \u003e LongLoRA(Shifted Sparse Attention), StreamingLLM(Attention Sink)\n\n  \u003e The Lost in the Middle Phenomenon, Needle In A Haystack Analysis, LongBench\n\n- [Efficient Attention, Beyond Transformers](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2024/lec15/summary02)\n\n  \u003e DuoAttention(Retrieval, Streaming Head), Quest(Query-Aware Sparsity)\n\n  \u003e Mamba(Selective State-Space Models), Jamba\n\n  ---\n\n- [Vision Transformer](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2023/lec14/summary01)\n\n  \u003e Vision Transformer, High-Resolution Dense Prediction(Segment Anything)\n  \n  \u003e Window Attention(Swin Transformer, FlatFormer), ReLU Linear Attention(EfficientViT, EfficientViT-SAM), Sparse Attention(SparseViT)\n\n- [ViT Training, AR Image Generation](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2023/lec14/summary02)\n\n  \u003e Contrastive Learning(CLIP), Masked Image Modeling\n\n  \u003e Multi-modal LLMs(Flamingo, PaLM-E)\n  \n  \u003e Autoregressive Image Generation(VAR, HART)\n\n- [Efficient Video Understanding](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec19/summary01)\n\n  \u003e 2D CNNs for Video Understanding, 3D CNNs for Video Understanding(I3D), Temporal Shift Module(TSM)\n\n  \u003e Other Efficient Methods: Kernel Decomposition, Multi-Scale Modeling, Neural Architecture Search(X3D), Skipping Redundant Frames/Clips, Utilizing Spatial Redundancy\n\n- [Generative Adversarial Networks (GANs)](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec19/summary02)\n\n  \u003e GANs(Generator, Discriminator), Conditional/Unconditional GANs, Difficulties in GANs\n\n  \u003e Compress Generator(GAN Compression), Dynamic Cost GANs(Anycost GANs), Data-Efficient GANs(Differentiable Augmenatation)\n\n- [Efficient Point Cloud Recognition](https://github.com/erectbranch/MIT-Efficient-AI/tree/master/2022/lec18/summary01)\n\n  \u003e 3D Data: Point Clouds, Multi-view Images(Range Image, BEV Projection), Voxels(Dense, Sparse), Octree\n\n  \u003e Hybrid: Point-Voxel CNN(PVCNN, SPVConv), BEVFusion","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferectbranch%2Fmit-efficient-ai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ferectbranch%2Fmit-efficient-ai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferectbranch%2Fmit-efficient-ai/lists"}