{"id":27427320,"url":"https://github.com/deepbiolab/mamba-in-depth","last_synced_at":"2025-04-14T12:49:53.320Z","repository":{"id":285251898,"uuid":"957497733","full_name":"deepbiolab/mamba-in-depth","owner":"deepbiolab","description":"A comprehensive learning guide for understanding S4 (Structured State Space Sequence Model) and Mamba architectures, from fundamentals through implementation, with curated resources and hands-on examples.","archived":false,"fork":false,"pushed_at":"2025-03-31T15:44:04.000Z","size":3597,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-31T17:00:03.762Z","etag":null,"topics":["learning-resources","mamba","state-space-models"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepbiolab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-30T14:22:21.000Z","updated_at":"2025-03-31T15:44:07.000Z","dependencies_parsed_at":"2025-03-31T17:00:05.057Z","dependency_job_id":null,"html_url":"https://github.com/deepbiolab/mamba-in-depth","commit_stats":null,"previous_names":["deepbiolab/s4-in-depth","deepbiolab/mamba-in-depth"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepbiolab%2Fmamba-in-depth","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepbiolab%2Fmamba-in-depth/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepbiolab%2Fmamba-in-depth/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepbiolab%2Fmamba-in-depth/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepbiolab","download_url":"https://codeload.github.com/deepbiolab/mamba-in-depth/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248884374,"owners_count":21177397,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["learning-resources","mamba","state-space-models"],"created_at":"2025-04-14T12:49:52.576Z","updated_at":"2025-04-14T12:49:53.291Z","avatar_url":"https://github.com/deepbiolab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# S4 \u0026 Mamba In-Depth Guide\n\n\n## 📚 Learning Path Overview\n\nThis guide provides a structured learning path to understand S4 (Structured State Space Sequence Model) and Mamba from fundamentals to implementation. Follow these steps sequentially for the best learning experience.\n\n### 1️⃣ Introduction to Sequence Modeling\n**Goal**: Understand the basics and motivation behind state space models\n\n- 📖 Start with [A Visual Guide to Mamba and State Space Models](https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state)\n  - Focus on [Part 1: The Problem with Transformers](https://newsletter.maartengrootendorst.com/i/141228095/part-the-problem-with-transformers)\n  - This explains why we need alternatives to Transformers and introduces S4\n\n### 2️⃣ Deep Dive into State Space Models (SSMs)\n**Goal**: Master the fundamental concepts of State Space Models\n\n1. 🎥 Watch [State Space Models (SSMs) and Mamba](https://www.youtube.com/watch?v=g1AqUhP00Do) by Serrano.Academy\n   - Provides excellent visualizations and examples of SSMs\n\n2. 📘 Continue with [Part 2: The State Space Model](https://newsletter.maartengrootendorst.com/i/141228095/part-the-state-space-model-ssm)\n   - Builds on the theoretical foundation\n\n3. 💻 Study [The Annotated S4](https://srush.github.io/annotated-s4/#part-1b-addressing-long-range-dependencies-with-hippo)\n   \u003e Note: Since the original repo is outdated, use the [s4_in_depth.ipynb](./s4_in_depth.ipynb) notebook in this repository\n   \u003e\n   \u003e 对于中文读者，我已经将原作者的The Annotated S4翻译成了中文，放在了[cn 文件夹](./cn/The-Annotated-S4-CN-Part1.pdf)中，方便中文读者阅读。\n\n### 3️⃣ Mamba Architecture\n**Goal**: Understand how Mamba builds upon and improves SSMs\n\n- 📖 Read [Part 3: Visual Guide to Mamba](https://newsletter.maartengrootendorst.com/i/141228095/part-mamba-a-selective-ssm)\n  - Explains how Mamba extends and improves upon S4\n\n### 4️⃣ Hands-On Implementation\n**Goal**: Get practical experience with Mamba\n\n- 💻 Study [The Annotated Mamba](https://srush.github.io/annotated-mamba/hard.html)\n  - Detailed implementation walkthrough\n  - Includes code examples and explanations\n\n### 5️⃣ Applications \u0026 Real-World Usage\n**Goal**: Explore practical applications and stay updated\n\n1. 📚 Explore [Awesome-Mamba-Collection](https://github.com/XiudingCai/Awesome-Mamba-Collection?tab=readme-ov-file#head18)\n   - Comprehensive overview of Mamba applications\n   - Covers NLP, time series analysis, and more\n\n2. 🔍 Additional Resources:\n   - [Official Mamba Repository](https://github.com/state-spaces/mamba)\n   - 📄 [Research Paper: Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752)\n\n\n\u003e 💡 **Tip**: Follow this path sequentially for the best learning experience. Each section builds upon the knowledge from previous sections.\n\n\n\n## 🚀 Implementation Example\n\nThis repository contains a PyTorch implementation of the S4 (Structured State Space Sequence Model) for sequence modeling and classification tasks. The implementation focuses on the MNIST dataset with two main tasks:\n\n1. MNIST Sequence Modeling: Predict next pixel value given history (784 pixels x 256 values)\n2. MNIST Classification: Predict digit class using sequence model (784 pixels =\u003e 10 classes)\n\n### Key Components\n\n- **SSM Kernel**: Basic implementation of State Space Model kernel\n- **S4 Kernel**: Advanced implementation with HiPPO-based initialization\n- **Sequence Processing**: Layer normalization, SSM/S4 layer, and MLP components\n- **Model Architecture**: Stacked sequence model with configurable parameters\n\n### Usage\n\n```python\n# Run S4 model for both sequence modeling / classification case\npython train_s4.py\n```\n\n\u003e Notes\n\u003e\n\u003e - Model uses reduced dimensions for demonstration\n\u003e - Supports both CNN and RNN modes\n\nFor detailed implementation, see [train_s4.py](./train_s4.py).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepbiolab%2Fmamba-in-depth","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepbiolab%2Fmamba-in-depth","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepbiolab%2Fmamba-in-depth/lists"}