https://github.com/jacklishufan/Mamba-ND

Ofiicial Implementation for Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data
https://github.com/jacklishufan/Mamba-ND

Last synced: 3 months ago
JSON representation

Ofiicial Implementation for Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data

• Host: GitHub
• URL: https://github.com/jacklishufan/Mamba-ND
• Owner: jacklishufan
• Created: 2024-03-12T10:24:54.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2024-07-01T14:49:44.000Z (7 months ago)
• Last Synced: 2024-08-01T04:02:16.360Z (6 months ago)
• Language: Python
• Homepage:
• Size: 279 KB
• Stars: 37
• Watchers: 3
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

• Awesome-state-space-models - Vision

README

        
# Ofiicial Implementation for Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data

## Updates

- Jul 1: Our paper was accepted at ECCV 2024. We also released K400 results and checkpoints.

[Paper](https://arxiv.org/abs/2402.05892)

## Model Zoo

### Image Classification 

Checkpoints available at

| Syntax      | Acc         | Weight |

| ----------- | ----------- |--------| 

| Mamba2D-S/8 | 81.7     | [weight](https://huggingface.co/jacklishufan/Mamba-ND/blob/main/in1k/mamba2d_s.pth)

| Mamba2D-B/8 |  83.0   |   [weight](https://huggingface.co/jacklishufan/Mamba-ND/blob/main/in1k/mamba2d_b.pth) |

### Video Classification 

| Syntax      | Acc | Weight |

| ----------- | ----------- | ----------- |

| UCF-101 (Scratch)  | 89.6       |[weight](https://huggingface.co/jacklishufan/Mamba-ND/blob/main/video/ucf101/ucf101.pth)

| HMDB-51 (Scratch)  |  60.9  | [weight](https://huggingface.co/jacklishufan/Mamba-ND/blob/main/video/hmdb51/hmdb51.pth)

| K400 |  81.9  | [weight](https://huggingface.co/jacklishufan/Mamba-ND/tree/main/k400) |

### 3D Segmentation

| Syntax      | Feature Size |  Dice         | Weight |

| ----------- | ----------- |  ----------- |--------| 

| Mamba3D-S/16 |32|  83.1  |   [weight](https://huggingface.co/jacklishufan/Mamba-ND/blob/main/btcv/mamband-s.pt) |

| Mamba3D-S+/16 |32|  83.9  |   [weight](https://huggingface.co/jacklishufan/Mamba-ND/blob/main/btcv/mamband-s_plus.pt) |

| Mamba3D-B/16 |32|  82.7  |   [weight](https://huggingface.co/jacklishufan/Mamba-ND/blob/main/btcv/mamband-b-32.pt) |

| Mamba3D-B/16 |64|  84.7  |   [weight](https://huggingface.co/jacklishufan/Mamba-ND/blob/main/btcv/mamband-b-64.pt) |

## Environment Setup

```

pip install causal-conv1d>=1.2.0

git install git+https://github.com/state-spaces/mamba.git

```

For image classification, [mmpretrain](https://mmpretrain.readthedocs.io/en/latest/) is required. For video classification, [mmaction](https://mmpretrain.readthedocs.io/en/latest/) is required.  Please see offical documentation for installation instructions.

## Training

Please see refer to the following instructions for each task:

[Image classification](image_classification/readme.MD)

[Video classification](video_classification/readme.MD)

[Video classification (K400 Pretraining)](video_pretraining/readme.MD)

[3D segmentation](btcv/readme.MD)

## Citation

```

@article{li2024mamba,

  title={Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data},

  author={Li, Shufan and Singh, Harkanwar and Grover, Aditya},

  journal={arXiv preprint arXiv:2402.05892},

  year={2024}

}

```