https://github.com/sayakpaul/mlp-mixer-cifar10

Implements MLP-Mixer (https://arxiv.org/abs/2105.01601) with the CIFAR-10 dataset.
https://github.com/sayakpaul/mlp-mixer-cifar10

keras mlp non-conv non-self-attention tensorflow vision

Last synced: 25 days ago
JSON representation

Implements MLP-Mixer (https://arxiv.org/abs/2105.01601) with the CIFAR-10 dataset.

• Host: GitHub
• URL: https://github.com/sayakpaul/mlp-mixer-cifar10
• Owner: sayakpaul
• License: apache-2.0
• Created: 2021-05-24T13:51:22.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2022-06-26T10:10:29.000Z (over 2 years ago)
• Last Synced: 2025-01-09T21:16:20.812Z (28 days ago)
• Topics: keras, mlp, non-conv, non-self-attention, tensorflow, vision
• Language: Jupyter Notebook
• Homepage:
• Size: 7.03 MB
• Stars: 54
• Watchers: 2
• Forks: 7
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# MLP-Mixer-CIFAR10

This repository implements MLP-Mixer as proposed in [MLP-Mixer: An all-MLP Architecture for Vision](https://arxiv.org/abs/2105.01601). The paper introduces an all MLP (Multi-layer Perceptron) architecture for computer vision tasks. Yannic Kilcher walks through the architecture in [this video](https://www.youtube.com/watch?v=7K4Z8RqjWIk). 

Experiments reported in this repository are on CIFAR-10. 

## What's included?

* Distributed training with mixed-precision.

* Visualization of the token-mixing MLP weights.

* A TensorBoard callback to keep track of the learned linear projections of the image patches.

https://user-images.githubusercontent.com/22957388/119498624-71a5f880-bd83-11eb-8b4e-c9e110f6c3db.mov

## Notebooks

* [`MLP_Mixer_Training.ipynb`](https://github.com/sayakpaul/MLP-Mixer-CIFAR10/blob/main/MLP_Mixer_Training.ipynb): MLP-Mixer utilities along with model training. 

* [`ResNet20.ipynb`](https://github.com/sayakpaul/MLP-Mixer-CIFAR10/blob/main/ResNet20.ipynb): Trains a ResNet20 for comparison purposes.

* [`Visualization.ipynb`](https://github.com/sayakpaul/MLP-Mixer-CIFAR10/blob/main/Visualization.ipynb): Visualizes the learned projections and token-mixing MLPs.

**Note**: These notebooks are runnable on Colab. If you don't have access to a tensor-core GPU, please disable the mixed-precision block while running the code. 

## Results

MLP-Mixer achieves competitive results. The figure below summarizes top-1 accuracies on CIFAR-10 test set with respect to varying MLP blocks. 



	




Notable hyperparameters are:

* Image size: 72x72

* Patch size: 9x9

* Hidden dimension for patches: 64

* Hidden dimension for patches: 128

The table below reports the parameter counts for the different MLP-Mixer variants:



	




> ResNet20 (0.571969 Million) achieves **78.14%** under the exact same training configuration. Refer to [this notebook](https://github.com/sayakpaul/MLP-Mixer-CIFAR10/blob/main/ResNet20.ipynb) for more details. 

## Models

You can reproduce the results reported above. The model files are available [here](https://github.com/sayakpaul/MLP-Mixer-CIFAR10/releases/download/Models/models.zip). 

## Acknowledgements

[ML-GDE Program](https://developers.google.com/programs/experts/) for providing GCP credits.