https://github.com/reshalfahsi/next-frame-prediction

Next-Frame Prediction Using Convolutional LSTM
https://github.com/reshalfahsi/next-frame-prediction

convolutional-lstm convolutional-lstm-network moving-mnist next-frame-prediction pytorch-lightning

Last synced: about 6 hours ago
JSON representation

Next-Frame Prediction Using Convolutional LSTM

• Host: GitHub
• URL: https://github.com/reshalfahsi/next-frame-prediction
• Owner: reshalfahsi
• Created: 2024-01-28T01:26:19.000Z (10 months ago)
• Default Branch: master
• Last Pushed: 2024-01-28T09:35:33.000Z (10 months ago)
• Last Synced: 2024-01-29T02:32:03.840Z (10 months ago)
• Topics: convolutional-lstm, convolutional-lstm-network, moving-mnist, next-frame-prediction, pytorch-lightning
• Language: Jupyter Notebook
• Homepage:
• Size: 4.08 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Next-Frame Prediction Using Convolutional LSTM



    

    




In the next-frame prediction problem, we strive to generate the subsequent frame of a given video. Inherently, video has two kinds of information to take into account, i.e., image (spatial) and temporal. Using the Convolutional LSTM model, we can manage to feature-extract and process both pieces of information with their inductive biases. In Convolutional LSTM, instead of utilizing fully connected layers within the LSTM cell, convolution operations are adopted. To evaluate the model, the moving MNIST dataset is used. To evalute the model, the Moving MNIST dataset is used.

## Experiment

Have a dive into this [link](https://colab.research.google.com/github/reshalfahsi/next-frame-prediction/blob/master/Next_Frame_Prediction_Using_Convolutional_LSTM.ipynb) and immerse yourself in the next-frame prediction implementation.

## Result

## Quantitative Result

Inspect this table to catch sight of the model's feat.

Test Metric  | Score

------------ | -------------

Loss         | 0.006

MAE          | 0.021

PSNR         | 22.120

SSIM         | 0.881

## Evaluation Metric Curve

  
 The loss curve on the training and validation sets of the Convolutional LSTM model. 


  
 The MAE curve on the training and validation sets of the Convolutional LSTM model. 


  
 The PSNR curve on the training and validation sets of the Convolutional LSTM model. 


  
 The SSIM curve on the training and validation sets of the Convolutional LSTM model. 


## Qualitative Result

This GIF displays the qualitative result of the frame-by-frame prediction of the Convolutional LSTM model.

  
 The Convolutional LSTM model predicts the ensuing frame-by-frame from t = 1 to t = 19. 


## Credit

- [Next-Frame Video Prediction with Convolutional LSTMs](https://keras.io/examples/vision/conv_lstm/)

- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://papers.nips.cc/paper/2015/file/07563a3fe3bbe7e3ba84431ad9d055af-Paper.pdf)

- [On the difficulty of training Recurrent Neural Networks](https://arxiv.org/pdf/1211.5063.pdf)

- [Statistical Language Models Based on Neural Networks](https://www.fit.vutbr.cz/~imikolov/rnnlm/thesis.pdf)

- [Unsupervised Learning of Video Representations using LSTMs](http://www.cs.toronto.edu/~nitish/unsup_video.pdf)

- [Moving MNIST](http://www.cs.toronto.edu/~nitish/unsupervised_video/)

- [Long Short-Term Memory](https://www.bioinf.jku.at/publications/older/2604.pdf)

- [Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition](https://arxiv.org/pdf/1402.1128.pdf)

- [Forked Torchvision by Henry Xia](https://github.com/ehnryx/vision/tree/be6f398c0612c245b0019a286a99f80aca81de7d/torchvision/transforms)

- [PyTorch Lightning](https://lightning.ai/docs/pytorch/latest/)