https://github.com/akshayxml/corrmcnn

Implementation of 'Common Representation Learning Using Step-basedCorrelation Multi-Modal CNN' paper.
https://github.com/akshayxml/corrmcnn

Last synced: 3 months ago
JSON representation

Implementation of 'Common Representation Learning Using Step-basedCorrelation Multi-Modal CNN' paper.

• Host: GitHub
• URL: https://github.com/akshayxml/corrmcnn
• Owner: akshayxml
• Created: 2021-04-07T12:19:17.000Z (about 4 years ago)
• Default Branch: main
• Last Pushed: 2021-05-04T06:02:19.000Z (about 4 years ago)
• Last Synced: 2025-01-25T04:09:47.490Z (4 months ago)
• Language: Jupyter Notebook
• Size: 21 MB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Common Representation Learning using Step-based Correlation Multi-Modal CNN

Implementation of the paper *[Common Representation Learning Using Step-based Correlation Multi-Modal CNN](https://arxiv.org/pdf/1711.00003.pdf)* using both Keras and Pytorch.

# Aim

To make a novel step-based correlation multi-modal CNN(CorrMCNN) which reconstructs one view of the data given the other while increasing the interaction between the representations at each hidden layer or every intermediate step.

# Dataset

- MNIST handwritten digits dataset -60,000 images for training and 10,000 for testing.

- Each image is split vertically into two halves so as to obtain an image of 28 x 14 = 392 features

![Dataset](https://github.com/AkshayViru/CorrMCNN/blob/main/images/dataset.png)

# Technique: Deep Autoencoder based Approach

Multi-Modal Autoencoder is used which is two channeled AE that performs 2 types of reconstructions which provide the ability to adapt towards transfer learning tasks:

- Self-reconstruction of view from itself.

- Cross-reconstruction where one view is reconstructed given the other.

# Implementation

This research paper is an improvement over the *[Correlational Neural Networks](https://arxiv.org/pdf/1504.07225.pdf)* paper with the following additions:

- Introduced convolution layer in the encoding phase and deconvolution layer in the decoding stage of the Correlation multi-modal CNN(CorrMCNN)

- Batch Normalization in the intermediate layers

- Instead of using final hidden representations in the correlation loss, correlation is computed at each intermediate layer.

# Architecture

![CorrMCNN Architecture](https://github.com/AkshayViru/CorrMCNN/blob/main/images/architecture.png)

# Results

![CorrMCNN Architecture](https://github.com/AkshayViru/CorrMCNN/blob/main/images/results_1.JPG)

![CorrMCNN Architecture](https://github.com/AkshayViru/CorrMCNN/blob/main/images/results_2.JPG)