{"id":19459826,"url":"https://github.com/jsflo/capsnet","last_synced_at":"2026-04-09T21:02:04.259Z","repository":{"id":187794757,"uuid":"113635393","full_name":"JsFlo/CapsNet","owner":"JsFlo","description":"A Tensorflow implementation of a Capsule Network that allows you to separate/export the decoder and modify or tweak the dimensions on an Android app","archived":false,"fork":false,"pushed_at":"2018-01-03T02:02:42.000Z","size":23905,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-08T02:16:17.132Z","etag":null,"topics":["android","capsule-net","capsule-network","capsule-networks","kotlin","kotlin-android","machine-learning","machine-learning-algorithms","python","python3","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JsFlo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-12-09T02:29:00.000Z","updated_at":"2022-11-12T03:24:15.000Z","dependencies_parsed_at":"2023-08-12T06:52:19.211Z","dependency_job_id":null,"html_url":"https://github.com/JsFlo/CapsNet","commit_stats":null,"previous_names":["jsflo/capsnet"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JsFlo%2FCapsNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JsFlo%2FCapsNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JsFlo%2FCapsNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JsFlo%2FCapsNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JsFlo","download_url":"https://codeload.github.com/JsFlo/CapsNet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240664481,"owners_count":19837563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["android","capsule-net","capsule-network","capsule-networks","kotlin","kotlin-android","machine-learning","machine-learning-algorithms","python","python3","tensorflow"],"created_at":"2024-11-10T17:34:12.888Z","updated_at":"2026-04-09T21:02:04.145Z","avatar_url":"https://github.com/JsFlo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CapsNetTweak\nAndroid App: https://play.google.com/store/apps/details?id=fhc.tfsandbox.capsnettweak\n\n\nA Tensorflow implementation of the recently published Capsule Network that allows you to separate/export the `decoder`.\n\nCapsule Network: \u003cimg src=\"images/fig1.png\" width=\"200\"\u003e\n\nDecoder:  \u003cimg src=\"images/fig2.png\" width=\"200\"\u003e  \n\nThe `decoder` is found at the end of Capsule Networks and used to recreate the source images. The capsule network is trained taking this **reconstruction loss** into consideration which means that the Capsule Network will learn to give capsules meaningful parameters.\n\nThe paper, [*Dynamic Routing Between Capsules*](https://arxiv.org/abs/1710.09829), shows some modification of these Capsule dimensions/parameters.\n\n\u003cimg src=\"images/fig4.png\" width=\"600\"\u003e\n\nAfter exporting the `decoder` (freezing and exporting only the data/graph structure associated with it) I created an Android Application(**./CapsNetTweak**) that can import the model and run inference on the device.\n\nThe Android app gives a user the ability to modify or *tweak* any dimension in a Capsule\nand run inference on the modified capsule to see the recreation.\n\n\u003cimg src=\"images/capsule_outputs.png\" width=\"200\"\u003e\u003cimg src=\"images/reconstruction.png\" width=\"200\"\u003e\u003cimg src=\"images/reconstruction_modified.png\" width=\"200\"\u003e\n\n\nHopefully the modification and recreation of the images will give some insights into what the model learned for each dimension in a Capsule\n\nGoals of the project:\n  * Implement and understand a Capsule Network\n  * Experience++ with Kotlin and python\n  * Sharing a database between python and Kotlin (Android App)\n  * Saving models and checkpoints in Tensorflow\n  * Separating/Extracting only specific sections of a fully trained model\n\n\nThe new algorithm is described in the recently published papers([*Dynamic Routing Between Capsules*](https://arxiv.org/abs/1710.09829), [*Matrix Capsules with EM Routing*](https://arxiv.org/abs/1710.09829)) by Geoffrey Hinton and team.\n\n## Flow overview\n1. Train a model and get a **checkpoint file**\n2. Separate and **export** only the **decoder** section of the Capsule Network\n3. Use the **decoder** and some **capsules** to reconstruct an image based on the capsules on the Android device\n\n## Getting Started\n\nThese instructions should help you train your own model and export the `minimal_decoder` after training.\n\n *The Android instructions will be in the README found in the CapsNetTweak folder*\n\n### Project overview\n\n* Common/frequent Scripts\n  * `train.py` - Train a model and **output** a **checkpoint file**\n  * `prediction.py` - Uses a **checkpoint file** to show source images vs prediction images\n  * `evaluation.py` - Evaluates a **checkpoint file**: Tests the model against the MNIST validation set\n* `caps_net_model/`\n  * `model.py` - Creating the full model and exposes different graph operations for different scripts\n  * `digit_capsules.py` - Create the digit_capsules\n  * `primary_capsules.py` - Create the primary_capsules\n  * `decoder.py` - Creates the decoder at the end (3 FC layers)\n* `minimal_decoder/`\n  * `export_minimal_decoder.py` - Takes the **checkpoint file** and outputs just the minimal_decoder **graph** (.pb file)\n  * `minimal_decoder_inference.py` - Runs inference on the exported **graph**\n* `CapsNetTweak` - Takes the **graph** exported and imports into an Android App(*more info in subfolder README*)\n### Prerequisites\n\nFor the python training phase we will need:\n\n* Most of the python scripts depend on\n\n  ```\n  from __future__ import division, print_function, unicode_literals\n  import numpy\n  import tensorflow\n  import argparse\n  ```\n* `export_minimal_decoder.py` depends on\n  ```\n  import matplotlib.pyplot as plt\n  import io\n  import scipy.misc\n  import sys\n  import sqlite3\n  ```\n## Training\n\nTraining will be done with `train.py` which will output a **checkpoint** file.\n\n#### Args\n* Required\n  * Checkpoint path - Where to save the checkpoints and/or where to look for a checkpoint if you want to continue training an old checkpoint\n    * `python train.py --checkpoint_path=./my_checkpoints`\n  * Checkpoint Name - What to name the checkpoint\n    * `python train.py --checkpoint_path=./my_checkpoints --checkpoint_name=my_awesome_checkpoint`\n* Optional\n  * Restore Checkpoint - Flag used to restore training from an old checkpoint if applicable (*if there's an old checkpoint*)\n    *  `python train.py --checkpoint_path=./my_checkpoints --checkpoint_name=my_awesome_checkpoint --restore_checkpoints=True`\n    * Default: True\n  *  Training batch size -\n    * `python train.py --checkpoint_path=./my_checkpoints --checkpoint_name=my_awesome_checkpoint --batch_size=50`\n    * Default: 50\n  * Number of epochs -\n    * `python train.py --checkpoint_path=./my_checkpoints --checkpoint_name=my_awesome_checkpoint --n_epochs=10`\n    * Default: 10\n\n\n#### Outputs:\nAfter running the training script the output should be a **checkpoint file** in the directory specified.\n\n#### Example\n`python train.py --checkpoint_path=./my_checkpoints --checkpoint_name=my_awesome_checkpoint`\n\n**IMPORTANT**\nSo the path to our checkpoint is `./my_checkpoints/my_awesome_checkpoint` but if you\nlook in the folder you'll see there's not **a** file with that name.\n```\n.\n└── my_checkpoints\n    ├── checkpoint\n    ├── my_awesome_checkpoint.data-00000-of-00001\n    ├── my_awesome_checkpoint.index\n    └── my_awesome_checkpoint.meta\n\n```\n\nTensorflow exports checkpoints like this and it is used to separate the graph from the actual values but the only thing we need to worry about is that when we are **passing** the **checkpoint file** to the other scripts we need to just reference the model name.\n\n**So don't do this**\n```\nsomeScriptThatNeedsCheckpoints --model_path=./my_checkpoints/my_awesome_checkpoint.data-00000-of-00001\nsomeScriptThatNeedsCheckpoints --model_path=./my_checkpoints/my_awesome_checkpoint.index\nsomeScriptThatNeedsCheckpoints --model_path=./my_checkpoints/my_awesome_checkpoint.meta\nsomeScriptThatNeedsCheckpoints --model_path=./my_checkpoints/my_awesome_checkpoint.\n```\n**Do this**\n```\nsomeScriptThatNeedsCheckpoints --model_path=./my_checkpoints/my_awesome_checkpoint\n```\n\n## Testing\nWe can then evaluate the model performance by running `evaluation.py`.\nWe can also see the **reconstructed** image **vs** the **source images** using `prediction.py`.\n\nBoth of these scripts depend on a **checkpoint file** (*trained above using train.py*)\n#### Example usage\n`python evaluation.py --checkpoint=./my_checkpoints/my_awesome_checkpoint --batch_size=10`\n\n`python prediction.py --checkpoint=./my_checkpoints/my_awesome_checkpoint --batch_size=10`\n\n#### Outputs\n`prediction.py` outputs the **generalization error** (*tested against MNIST validation set*): 99.4!\n\n`evaluation.py` outputs an image comparison of the source image and reconstruction\n\n\u003cimg src=\"images/label_prediction.png\" width=\"600\"\u003e\n\n## Extracting the decoder\n`export_minimal_decoder.py`\n\nAt this point we have a **checkpoint file** and we want to export a **minimal decoder**.\n\nThis means that we will recreate a graph for only the `decoder` and freeze *only* the weights that correspond to the decoder. This keeps the graph and data small enough to throw into an Android app.\n\n#### Args\n* Required\n  * checkpoint - checkpoint file\n    * `export_minimal_decoder.py --checkpoint=./my_checkpoints/my_awesome_checkpoint`\n* Optional\n  * model output - Where to save the model **graph** and **example source and capsules** (database and numpy array files)\n    * `export_minimal_decoder.py --checkpoint=./my_checkpoints/my_awesome_checkpoint  --model_output=./model_output`\n  * model name -\n    * `export_minimal_decoder.py --checkpoint=./my_checkpoints/my_awesome_checkpoint --model_name=\"my_model_graph.pb\"`\n  * masked capsules - number of examples to export with the graph\n    * `export_minimal_decoder.py --checkpoint=./my_checkpoints/my_awesome_checkpoint --masked_capsules=25`\n\n#### Outputs\n`export_minimal_decoder.py` will output a **model graph (.pb)** and a **database** that will be shared with the Android app.\n\nThe **database** will have **source images (MNIST Images)** as well as the **capsules** produced by the full Capsule Network.\n```\n.\n├── database\n│   └── minimal_decoder.db\n├── masked_capsules\n│   └── masked_output.npy\n├── model_graph.pb\n└── source_images\n    └── source_images.npy\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsflo%2Fcapsnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjsflo%2Fcapsnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsflo%2Fcapsnet/lists"}