{"id":13454793,"url":"https://github.com/gyang274/capsulesEM","last_synced_at":"2025-03-24T06:31:28.987Z","repository":{"id":65005245,"uuid":"111111150","full_name":"gyang274/capsulesEM","owner":"gyang274","description":"A tensorflow implementation of Hinton's [matrix capsules with EM routing](https://openreview.net/pdf?id=HJWLfGWRb)","archived":false,"fork":false,"pushed_at":"2018-01-22T17:21:51.000Z","size":11383,"stargazers_count":62,"open_issues_count":5,"forks_count":31,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-28T21:41:47.822Z","etag":null,"topics":["capsnet","capsnet-tensorflow","capsule","capsule-net","capsule-network","capsule-networks","capsules","capsules-em","capsules-net","hinton","matrix-capsules","tensorflow"],"latest_commit_sha":null,"homepage":"https://gyang274.github.io/capsulesEM/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gyang274.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-17T14:14:58.000Z","updated_at":"2024-05-19T06:31:19.000Z","dependencies_parsed_at":"2022-12-21T06:16:25.253Z","dependency_job_id":null,"html_url":"https://github.com/gyang274/capsulesEM","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyang274%2FcapsulesEM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyang274%2FcapsulesEM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyang274%2FcapsulesEM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyang274%2FcapsulesEM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gyang274","download_url":"https://codeload.github.com/gyang274/capsulesEM/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245222566,"owners_count":20580185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["capsnet","capsnet-tensorflow","capsule","capsule-net","capsule-network","capsule-networks","capsules","capsules-em","capsules-net","hinton","matrix-capsules","tensorflow"],"created_at":"2024-07-31T08:00:57.993Z","updated_at":"2025-03-24T06:31:27.712Z","avatar_url":"https://github.com/gyang274.png","language":"Python","funding_links":[],"categories":["Implementations"],"sub_categories":["Tensorflow"],"readme":"# Capsule\n\nA Tensorflow Implementation of Hinton's __[Matrix Capsules with EM Routing](https://openreview.net/pdf?id=HJWLfGWRb)__.\n\n## Quick Start\n\n```\n$ git clone https://github.com/gyang274/capsulesEM.git \u0026\u0026 cd capsulesEM\n\n$ cd src\n\n$ python train.py\n\n# open a new terminal (ctrl + alt + t)\n\n$ python tests.py\n```\n\nNote: \n\n1. Tensorflow v1.4.0.\n\n2. This `train.py` and `tests.py` assumes the user have 2 GPU card: `train.py` will use the first GPU card, and `tests.py` will use the second one. In case a different setting required, or multiple GPUs are available for training, modify `visible_device_list` in `session_config` in `slim.learning.train()` in `train.py`, or modify `visible_device_list` in `session_config` in `slim.evaluation.evaluation_loop()` in `tests.py`.\n\n## Status\n\n### MNIST\n\n1. (R0I1) Network architecture same as in paper, Matrix Capsules with EM Routing, Figure 1.\n\n    - Spread loss only, no reconstruction loss.\n    \n    - Adam Optimizer, learning rate default 0.001, no learning rate decay. \n    \n    - Batch size 24 (due to limit of GPU memory), iteration 1. \n    \n    - GPU: half K80 12GB memory, 2s-3s per training step.\n\n    - Step: 43942, Test Accuracy: __99.37%__.\n\n    ![Screenshot Tensorboard](doc/src/fig/capsulesEM-V0-R0I1-screenshot-eval-loss.png)\n    \n    __Remark__: Because of `allow_smaller_final_batch=False` and `batch_size=24`, test is running on a random sample 9984 of 10000, so worse case test accuracy could be 99.21%. Modify the `src/datasets/mnist.py` and `src/test.py` to run test on full test dataset.\n\n1. (R0I2) As above, except iteration 2. (TODO)\n\n1. (R1I2) As above, add reconstruction loss, iteration 2. (TODO)\n\n## Matrix Capsules Nets and Layers\n\nBuild a matrix capsules neural network as the same way of building CNN:\n\n```\ndef capsules_net(inputs, num_classes, iterations, name='CapsuleEM-V0'):\n  \"\"\"Replicate the network in `Matrix Capsules with EM Routing.`\n  \"\"\"\n\n  with tf.variable_scope(name) as scope:\n\n    # inputs [N, H, W, C] -\u003e conv2d, 5x5, strides 2, channels 32 -\u003e nets [N, OH, OW, 32]\n    nets = _conv2d_wrapper(\n      inputs, shape=[5, 5, 1, 32], strides=[1, 2, 2, 1], padding='SAME', add_bias=True, activation_fn=tf.nn.relu, name='conv1'\n    )\n    # inputs [N, H, W, C] -\u003e conv2d, 1x1, strides 1, channels 32x(4x4+1) -\u003e (poses, activations)\n    nets = capsules_init(\n      nets, shape=[1, 1, 32, 32], strides=[1, 1, 1, 1], padding='VALID', pose_shape=[4, 4], name='capsule_init'\n    )\n    # inputs: (poses, activations) -\u003e capsule-conv 3x3x32x32x4x4, strides 2 -\u003e (poses, activations)\n    nets = capsules_conv(\n      nets, shape=[3, 3, 32, 32], strides=[1, 2, 2, 1], iterations=iterations, name='capsule_conv1'\n    )\n    # inputs: (poses, activations) -\u003e capsule-conv 3x3x32x32x4x4, strides 1 -\u003e (poses, activations)\n    nets = capsules_conv(\n      nets, shape=[3, 3, 32, 32], strides=[1, 1, 1, 1], iterations=iterations, name='capsule_conv2'\n    )\n    # inputs: (poses, activations) -\u003e capsule-fc 1x1x32x10x4x4 shared view transform matrix within each channel -\u003e (poses, activations)\n    nets = capsules_fc(\n      nets, num_classes, iterations=iterations, name='capsule_fc'\n    )\n\n    poses, activations = nets\n\n  return poses, activations\n```\n\nIn particular,\n\n- `capsules_init()` takes a CNN layer as inputs, and produces a matrix capsule layer (e.g., primaryCaps) as output. \n\n    This operation is corresponding to the layer `A -\u003e B` in the paper.\n\n- `capsules_conv()` takes a matrix capsule layer (e.g., primaryCaps, ConvCaps1) as inputs, and produces a matrix capsule layer (e.g., ConvCaps1, ConvCaps2) as output.\n\n    This operation is corresponding to the layer `B -\u003e C` and `C -\u003e D` in the paper.\n \n- `capsules_fc()` takes a matrix capsule layer (e.g., ConvCaps2) as inputs, and produces an output matrix capsule layer with poses and activations (e.g., Class Capsules) as output. \n    \n    This operation is correponding to the layer `D -\u003e E` in the paper. \n\n## TODO\n\n1. How `tf.stop_gradient()` in EM? How iteration \u003e 1 cause NaN in loss and capsules_init() activations?\n\n1. Add `learning_rate decay` in `train.py`\n\n1. Add train.py/tests.py on smallNORB.\n\n## Questions\n\n1. $$\\lambda$$ schedule is never mentioned in paper.\n\n1. The place encode in lower level and rate encode in higher level is not discussed, other than a coordinate addition in last layer.\n\n## GitHub Page\n\nThis [`gh-pages`](https://gyang274.github.io/capsulesEM/) includes all notes.\n\n## GitHub Repository\n\nThis [github repository](https://github.com/gyang274/capsulesEM) includes all source codes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgyang274%2FcapsulesEM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgyang274%2FcapsulesEM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgyang274%2FcapsulesEM/lists"}