Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Rishit-dagli/GLOM-TensorFlow
An attempt at the implementation of GLOM, Geoffrey Hinton's paper for emergent part-whole hierarchies from data
https://github.com/Rishit-dagli/GLOM-TensorFlow
capsule-neural-networks convolutional-neural-networks deep-learning embedding geoffrey-hinton glom keras keras-tensorflow machine-learning neural-network representation-learning tensorflow tensorflow2
Last synced: about 1 month ago
JSON representation
An attempt at the implementation of GLOM, Geoffrey Hinton's paper for emergent part-whole hierarchies from data
- Host: GitHub
- URL: https://github.com/Rishit-dagli/GLOM-TensorFlow
- Owner: Rishit-dagli
- License: apache-2.0
- Created: 2021-03-26T14:14:42.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2021-03-27T06:59:14.000Z (almost 4 years ago)
- Last Synced: 2024-11-14T05:41:54.914Z (about 2 months ago)
- Topics: capsule-neural-networks, convolutional-neural-networks, deep-learning, embedding, geoffrey-hinton, glom, keras, keras-tensorflow, machine-learning, neural-network, representation-learning, tensorflow, tensorflow2
- Language: Python
- Homepage: https://pypi.org/project/glom-tf/
- Size: 283 KB
- Stars: 37
- Watchers: 4
- Forks: 5
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# GLOM TensorFlow [](https://twitter.com/intent/tweet?text=Wow:&url=https%3A%2F%2Fgithub.com%2FRishit-dagli%2FGLOM-TensorFlow)
[](https://github.com/Rishit-dagli/GLOM-TensorFlow/actions/workflows/flake8-lint.yml)
[](https://github.com/Rishit-dagli/GLOM-TensorFlow/actions/workflows/python-publish.yml)
[](https://mybinder.org/v2/gh/Rishit-dagli/GLOM-TensorFlow/HEAD)
[](https://colab.research.google.com/github/Rishit-dagli/GLOM-TensorFlow/blob/main/examples/glom_example.ipynb)
[](LICENSE)
[](https://www.python.org/dev/peps/pep-0008/)
[](https://github.com/Rishit-dagli/GLOM-TensorFlow/stargazers)
[](https://github.com/Rishit-dagli)
[](https://twitter.com/intent/follow?screen_name=rishit_dagli)
This Python package attempts to implement GLOM in TensorFlow, which allows advances made by several different groups
transformers, neural fields, contrastive representation learning, distillation and capsules to be combined. This was
suggested by Geoffrey Hinton in his paper
["How to represent part-whole hierarchies in a neural network"](https://arxiv.org/abs/2102.12627).
Further, [Yannic Kilcher's video](https://youtu.be/cllFzkvrYmE) and [Phil Wang's repo](https://github.com/lucidrains/glom-pytorch)
was very helpful for me to implement this project.
## Installation
Run the following to install:
```sh
pip install glom-tf
```
## Developing `glom-tf`
To install `glom-tf`, along with tools you need to develop and test, run the following in your virtualenv:
```sh
git clone https://github.com/Rishit-dagli/GLOM-TensorFlow.git
# or clone your own fork
cd GLOM-TensorFlow
pip install -e .[dev]
```
## A bit about GLOM
The GLOM architecture is composed of a large number of columns which
all use exactly the same weights. Each column is a stack of spatially local
autoencoders that learn multiple levels of representation for what is happening
in a small image patch. Each autoencoder transforms the embedding at one level
into the embedding at an adjacent level using a multilayer bottom-up encoder
and a multilayer top-down decoder. These levels correspond to the levels in a
part-whole hierarchy.
Interactions among the 3 levels in one column
An example shared by the author was as an example when show a face image, a single column might converge on embedding
vectors representing a nostril, a nose, a face, and a person.
At each discrete time and in each column separately, the embedding at a
level is updated to be the weighted average of:
- bottom-up neural net acting on the embedding at the level below at the previous time
- top-down neural net acting on the embedding at the level above at the previous time
- embedding vector at the previous time step
- attention-weighted average of the embeddings at the same level in nearby columns at the previous time
For a static image, the embeddings at a level should settle down over time to produce similar vectors.
A picture of the embeddings at a particular time
## Usage
```python
from glomtf import Glom
model = Glom(dim = 512,
levels = 5,
image_size = 224,
patch_size = 14)
img = tf.random.normal([1, 3, 224, 224])
levels = model(img, iters = 12) # (1, 256, 5, 12)
# 1 - batch
# 256 - patches
# 5 - levels
# 12 - dimensions
```
Use the `return_all = True` argument to get all the column and level states per iteration. This also gives you access
to all the level data across iterations for clustering, from which you can inspect the islands too.
```python
from glomtf import Glom
model = Glom(dim = 512,
levels = 5,
image_size = 224,
patch_size = 14)
img = tf.random.normal([1, 3, 224, 224])
all_levels = model(img, iters = 12, return_all = True) # (13, 1, 256, 5, 12)
# 13 - time
# top level outputs after iteration 6
top_level_output = all_levels[7, :, :, -1] # (1, 256, 512)
# 1 - batch
# 256 - patches
# 512 - dimensions
```
## Want to Contribute 🙋♂️?
Awesome! If you want to contribute to this project, you're always welcome! See [Contributing Guidelines](CONTRIBUTING.md). You can also take a look at [open issues](https://github.com/Rishit-dagli/GLOM-TensorFlow/issues) for getting more information about current or upcoming tasks.
## Want to discuss? 💬
Have any questions, doubts or want to present your opinions, views? You're always welcome. You can [start discussions](https://github.com/Rishit-dagli/GLOM-TensorFlow/discussions).
## Citations
```bibtex
@misc{hinton2021represent,
title = {How to represent part-whole hierarchies in a neural network},
author = {Geoffrey Hinton},
year = {2021},
eprint = {2102.12627},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}
```
## License
```
Copyright 2020 Rishit Dagli
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```