{"id":13738548,"url":"https://github.com/Rishit-dagli/GLOM-TensorFlow","last_synced_at":"2025-05-08T16:34:41.099Z","repository":{"id":53507984,"uuid":"351805079","full_name":"Rishit-dagli/GLOM-TensorFlow","owner":"Rishit-dagli","description":"An attempt at the implementation of GLOM, Geoffrey Hinton's paper for emergent part-whole hierarchies from data","archived":false,"fork":false,"pushed_at":"2021-03-27T06:59:14.000Z","size":290,"stargazers_count":37,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-11-14T05:41:54.914Z","etag":null,"topics":["capsule-neural-networks","convolutional-neural-networks","deep-learning","embedding","geoffrey-hinton","glom","keras","keras-tensorflow","machine-learning","neural-network","representation-learning","tensorflow","tensorflow2"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/glom-tf/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rishit-dagli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2021-03-26T14:14:42.000Z","updated_at":"2024-10-11T16:16:40.000Z","dependencies_parsed_at":"2022-09-04T15:32:56.373Z","dependency_job_id":null,"html_url":"https://github.com/Rishit-dagli/GLOM-TensorFlow","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FGLOM-TensorFlow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FGLOM-TensorFlow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FGLOM-TensorFlow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FGLOM-TensorFlow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rishit-dagli","download_url":"https://codeload.github.com/Rishit-dagli/GLOM-TensorFlow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224727033,"owners_count":17359535,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["capsule-neural-networks","convolutional-neural-networks","deep-learning","embedding","geoffrey-hinton","glom","keras","keras-tensorflow","machine-learning","neural-network","representation-learning","tensorflow","tensorflow2"],"created_at":"2024-08-03T03:02:26.362Z","updated_at":"2024-11-15T07:31:23.139Z","avatar_url":"https://github.com/Rishit-dagli.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# GLOM TensorFlow [![Twitter](https://img.shields.io/twitter/url?style=social\u0026url=https%3A%2F%2Fgithub.com%2FRishit-dagli%2FGLOM-TensorFlow)](https://twitter.com/intent/tweet?text=Wow:\u0026url=https%3A%2F%2Fgithub.com%2FRishit-dagli%2FGLOM-TensorFlow)\n\n![PyPI](https://img.shields.io/pypi/v/glom-tf)\n[![Flake8 Lint](https://github.com/Rishit-dagli/GLOM-TensorFlow/actions/workflows/flake8-lint.yml/badge.svg)](https://github.com/Rishit-dagli/GLOM-TensorFlow/actions/workflows/flake8-lint.yml)\n[![Upload Python Package](https://github.com/Rishit-dagli/GLOM-TensorFlow/actions/workflows/python-publish.yml/badge.svg)](https://github.com/Rishit-dagli/GLOM-TensorFlow/actions/workflows/python-publish.yml)\n![Python Version](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9-blue)\n\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/Rishit-dagli/GLOM-TensorFlow/HEAD)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Rishit-dagli/GLOM-TensorFlow/blob/main/examples/glom_example.ipynb)\n\n[![GitHub license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)\n[![PEP8](https://img.shields.io/badge/code%20style-pep8-orange.svg)](https://www.python.org/dev/peps/pep-0008/)\n[![GitHub stars](https://img.shields.io/github/stars/Rishit-dagli/GLOM-TensorFlow?style=social)](https://github.com/Rishit-dagli/GLOM-TensorFlow/stargazers)\n[![GitHub followers](https://img.shields.io/github/followers/Rishit-dagli?label=Follow\u0026style=social)](https://github.com/Rishit-dagli)\n[![Twitter Follow](https://img.shields.io/twitter/follow/rishit_dagli?style=social)](https://twitter.com/intent/follow?screen_name=rishit_dagli)\n\nThis Python package attempts to implement GLOM in TensorFlow, which allows advances made by several different groups \ntransformers, neural fields, contrastive representation learning, distillation and capsules to be combined. This was \nsuggested by Geoffrey Hinton in his paper \n[\"How to represent part-whole hierarchies in a neural network\"](https://arxiv.org/abs/2102.12627).\n\nFurther, [Yannic Kilcher's video](https://youtu.be/cllFzkvrYmE) and [Phil Wang's repo](https://github.com/lucidrains/glom-pytorch)\nwas very helpful for me to implement this project.\n\n## Installation\n\nRun the following to install:\n\n```sh\npip install glom-tf\n```\n\n## Developing `glom-tf`\n\nTo install `glom-tf`, along with tools you need to develop and test, run the following in your virtualenv:\n\n```sh\ngit clone https://github.com/Rishit-dagli/GLOM-TensorFlow.git\n# or clone your own fork\n\ncd GLOM-TensorFlow\npip install -e .[dev]\n```\n\n## A bit about GLOM\n\nThe GLOM architecture is composed of a large number of columns which\nall use exactly the same weights. Each column is a stack of spatially local\nautoencoders that learn multiple levels of representation for what is happening\nin a small image patch. Each autoencoder transforms the embedding at one level\ninto the embedding at an adjacent level using a multilayer bottom-up encoder\nand a multilayer top-down decoder. These levels correspond to the levels in a\npart-whole hierarchy.\n\n![](images/interactions.png)\n\u003cp align=\"center\"\u003e\n\u003csmall\u003eInteractions among the 3 levels in one column\u003c/small\u003e\n\u003c/p\u003e\n\nAn example shared by the author was as an example when show a face image, a single column might converge on embedding \nvectors representing a nostril, a nose, a face, and a person.\n\nAt each discrete time and in each column separately, the embedding at a\nlevel is updated to be the weighted average of:\n- bottom-up neural net acting on the embedding at the level below at the previous time\n- top-down neural net acting on the embedding at the level above at the previous time\n- embedding vector at the previous time step\n- attention-weighted average of the embeddings at the same level in nearby columns at the previous time\n\nFor a static image, the embeddings at a level should settle down over time to produce similar vectors.\n\n![](images/embeddings.png)\n\u003cp align=\"center\"\u003e\n\u003csmall\u003eA picture of the embeddings at a particular time\u003c/small\u003e\n\u003c/p\u003e\n\n## Usage\n\n```python\nfrom glomtf import Glom\n\nmodel = Glom(dim = 512,\n             levels = 5,\n             image_size = 224,\n             patch_size = 14)\n\nimg = tf.random.normal([1, 3, 224, 224])\nlevels = model(img, iters = 12) # (1, 256, 5, 12)\n# 1 - batch\n# 256 - patches\n# 5 - levels\n# 12 - dimensions\n```\n\nUse the `return_all = True` argument to get all the column and level states per iteration. This also gives you access \nto all the level data across iterations for clustering, from which you can inspect the islands too.\n\n```python\nfrom glomtf import Glom\n\nmodel = Glom(dim = 512,\n             levels = 5,\n             image_size = 224,\n             patch_size = 14)\n\nimg = tf.random.normal([1, 3, 224, 224])\nall_levels = model(img, iters = 12, return_all = True) # (13, 1, 256, 5, 12)\n# 13 - time\n\n# top level outputs after iteration 6\ntop_level_output = all_levels[7, :, :, -1] # (1, 256, 512)\n# 1 - batch\n# 256 - patches\n# 512 - dimensions\n```\n\n## Want to Contribute 🙋‍♂️?\n\nAwesome! If you want to contribute to this project, you're always welcome! See [Contributing Guidelines](CONTRIBUTING.md). You can also take a look at [open issues](https://github.com/Rishit-dagli/GLOM-TensorFlow/issues) for getting more information about current or upcoming tasks.\n\n## Want to discuss? 💬\n\nHave any questions, doubts or want to present your opinions, views? You're always welcome. You can [start discussions](https://github.com/Rishit-dagli/GLOM-TensorFlow/discussions).\n\n## Citations\n\n```bibtex\n@misc{hinton2021represent,\n    title   = {How to represent part-whole hierarchies in a neural network}, \n    author  = {Geoffrey Hinton},\n    year    = {2021},\n    eprint  = {2102.12627},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CV}\n}\n```\n\n## License\n\n```\nCopyright 2020 Rishit Dagli\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRishit-dagli%2FGLOM-TensorFlow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRishit-dagli%2FGLOM-TensorFlow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRishit-dagli%2FGLOM-TensorFlow/lists"}