{"id":15293055,"url":"https://github.com/rishit-dagli/perceiver","last_synced_at":"2025-04-13T12:27:02.281Z","repository":{"id":52548198,"uuid":"357042370","full_name":"Rishit-dagli/Perceiver","owner":"Rishit-dagli","description":"Implementation of Perceiver, General Perception with Iterative Attention","archived":false,"fork":false,"pushed_at":"2021-04-26T13:47:11.000Z","size":540,"stargazers_count":88,"open_issues_count":3,"forks_count":9,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-13T12:26:45.114Z","etag":null,"topics":["artificial-intelligence","attention","attention-mechanism","deep-learning","machine-learning","neural-network","python","tensorflow","transformer","transformers"],"latest_commit_sha":null,"homepage":"https://rishit-dagli.github.io/Perceiver/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rishit-dagli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2021-04-12T03:13:09.000Z","updated_at":"2025-02-07T02:41:09.000Z","dependencies_parsed_at":"2022-08-31T02:11:31.783Z","dependency_job_id":null,"html_url":"https://github.com/Rishit-dagli/Perceiver","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FPerceiver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FPerceiver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FPerceiver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FPerceiver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rishit-dagli","download_url":"https://codeload.github.com/Rishit-dagli/Perceiver/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248713369,"owners_count":21149675,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention","attention-mechanism","deep-learning","machine-learning","neural-network","python","tensorflow","transformer","transformers"],"created_at":"2024-09-30T16:38:46.691Z","updated_at":"2025-04-13T12:27:02.256Z","avatar_url":"https://github.com/Rishit-dagli.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Perceiver [![Twitter](https://img.shields.io/twitter/url?style=social\u0026url=https%3A%2F%2Fgithub.com%2FRishit-dagli%2FPerceiver)](https://twitter.com/intent/tweet?text=Wow:\u0026url=https%3A%2F%2Fgithub.com%2FRishit-dagli%2FPerceiver)\n\n![PyPI](https://img.shields.io/pypi/v/perceiver)\n[![Lint with Black⬛](https://github.com/Rishit-dagli/Perceiver/actions/workflows/black.yml/badge.svg)](https://github.com/Rishit-dagli/Perceiver/actions/workflows/black.yml)\n[![Upload Python Package](https://github.com/Rishit-dagli/Perceiver/actions/workflows/python-publish.yml/badge.svg)](https://github.com/Rishit-dagli/Perceiver/actions/workflows/python-publish.yml)\n[![DOI](https://zenodo.org/badge/357042370.svg)](https://zenodo.org/badge/latestdoi/357042370)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n![GitHub License](https://img.shields.io/github/license/Rishit-dagli/Perceiver)\n[![GitHub stars](https://img.shields.io/github/stars/Rishit-dagli/Perceiver?style=social)](https://github.com/Rishit-dagli/Perceiver/stargazers)\n[![GitHub followers](https://img.shields.io/github/followers/Rishit-dagli?label=Follow\u0026style=social)](https://github.com/Rishit-dagli)\n[![Twitter Follow](https://img.shields.io/twitter/follow/rishit_dagli?style=social)](https://twitter.com/intent/follow?screen_name=rishit_dagli)\n\nThis Python package implements [Perceiver: General Perception with Iterative Attention](https://arxiv.org/abs/2103.03206) by Andrew Jaegle in TensorFlow. This model builds on top \nof Transformers such that the data only enters through the cross attention mechanism (see figure) and allow it to scale to hundreds of thousands of inputs, like ConvNets. This, in \npart also solves the Transformers Quadratic compute and memory bottleneck.\n\nYannic Kilcher's [video](https://youtu.be/P_xeshTnPZg) was very helpful.\n\n![](images/architecture.PNG)\n\n## Installation\n\nRun the following to install:\n\n```sh\npip install perceiver\n```\n\n## Developing `perceiver`\n\nTo install `perceiver`, along with tools you need to develop and test, run the following in your virtualenv:\n\n```sh\ngit clone https://github.com/Rishit-dagli/Perceiver.git\n# or clone your own fork\n\ncd perceiver\npip install -e .[dev]\n```\n\n## A bit about Perceiver\n\nThe Perceiver model aims to deal with arbitrary configurations of different modalities using a single transformer-based architecture. Transformers are often flexible and make few assumptions about their inputs, but that also scale quadratically with the number of inputs in terms of both memory and computation. This model proposes a mechanism that makes it possible to deal with high-dimensional inputs, while retaining the expressivity and flexibility to deal with arbitrary input configurations.\n\n![](images/architecture.PNG)\n\nThe idea here is to introduce a small set of latent units that forms an attention bottleneck through which the inputs must pass. This avoids the quadratic scaling problem of all-to-all attention of a classical transformer. The model can be seen as performing a fully end-to-end clustering of the inputs, with the latent units as the cluster centres, leveraging a highly asymmetric crossattention layer. For spatial information the authors compensate for the lack of explicit grid structures in our model by associating Fourier feature encodings.\n\n## Usage\n\n```python\nfrom perceiver import Perceiver\nimport tensorflow as tf\n\nmodel = Perceiver(\n    input_channels = 3,          # number of channels for each token of the input\n    input_axis = 2,              # number of axis for input data (2 for images, 3 for video)\n    num_freq_bands = 6,          # number of freq bands, with original value (2 * K + 1)\n    max_freq = 10.,              # maximum frequency, hyperparameter depending on how fine the data is\n    depth = 6,                   # depth of net\n    num_latents = 256,           # number of latents\n    latent_dim = 512,            # latent dimension\n    cross_heads = 1,             # number of heads for cross attention. paper said 1\n    latent_heads = 8,            # number of heads for latent self attention, 8\n    cross_dim_head = 64,\n    latent_dim_head = 64,\n    num_classes = 1000,          # output number of classes\n    attn_dropout = 0.,\n    ff_dropout = 0.,\n)\n\nimg = tf.random.normal([1, 224, 224, 3]) # replicating 1 imagenet image\nmodel(img) # (1, 1000)\n```\n\n## About the notebooks\n\n### [`perceiver_example`](https://github.com/Rishit-dagli/Perceiver/blob/main/examples/perceiver_example.ipynb)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Rishit-dagli/Perceiver/blob/main/examples/perceiver_example.ipynb)\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/Rishit-dagli/Perceiver/HEAD?filepath=examples%2Fperceiver_example.ipynb)\n\nThis notebook installs the `perceiver` package and shows an example of running it on a single imagenet image (`[1, 224, 224, 3]`) with 1000 classes to demonstarte the working of \nthis model.\n\n## Want to Contribute 🙋‍♂️?\n\nAwesome! If you want to contribute to this project, you're always welcome! See [Contributing Guidelines](CONTRIBUTING.md). You can also take a look at [open issues](https://github.com/Rishit-dagli/Perceiver/issues) for getting more information about current or upcoming tasks.\n\n## Want to discuss? 💬\n\nHave any questions, doubts or want to present your opinions, views? You're always welcome. You can [start discussions](https://github.com/Rishit-dagli/Perceiver/discussions).\n\n## Citations\n\n```bibtex\n@misc{jaegle2021perceiver,\n    title   = {Perceiver: General Perception with Iterative Attention},\n    author  = {Andrew Jaegle and Felix Gimeno and Andrew Brock and Andrew Zisserman and Oriol Vinyals and Joao Carreira},\n    year    = {2021},\n    eprint  = {2103.03206},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CV}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frishit-dagli%2Fperceiver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frishit-dagli%2Fperceiver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frishit-dagli%2Fperceiver/lists"}