{"id":15674476,"url":"https://github.com/rishit-dagli/compositional-attention","last_synced_at":"2025-05-06T23:11:03.815Z","repository":{"id":53901890,"uuid":"498183595","full_name":"Rishit-dagli/Compositional-Attention","owner":"Rishit-dagli","description":"An implementation of Compositional Attention: Disentangling Search and Retrieval by MILA","archived":false,"fork":false,"pushed_at":"2022-06-01T03:01:46.000Z","size":166,"stargazers_count":14,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-12T02:24:24.691Z","etag":null,"topics":["artificial-intelligence","attention","attention-mechanism","machine-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rishit-dagli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2022-05-31T04:19:17.000Z","updated_at":"2023-06-09T14:41:30.000Z","dependencies_parsed_at":"2022-08-13T03:50:26.414Z","dependency_job_id":null,"html_url":"https://github.com/Rishit-dagli/Compositional-Attention","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FCompositional-Attention","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FCompositional-Attention/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FCompositional-Attention/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FCompositional-Attention/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rishit-dagli","download_url":"https://codeload.github.com/Rishit-dagli/Compositional-Attention/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252782783,"owners_count":21803408,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention","attention-mechanism","machine-learning"],"created_at":"2024-10-03T15:45:28.629Z","updated_at":"2025-05-06T23:11:03.780Z","avatar_url":"https://github.com/Rishit-dagli.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Compositional Attention\n\n![PyPI](https://img.shields.io/pypi/v/compositional-attention)\n[![Upload Python Package](https://github.com/Rishit-dagli/compositional-attention/actions/workflows/python-publish.yml/badge.svg)](https://github.com/Rishit-dagli/compositional-attention/actions/workflows/python-publish.yml)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n[![GitHub stars](https://img.shields.io/github/stars/Rishit-dagli/compositional-attention?style=social)](https://github.com/Rishit-dagli/compositional-attention/stargazers)\n[![GitHub followers](https://img.shields.io/github/followers/Rishit-dagli?label=Follow\u0026style=social)](https://github.com/Rishit-dagli)\n[![Twitter Follow](https://img.shields.io/twitter/follow/rishit_dagli?style=social)](https://twitter.com/intent/follow?screen_name=rishit_dagli)\n\nThis repository is an implementation of [Compositional Attention: Disentangling Search and Retrieval](https://arxiv.org/abs/2110.09419) by MILA. Revisiting standard Multi-head attention through the lens of multiple parallel and independent search and retrieval mechanisms, this leads to static pairings between searches and retrievals, often leading to redundancy of parameters. They reframe the \"heads\" of multi-head attention as \"searches\", and once the multi-headed/searched values are aggregated, there is an extra retrieval step (using attention) off the searched results. The experiments establish this as an easy drop-in replacement for Multi-head attention.\n\n![](media/architecture.PNG)\n\n## Installation\n\nRun the following to install:\n\n```sh\npip install compositional-attention\n```\n\n## Developing `compositional-attention`\n\nTo install `compositional-attention`, along with tools you need to develop and test, run the following in your virtualenv:\n\n```sh\ngit clone https://github.com/Rishit-dagli/compositional-attention.git\n# or clone your own fork\n\ncd compositional-attention\npip install -e .[dev]\n```\n\n## Usage\n\n```py\nimport tensorflow as tf\nfrom compositional_attention import CompositionalAttention\n\nattn = CompositionalAttention(\n    dim = 1024,            # input dimension\n    dim_head = 64,         # dimension per attention 'head' - head is now either search or retrieval\n    num_searches = 8,      # number of searches\n    num_retrievals = 2,    # number of retrievals\n    dropout = 0.1,          # dropout of attention of search and retrieval\n)\n\ntokens = tf.random.uniform([1, 512, 1024])  # tokens\nmask = tf.ones([1, 512], dtype=tf.dtypes.bool)  # mask\n\nout = attn(tokens, mask = mask) # (1, 512, 1024)\n```\n\n## Want to Contribute 🙋‍♂️?\n\nAwesome! If you want to contribute to this project, you're always welcome! See [Contributing Guidelines](CONTRIBUTING.md). You can also take a look at [open issues](https://github.com/Rishit-dagli/Compositional-Attention/issues) for getting more information about current or upcoming tasks.\n\n## Want to discuss? 💬\n\nHave any questions, doubts or want to present your opinions, views? You're always welcome. You can [start discussions](https://github.com/Rishit-dagli/Compositional-Attention/discussions).\n\n## Citations\n\n```bibtex\n@misc{https://doi.org/10.48550/arxiv.2110.09419,\n  doi = {10.48550/ARXIV.2110.09419},\n  \n  url = {https://arxiv.org/abs/2110.09419},\n  \n  author = {Mittal, Sarthak and Raparthy, Sharath Chandra and Rish, Irina and Bengio, Yoshua and Lajoie, Guillaume},\n  \n  keywords = {Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},\n  \n  title = {Compositional Attention: Disentangling Search and Retrieval},\n  \n  publisher = {arXiv},\n  \n  year = {2021},\n  \n  copyright = {arXiv.org perpetual, non-exclusive license}\n}\n```\n\nOfficial [PyTorch implmentation](https://github.com/sarthmit/compositional-attention) and Phil Wang's [PyTorch implmenetation](https://github.com/lucidrains/compositional-attention-pytorch).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frishit-dagli%2Fcompositional-attention","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frishit-dagli%2Fcompositional-attention","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frishit-dagli%2Fcompositional-attention/lists"}