{"id":15601058,"url":"https://github.com/lucidrains/conformer","last_synced_at":"2025-05-16T19:03:27.224Z","repository":{"id":46143345,"uuid":"282713820","full_name":"lucidrains/conformer","owner":"lucidrains","description":"Implementation of the convolutional module from the Conformer paper, for use in Transformers","archived":false,"fork":false,"pushed_at":"2023-05-17T21:08:21.000Z","size":48,"stargazers_count":397,"open_issues_count":6,"forks_count":52,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-05-16T19:02:04.962Z","etag":null,"topics":["artificial-intelligence","audio","deep-learning","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-26T18:56:01.000Z","updated_at":"2025-05-14T08:37:48.000Z","dependencies_parsed_at":"2024-06-20T21:57:05.790Z","dependency_job_id":"04ffde03-ddd2-4bb1-b1eb-46b756315f1b","html_url":"https://github.com/lucidrains/conformer","commit_stats":{"total_commits":24,"total_committers":1,"mean_commits":24.0,"dds":0.0,"last_synced_commit":"fc70d518d3770788d17a5d9799e08d23ad19c525"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fconformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fconformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fconformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fconformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/conformer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254592368,"owners_count":22097011,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","audio","deep-learning","transformer"],"created_at":"2024-10-03T02:13:22.393Z","updated_at":"2025-05-16T19:03:27.204Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"./conformer-conv-module.png\" width=\"600px\"\u003e\u003c/img\u003e\n\n## Conformer\n\nImplementation of the convolutional module from the \u003ca href=\"https://arxiv.org/abs/2005.08100\"\u003eConformer\u003c/a\u003e paper, for improving the local inductive bias in Transformers.\n\n## Install\n\n```bash\n$ pip install conformer\n```\n\n## Usage\n\nThe Conformer convolutional module, the main novelty of the paper\n\n```python\nimport torch\nfrom conformer import ConformerConvModule\n\nlayer = ConformerConvModule(\n    dim = 512,\n    causal = False,             # auto-regressive or not - 1d conv will be made causal with padding if so\n    expansion_factor = 2,       # what multiple of the dimension to expand for the depthwise convolution\n    kernel_size = 31,           # kernel size, 17 - 31 was said to be optimal\n    dropout = 0.                # dropout at the very end\n)\n\nx = torch.randn(1, 1024, 512)\nx = layer(x) + x\n```\n\n1 Conformer Block\n\n```python\nimport torch\nfrom conformer import ConformerBlock\n\nblock = ConformerBlock(\n    dim = 512,\n    dim_head = 64,\n    heads = 8,\n    ff_mult = 4,\n    conv_expansion_factor = 2,\n    conv_kernel_size = 31,\n    attn_dropout = 0.,\n    ff_dropout = 0.,\n    conv_dropout = 0.\n)\n\nx = torch.randn(1, 1024, 512)\n\nblock(x) # (1, 1024, 512)\n```\n\nConformer - just multiple `ConformerBlock` from above\n\n```python\nimport torch\nfrom conformer import Conformer\n\nconformer = Conformer(\n    dim = 512,\n    depth = 12,          # 12 blocks\n    dim_head = 64,\n    heads = 8,\n    ff_mult = 4,\n    conv_expansion_factor = 2,\n    conv_kernel_size = 31,\n    attn_dropout = 0.,\n    ff_dropout = 0.,\n    conv_dropout = 0.\n)\n\nx = torch.randn(1, 1024, 512)\n\nconformer(x) # (1, 1024, 512)\n```\n\n## Todo\n\n- [ ] switch to a better relative positional encoding. shaw's is dated\n- [ ] flash attention with a better RPE\n\n## Citations\n\n```bibtex\n@misc{gulati2020conformer,\n    title   = {Conformer: Convolution-augmented Transformer for Speech Recognition},\n    author  = {Anmol Gulati and James Qin and Chung-Cheng Chiu and Niki Parmar and Yu Zhang and Jiahui Yu and Wei Han and Shibo Wang and Zhengdong Zhang and Yonghui Wu and Ruoming Pang},\n    year    = {2020},\n    eprint  = {2005.08100},\n    archivePrefix = {arXiv},\n    primaryClass = {eess.AS}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fconformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fconformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fconformer/lists"}