{"id":15036383,"url":"https://github.com/apple/ml-sigmoid-attention","last_synced_at":"2025-04-09T06:04:58.476Z","repository":{"id":256101417,"uuid":"852593640","full_name":"apple/ml-sigmoid-attention","owner":"apple","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-21T16:39:16.000Z","size":7812,"stargazers_count":261,"open_issues_count":2,"forks_count":14,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-04-02T04:45:22.132Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apple.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-05T04:41:45.000Z","updated_at":"2025-03-20T06:07:44.000Z","dependencies_parsed_at":"2024-09-09T02:25:56.421Z","dependency_job_id":"a56cee96-b582-4a8d-9d4d-2fff84aa917b","html_url":"https://github.com/apple/ml-sigmoid-attention","commit_stats":null,"previous_names":["apple/ml-sigmoid-attention"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-sigmoid-attention","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-sigmoid-attention/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-sigmoid-attention/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-sigmoid-attention/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apple","download_url":"https://codeload.github.com/apple/ml-sigmoid-attention/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247987184,"owners_count":21028891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T20:30:59.384Z","updated_at":"2025-04-09T06:04:58.453Z","avatar_url":"https://github.com/apple.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sigmoid Attention\n\nThis repo contains the code associated with [Theory, Analysis, and Best Practices for\nSigmoid Self-Attention](https://arxiv.org/abs/2409.04431)\n\n## Components\n\nThe three components of this release are:\n\n  - [FlashSigmoid](./flash_sigmoid): A hardware aware implementation of Sigmoid Attention.\n  - [Optorch](./optorch): PyTorch-based functional implementation of standard optimizers.\n  - [Attention Simulator](./attention_simulator): A research friendly codebase for diagnosing and debugging attention.\n\n## Installation\n\nSee the `README.md` in the corresponding component for installation and usage instructions.  \n\nWe provide a convenience installation helper for all three packages:\n```bash\n# Create an environment for sigmoid attention, if not done already.\nconda create -n sigmoid-attn-py310 python=3.10\nconda activate sigmoid-attn-py310\n\n# Setup Flashsigmoid -\u003e Optorch -\u003e Attention Simulator.\nbash setup.bash\n```\n\n## Performance\n\n|                                      Forward pass kernels on H100.                                       |                                Backward pass kernels on H100.                                |\n|:--------------------------------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------------------:|\n| ![Sigmoid vs. Softmax Forward Kernels](./figures/H100_noalibi_FWD_Full_17.39_0.07_Causal_18.76_0.06.png) | ![Sigmoid vs. Softmax Backward Kernels](./figures/H100_noalibi_BWD_Full_2.7_0.06_Causal_6.19_0.06.png) |\n\n\n|                   Train losses comparing SigmoidAttn with SoftmaxAttn.                   |\n|:----------------------------------------------------------------------------------------:|\n| ![SigmoidAttn vs. SoftmaxAttn Train Losses](./figures/train_nll_softmax_vs_sigmoid.png) |\n\n\n\n## Citation\nIf you find this work useful in your research, please cite:\n```\n@misc{ramapuram2024theoryanalysisbestpractices,\n      title={Theory, Analysis, and Best Practices for Sigmoid Self-Attention},\n      author={Jason Ramapuram and Federico Danieli and Eeshan Dhekane and Floris Weers and Dan Busbridge and Pierre Ablin and Tatiana Likhomanenko and Jagrit Digani and Zijin Gu and Amitis Shidani and Russ Webb},\n      year={2024},\n      eprint={2409.04431},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={https://arxiv.org/abs/2409.04431},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-sigmoid-attention","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapple%2Fml-sigmoid-attention","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-sigmoid-attention/lists"}