{"id":15601029,"url":"https://github.com/lucidrains/htm-pytorch","last_synced_at":"2025-07-28T21:08:47.507Z","repository":{"id":60722445,"uuid":"406432863","full_name":"lucidrains/HTM-pytorch","owner":"lucidrains","description":"Implementation of Hierarchical Transformer Memory (HTM) for Pytorch","archived":false,"fork":false,"pushed_at":"2021-09-15T02:23:28.000Z","size":80,"stargazers_count":75,"open_issues_count":1,"forks_count":6,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-07-23T18:53:52.187Z","etag":null,"topics":["artificial-intelligence","attention-mechanism","deep-learning","memory"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-14T15:56:17.000Z","updated_at":"2025-06-22T13:37:59.000Z","dependencies_parsed_at":"2022-10-03T21:01:28.734Z","dependency_job_id":null,"html_url":"https://github.com/lucidrains/HTM-pytorch","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/lucidrains/HTM-pytorch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FHTM-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FHTM-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FHTM-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FHTM-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/HTM-pytorch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FHTM-pytorch/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267585787,"owners_count":24111577,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-28T02:00:09.689Z","response_time":68,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention-mechanism","deep-learning","memory"],"created_at":"2024-10-03T02:12:33.522Z","updated_at":"2025-07-28T21:08:47.456Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"./htm.png\" width=\"500px\"\u003e\u003c/img\u003e\n\n## Hierarchical Transformer Memory (HTM) - Pytorch\n\nImplementation of \u003ca href=\"https://arxiv.org/abs/2105.14039\"\u003eHierarchical Transformer Memory\u003c/a\u003e (HTM) for Pytorch. This Deepmind paper proposes a simple method to allow transformers to attend to memories of the past efficiently. \u003ca href=\"https://github.com/deepmind/deepmind-research/tree/master/hierarchical_transformer_memory\"\u003eOriginal Jax repository\u003c/a\u003e\n\n## Install\n\n```bash\n$ pip install htm-pytorch\n```\n\n## Usage\n\n```python\nimport torch\nfrom htm_pytorch import HTMAttention\n\nattn = HTMAttention(\n    dim = 512,\n    heads = 8,               # number of heads for within-memory attention\n    dim_head = 64,           # dimension per head for within-memory attention\n    topk_mems = 8,           # how many memory chunks to select for\n    mem_chunk_size = 32,     # number of tokens in each memory chunk\n    add_pos_enc = True       # whether to add positional encoding to the memories\n)\n\nqueries = torch.randn(1, 128, 512)     # queries\nmemories = torch.randn(1, 20000, 512)  # memories, of any size\nmask = torch.ones(1, 20000).bool()     # memory mask\n\nattended = attn(queries, memories, mask = mask) # (1, 128, 512)\n```\n\nIf you want the entire HTM Block (which contains the layernorm for the input followed by a skip connection), just import `HTMBlock` instead\n\n```python\nimport torch\nfrom htm_pytorch import HTMBlock\n\nblock = HTMBlock(\n    dim = 512,\n    topk_mems = 8,\n    mem_chunk_size = 32\n)\n\nqueries = torch.randn(1, 128, 512)\nmemories = torch.randn(1, 20000, 512)\nmask = torch.ones(1, 20000).bool()\n\nout = block(queries, memories, mask = mask) # (1, 128, 512)\n```\n\n## Citations\n\n```bibtex\n@misc{lampinen2021mental,\n    title   = {Towards mental time travel: a hierarchical memory for reinforcement learning agents}, \n    author  = {Andrew Kyle Lampinen and Stephanie C. Y. Chan and Andrea Banino and Felix Hill},\n    year    = {2021},\n    eprint  = {2105.14039},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.LG}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fhtm-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fhtm-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fhtm-pytorch/lists"}