https://github.com/lucidrains/esbn-transformer

An attempt to merge ESBN with Transformers, to endow Transformers with the ability to emergently bind symbols
https://github.com/lucidrains/esbn-transformer

artificial-intelligence deep-learning neuro-symbolic-learning

Last synced: about 1 year ago
JSON representation

An attempt to merge ESBN with Transformers, to endow Transformers with the ability to emergently bind symbols

• Host: GitHub
• URL: https://github.com/lucidrains/esbn-transformer
• Owner: lucidrains
• License: mit
• Created: 2021-06-07T20:54:58.000Z (about 5 years ago)
• Default Branch: main
• Last Pushed: 2021-08-03T21:26:42.000Z (almost 5 years ago)
• Last Synced: 2025-05-06T22:09:54.564Z (about 1 year ago)
• Topics: artificial-intelligence, deep-learning, neuro-symbolic-learning
• Language: Python
• Homepage:
• Size: 5.86 KB
• Stars: 15
• Watchers: 4
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
## ESBN Transformer (wip)

An attempt to merge ESBN with Transformers, to endow Transformers with the ability to emergently bind symbols and improve extrapolation. The resulting architecture will be benchmarked with the Give-N task as outlined in this paper, commonly used to assess whether a child has acquired an understanding of counting.

## Usage

```python

import torch

from esbn_transformer import EsbnTransformer

model = EsbnTransformer(

    num_tokens = 256,

    dim = 512,

    depth = 4,

    max_seq_len = 512

)

x = torch.randint(0, 256, (1, 512))

out = model(x) # (1, 512, 256)

```

## Citations

```bibtex

@misc{webb2020emergent,

    title   = {Emergent Symbols through Binding in External Memory}, 

    author  = {Taylor W. Webb and Ishan Sinha and Jonathan D. Cohen},

    year    = {2020},

    eprint  = {2012.14601},

    archivePrefix = {arXiv},

    primaryClass = {cs.AI}

}

```

```bibtex

@misc{dulberg2021modelling,

    title   = {Modelling the development of counting with memory-augmented neural networks}, 

    author  = {Zack Dulberg and Taylor Webb and Jonathan Cohen},

    year    = {2021},

    eprint  = {2105.10577},

    archivePrefix = {arXiv},

    primaryClass = {cs.AI}

}

```