https://github.com/louishenrifranc/attention
Attention is All You Need in Sonnet
https://github.com/louishenrifranc/attention
estimator google-brain sonnet tensorflow tensorflow-experiments
Last synced: 9 months ago
JSON representation
Attention is All You Need in Sonnet
- Host: GitHub
- URL: https://github.com/louishenrifranc/attention
- Owner: louishenrifranc
- Created: 2017-06-18T20:41:21.000Z (about 9 years ago)
- Default Branch: master
- Last Pushed: 2017-08-30T21:51:27.000Z (almost 9 years ago)
- Last Synced: 2025-01-31T02:41:33.148Z (over 1 year ago)
- Topics: estimator, google-brain, sonnet, tensorflow, tensorflow-experiments
- Language: Python
- Homepage: https://louishenrifranc.github.io/techblog/2017/08/25/attention/
- Size: 85.9 KB
- Stars: 38
- Watchers: 5
- Forks: 12
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Implementation of Attention is All You Need in Sonnet/Tensorflow
Architecture:
Paper:
https://arxiv.org/abs/1706.03762
## Usage
1. Install requirements ```pip install -r requirements.txt```
2. Install [sonnet](https://github.com/deepmind/sonnet)
3. Run ```run_micro_services.sh```
## Organisation of the repository
Transformer's architecture is composed of blocks. The uses of _Sonnet_ makes the implementation very modular, and reusable. I tried to keep the blocks as much decouple as possible, following the paper:
* ```attention/algorithms/transformers```: Create an ```tf.contrib.learn.Experiment```, and the ```tf.contrib.data.Dataset```
* ```attention/modules/cores```: Implementation of the core blocks of Transformer such as ```MultiHeadAttention```, ```PointWiseFeedForward```
* ```attention/modules/decoders```: Implementation of a Decoder block, and a Decoder
* ```attention/modules/encoders```: Implementation of an Encoder block, and an Encoder.
* ```attention/models```: Implementation of a full Transformer Block. This Module is responsible to create the Encoder and the Decoder
* ```attention/services```: Micro Services that create the dataset, or train the model
* ```attention/utils```: Some classes uses as utility (recursive namespace, mocking object)
* ```attention/*/tests/```: Test of the Module/Algorithm/MicroService
### Training Task implemented
- [X] Copy inputs
- [ ] Dialogue generation
## Road Map
- [X] Code modules
- [X] Test modules
- [X] Construct input function
- [X] Build Estimator
- [X] Run estimator
- [X] Plug into a workflow
- [X] Add validation queue
- [ ] Iterate over model improvements