https://github.com/dohlee/tranception-pytorch
Implementation of Tranception, a SOTA transformer model for protein fitness prediction, in PyTorch.
https://github.com/dohlee/tranception-pytorch
bioinformatics biology computational-biology deep-learning protein-sequences reproduction reproduction-code
Last synced: about 1 month ago
JSON representation
Implementation of Tranception, a SOTA transformer model for protein fitness prediction, in PyTorch.
- Host: GitHub
- URL: https://github.com/dohlee/tranception-pytorch
- Owner: dohlee
- Created: 2023-02-22T15:45:50.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2023-03-18T14:43:25.000Z (about 2 years ago)
- Last Synced: 2024-10-04T20:36:11.143Z (8 months ago)
- Topics: bioinformatics, biology, computational-biology, deep-learning, protein-sequences, reproduction, reproduction-code
- Language: Jupyter Notebook
- Homepage:
- Size: 166 KB
- Stars: 3
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# tranception-pytorch
Implementation of Tranception, a SOTA transformer model for protein fitness prediction, in PyTorch. The official implementation of tranception can be found [here](https://github.com/OATML-Markslab/Tranception).
## Installation
```bash
$ pip install tranception-pytorch-dohlee
```
## Usage
```python
import torch
from tranception_pytorch import Tranception
# Parameters for Tranception S
num_heads = 12
num_layers = 12
embed_dim = 768
max_length = 1024
model = Tranception(
embed_dim=embed_dim,
num_heads=num_heads,
num_layers=num_layers,
max_length=max_length,
)
model = model.cuda()
x = torch.randn([1, 1000]) # batch, seq_len (encoded as integer in [0, 21], 21 for [MASK])
# Autoregressive inference
out = model.log_likelihood(x)
```
## Reproduction status
Live training metrics for Tranception S model using UniRef50 can be found [here](https://wandb.ai/dohlee/tranception-pytorch/reports/Tranception-training--VmlldzozODIyODI0).
## Citation
```bibtex
@inproceedings{notin2022tranception,
title={Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval},
author={Notin, Pascal and Dias, Mafalda and Frazer, Jonathan and Hurtado, Javier Marchena and Gomez, Aidan N and Marks, Debora and Gal, Yarin},
booktitle={International Conference on Machine Learning},
pages={16990--17017},
year={2022},
organization={PMLR}
}
```