https://github.com/eleutherai/improved-t5
Experiments for efforts to train a new and improved t5
https://github.com/eleutherai/improved-t5
Last synced: 4 months ago
JSON representation
Experiments for efforts to train a new and improved t5
- Host: GitHub
- URL: https://github.com/eleutherai/improved-t5
- Owner: EleutherAI
- Created: 2023-03-18T11:12:12.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-04-15T05:51:48.000Z (about 2 years ago)
- Last Synced: 2025-04-24T18:47:58.474Z (about 1 year ago)
- Language: Python
- Size: 42.7 MB
- Stars: 77
- Watchers: 6
- Forks: 5
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Improved T5
## Overview
- `configs/`: Configs for pretraining and finetuning
- `experiments/`: Scripts to launch pretraining and finetuning
- `convert_weights/`: Scripts to convert T5x checkpoints and upload them to HF
- `data/`: Seqio scripts for loading datasets
- `model/`: Gin config for models
- `tpu-scripts/`: Helper scripts for running training jobs on TPUs
## Running Experiments on TPUs
### Setup
Experiments were run on TPUs. The main scripts involve `send.sh`, `run.sh`, `setup.sh` and `kill.sh` in `tpu-scripts/`. To setup a TPU with the required libraries and dependencies,
```
bash send.sh setup.sh
```
Then run
```
bash run.sh "bash setup.sh"
```
To run an pretraining/finetuning job,
```
bash run.sh "source env-t5x/bin/activate; cd improved-t5/; bash "
```
If you need to rerun scripts on the node, you could make sure the node is empty by running
```
bash kill.sh <TPU Node Name>
```
If you are using a different zone than `us-central2-b`, you will need to changed the `--zone` argument in all the scripts.
## Convert Checkpoints to HF
In `convert_weights/` you can use scripts to convert T5x checkpoints.
```
bash scripts/convert_v1.sh MODEL_SIZE /PATH/TO/T5X_CHECPOINTS/ /PATH/TO/HF_CHECPOINTS/
```
```
bash scripts/convert_v2.sh MODEL_SIZE /PATH/TO/T5X_CHECPOINTS/ /PATH/TO/HF_CHECPOINTS/
```
Use absolute paths [issue](https://github.com/huggingface/transformers/issues/15464#issuecomment-1160318564)
You can also directly convert T5x checkpoints and upload them to your HF hub. For that see `convert_weights/upload.sh` and `convert_weights/upload-multiple.sh`.