{"id":20520286,"url":"https://github.com/yizhongw/tk-instruct","last_synced_at":"2025-08-21T02:32:14.229Z","repository":{"id":39965174,"uuid":"486317041","full_name":"yizhongw/Tk-Instruct","owner":"yizhongw","description":"Tk-Instruct is a Transformer model that is tuned to solve many NLP tasks by following instructions.","archived":false,"fork":false,"pushed_at":"2022-10-28T18:57:08.000Z","size":10415,"stargazers_count":178,"open_issues_count":12,"forks_count":27,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-12-10T20:24:53.839Z","etag":null,"topics":["cross-task-generalization","few-shot-learning","instruction","transformer-models","zero-shot-learning"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2204.07705","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yizhongw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-27T18:59:44.000Z","updated_at":"2024-11-24T14:59:47.000Z","dependencies_parsed_at":"2023-01-19T20:17:51.798Z","dependency_job_id":null,"html_url":"https://github.com/yizhongw/Tk-Instruct","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yizhongw%2FTk-Instruct","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yizhongw%2FTk-Instruct/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yizhongw%2FTk-Instruct/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yizhongw%2FTk-Instruct/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yizhongw","download_url":"https://codeload.github.com/yizhongw/Tk-Instruct/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230479864,"owners_count":18232630,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cross-task-generalization","few-shot-learning","instruction","transformer-models","zero-shot-learning"],"created_at":"2024-11-15T22:18:59.223Z","updated_at":"2024-12-19T18:18:13.028Z","avatar_url":"https://github.com/yizhongw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tk-Instruct\n\n- This repo releases our implementation for the Tk-Instruct model in the [Super-NaturalInstructions paper](https://arxiv.org/abs/2204.07705).\n- Tk-Instruct is a preliminary attempt towards general-purpose AI that can solve many NLP tasks by following in-context instructions (plain language task definitions or k-shot examples).\n- It is built based on the pretrained [T5 model](https://arxiv.org/abs/1910.10683), and finetuned on our [data](https://github.com/allenai/natural-instructions).\n- You can play with the 11B model via our online [demo](https://instructions.apps.allenai.org/demo)!\n\n## Requirements\n\nOur main experiments and analysis are conducted on the following environment:\n\n- CUDA (11.3)\n- cuDNN (8.2.0.53)\n- Pytorch (1.10.0)\n- Transformers (4.17.0)\n- DeepSpeed\n\nYou can refer to the [Dockerfile](Dockerfile) for setting up the environment and install the required python libraries by running\n\n```bash\npip install -r requirements.txt\n```\n\nNote: after the main exploration with 3B model, we train our 11B model on TPUs using the T5 code [here](https://github.com/google-research/text-to-text-transfer-transformer).\n\n## Data\n\nOur models are trained and evaluated on [Super-NaturalInstructions](https://github.com/allenai/natural-instructions), which can be cloned by running:\n\n```bash\ngit clone git@github.com:allenai/natural-instructions.git data\n```\n\nSince Super-NaturalInstructions didn't provide an official split for the development set, in order to do evaluation during training time, you can mannualy create a `dev_tasks.txt` in the `data/splits/default` folder. We found it unclear what should be a meaningful validation set, under such cross-task generalization setting. You can use a part of the training tasks for validation, or you can set apart tasks in some categories for validation.\n\nIf you want to use the T5 code [here](https://github.com/google-research/text-to-text-transfer-transformer), you can convert the data into text2text format with [`scripts/convert_data_to_s2s.sh`](scripts/convert_data_to_s2s.sh).\n\n## Training\n\nA sample script for training the Tk-Instruct 3B model in our paper can be found at [`scripts/train_tk_instruct.sh`](scripts/train_tk_instruct.sh). You can run it as follows:\n\n```bash\n./scripts/train_tk_instruct.sh\n```\n\nHowever, if you are familiar with [Beaker](https://beaker.org/), you can refer to the [`beaker_configs/default_experiment.yaml`](beaker_configs/default_experiment.yaml) for a sample experiment config, and modifying [`src/create_exps.py`](src/create_exps.py) to easily starts a set of experiments by running:\n\n```bash\npython src/create_exps.py\n```\n\n## Released Checkpoints\n\nOur 3B and 11B model checkpoints are accessible via the [Hugging Face Hub](https://huggingface.co/models?search=tk-instruct-). You can load them easily using the [Transformers](https://github.com/huggingface/transformers) library:\n\n```python\n\u003e\u003e\u003e from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n\n\u003e\u003e\u003e tokenizer = AutoTokenizer.from_pretrained(\"allenai/tk-instruct-3b-def\")\n\u003e\u003e\u003e model = AutoModelForSeq2SeqLM.from_pretrained(\"allenai/tk-instruct-3b-def\")\n\n\u003e\u003e\u003e input_ids = tokenizer.encode(\n        \"Definition: return the currency of the given country. Now complete the following example - Input: India. Output:\", \n        return_tensors=\"pt\"\n    )\n\u003e\u003e\u003e output = model.generate(input_ids, max_length=10)\n\u003e\u003e\u003e output = tokenizer.decode(output[0], skip_special_tokens=True)\n```\n\nThe model should generate `'Indian Rupee'` as the output.\n\n## Evaluation\n\nThe following script evaluates our 3B Tk-Instruct model that uses `task definition + 2 positive examples` as instructions:\n\n```bash\n./scripts/eval_tk_instruct.sh\n```\n\nThis should give you a ROUGE-L score of ~54.0, as is reported in the Table 3 of our [paper](https://arxiv.org/pdf/2204.07705.pdf).\n\nYou can also try other models under different encodings. You can control whether to include definition / explanation, or the number of pos/neg examples, by specifying the arguments in [`src/run_s2s.py`](src/run_s2s.py).\n\nThe numbers for heuristic baselines and GPT3 can be reproduced by using the following scripts:\n\n```bash\n./scripts/run_heuristics.sh\n./scripts/run_gpt3.sh\n```\n\n## Model Predictions and Performance\n\nThe predictions of our tested models can be found in the [`output`](output/) folder. You can evaluate each predition file in the following way:\n\n```bash\npython src/compute_metrics.py --predictions output/default/tk-instruct-3b-def-pos/predicted_examples.jsonl --track default --compute_per_category_metrics\npython src/compute_metrics.py --predictions output/xlingual/mtk-instruct-3b-def-pos/predicted_examples.jsonl --track xlingual --compute_per_category_metrics\n```\n\nHere are the performance numbers (in ROUGE-L) for our tested models:\n\n|                          | Models                  | Default Track (en) | X-lingual Track |\n|--------------------------|-------------------------|--------------------|-----------------|\n| Heuristic Baselines      | Copying Instance Input  | 14.20              | 5.44            |\n|                          | Copying Demo. Output    | 28.54              | 50.31           |\n| Pretrained LMs           | T5-LM (11B)             | 30.16              | -               |\n|                          | GPT3 (175B)             | 45.05              | 51.20           |\n| Instruction-tuned Models | T0 (11B)                | 32.28              | -               |\n|                          | GPT3-Instruct (175B)    | 52.06              | 53.74           |\n|                          | Tk-Instruct (Ours, 3B)  | 54.33              | -               |\n|                          | Tk-Instruct (Ours, 11B) | 60.07              | -               |\n|                          | mTk-Instruct (Ours, 3B) | -                  | 56.72           |\n\nNote that these numbers might be different from the numbers reported in the our arxiv paper, because we 1) resampled our evaluation instances; 2) updated our evaluation script. We will update the paper once allowed.\n\nWe will keep adding the predictions and performance of new models into this repository.\n\n## Citation\n\n```bib\n@inproceedings{supernaturalinstructions,\n  title={Super-NaturalInstructions:Generalization via Declarative Instructions on 1600+ Tasks},\n  author={Wang, Yizhong and Mishra, Swaroop and Alipoormolabashi, Pegah and Kordi, Yeganeh and Mirzaei, Amirreza and Arunkumar, Anjana and Ashok, Arjun and Dhanasekaran, Arut Selvan and Naik, Atharva and Stap, David and others},\n  booktitle={EMNLP},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyizhongw%2Ftk-instruct","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyizhongw%2Ftk-instruct","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyizhongw%2Ftk-instruct/lists"}