{"id":13795273,"url":"https://github.com/allenai/unifiedqa","last_synced_at":"2025-10-13T15:57:40.446Z","repository":{"id":41221863,"uuid":"258283918","full_name":"allenai/unifiedqa","owner":"allenai","description":"UnifiedQA: Crossing Format Boundaries With a Single QA System","archived":false,"fork":false,"pushed_at":"2022-05-09T20:51:31.000Z","size":29565,"stargazers_count":428,"open_issues_count":4,"forks_count":43,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-11-18T09:33:37.066Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2005.00700","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/allenai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-23T17:41:56.000Z","updated_at":"2024-10-17T11:51:19.000Z","dependencies_parsed_at":"2022-08-23T01:00:39.934Z","dependency_job_id":null,"html_url":"https://github.com/allenai/unifiedqa","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Funifiedqa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Funifiedqa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Funifiedqa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Funifiedqa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/allenai","download_url":"https://codeload.github.com/allenai/unifiedqa/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253833077,"owners_count":21971357,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T23:00:54.115Z","updated_at":"2025-10-13T15:57:40.335Z","avatar_url":"https://github.com/allenai.png","language":"Python","funding_links":[],"categories":["Datasets and Models","Python"],"sub_categories":["Modified from Traditional NLP"],"readme":"# UnifiedQA\n\n\nYou may want to check out:\n - Paper: https://arxiv.org/abs/2005.00700\n - Demo: https://unifiedqa.apps.allenai.org/\n\nUpdate (Feb '22): UnifiedQA-v2 \n - Paper: https://arxiv.org/abs/2202.12359\n\n\n## Using the models in PyTorch/HuggingFace\n\nYou can very easily load the models with [Transformers](https://github.com/huggingface/transformers/) \u003e=3.1, instead of downloading them manually. \nThe models are listed on [this page](https://huggingface.co/allenai). Here is a list of model these model names hosted on HuggingFace model hub: \n\n\n| Model Name                | Huggingface ID (s)                      |\n|---------------------------|-----------------------------------------|\n| UnifiedQA (T5) - small    | `allenai/unifiedqa-t5-small`            |\n| UnifiedQA (T5) - base     | `allenai/unifiedqa-t5-base`             |\n| UnifiedQA (T5) - large    | `allenai/unifiedqa-t5-large`            |\n| UnifiedQA (T5) - 3B       | `allenai/unifiedqa-t5-3b`               |\n| UnifiedQA (T5) - 11B      | `allenai/unifiedqa-t5-11b`              |\n| UnifiedQA-v2 (T5) - small | `allenai/unifiedqa-v2-t5-small-[ckpt]`  |\n| UnifiedQA-v2 (T5) - base  | `allenai/unifiedqa-v2-t5-base-[ckpt]`   |\n| UnifiedQA-v2 (T5) - large | `allenai/unifiedqa-v2-t5-large-[ckpt]`  |\n| UnifiedQA-v2 (T5) - 3B    | `allenai/unifiedqa-v2-t5-3b-[ckpt]`     |\n| UnifiedQA-v2 (T5) - 11B   | `allenai/unifiedqa-v2-t5-11b-[ckpt]`    |\n\n\nWhere `[ckpt]` can be either `1251000` or `1363200`.  The numbers in [the paper](https://arxiv.org/abs/2202.12359) are reported based on `1251000` checkpoints. \n\nHere is an examples: \n\n```python\nfrom transformers import T5Tokenizer, T5ForConditionalGeneration\n\nmodel_name = \"allenai/unifiedqa-t5-small\" # you can specify the model size here\ntokenizer = T5Tokenizer.from_pretrained(model_name)\nmodel = T5ForConditionalGeneration.from_pretrained(model_name)\n\ndef run_model(input_string, **generator_args):\n    input_ids = tokenizer.encode(input_string, return_tensors=\"pt\")\n    res = model.generate(input_ids, **generator_args)\n    return tokenizer.batch_decode(res, skip_special_tokens=True)\n```\n\nFor instance, here is how you can use it to answer a multiple-choice question: \n\n```python\nrun_model(\"which is best conductor? \\\\n (a) iron (b) feather\")\n```\nwhich gives: `['iron']`\n\n\n```python\nrun_model(\"scott filled a tray with juice and put it in a freezer. the next day, scott opened the freezer. how did the juice most likely change? \\\\n (a) it condensed. (b) it evaporated. (c) it became a gas. (d) it became a solid.\")\n```\nwhich produces: `['it condensed.']`.\n\n\nNote that you can also pass in the arguments for text generation to the `run_model(.)` function:\n```python\nrun_model(\"which is best conductor? \\\\n (a) iron (b) feather (c) wood (d) plastic\",\n         temperature=0.9, num_return_sequences=4, num_beams=20)\n```\n\n\n\n## Feeding data into UnifiedQA\nDatasets should be converted into a textin/text-out format.\n\n - Question always comes first.\n - We use `\\n` separators between different parts of the input. This ensures having a humanlike encoding while not making it overly-specific to a certain format.  Note that this separator isn't the newline character (which it looks suspiciously like), but rather backslash-n.\n - Make sure the whole input is correctly [pre-processed](https://github.com/allenai/unifiedqa/blob/7bf0653c6fb68a51019924fd4c51615155acbebe/tasks.py#L54-L58) (e.g., lower-cased)\n\nHere are several examples:\n\n|  **Dataset** | **SQuAD 1.1 (extractive QA)** |\n| :---: | :--- |\n|  **Encoded Input** | `At what speed did the turbine operate? \\n (Nikola_Tesla) On his 50th birthday in 1906, Tesla demonstrated his 200 horsepower (150 kilowatts) 16,000 rpm bladeless turbine. ...` |\n|  **Encoded Output** | `16,000 rpm` |\n|  **Dataset** | **NarrativeQA (Abstractive QA)** |\n|  **Encoded Input** | `What does a drink from narcissus's spring cause the drinker to do?  \\n  Mercury has awakened Echo, who weeps for Narcissus, and states that a drink from Narcissus's spring causes the drinkers to ''Grow dotingly enamored of themselves.'' ...` |\n|  **Encoded Output** | `fall in love with themselves` |\n|  **Dataset** | **ARC-challenge (Multiple-choice QA)** |\n|  **Encoded Input** | `What does photosynthesis produce that helps plants grow? \\n (A) water (B) oxygen (C) protein (D) sugar` |\n|  **Encoded Output** | `sugar` |\n|  **Dataset** | **MCTest (Multiple-choice QA)** |\n|  **Encoded Input** | `Who was Billy? \\n (A) The skinny kid (B) A teacher (C) A little kid (D) The big kid \\n Billy was like a king on the school yard. A king without a queen. He was the biggest kid in our grade, so he made all the rules during recess. ...` |\n|  **Encoded Output** | `The big kid` |\n|  **Dataset** | **BoolQ (Yes-no QA)** |\n|  **Encoded Input** | `Was America the first country to have a president?  \\n (President) The first usage of the word president to denote the highest official in a government was during the Commonwealth of England ...` |\n|  **Encoded Output** | `no` |\n\nIf you wanna see how this encoding is done on our datasets, check out this [script](encode_datasets.py).\n\n\n### The datasets/tasks used in the experiments\nWhile the datasets we used are all public, it could be a bit time-confusing to convert them all into text-to-text format. We're releasing the already-proccessed text-to-text datasets based on the encoding used in this work. Files are included in [this Google Cloud bucket](https://console.cloud.google.com/storage/browser/unifiedqa/data). [Here](encode_datasets.py) is the script we used in order to convert each dataset into text-in-text-out format.\n\n## Prediction files\nReach out to DanielK if you want them! :)\n\n\n## Released Model Checkpoints\n\nIf you intend to create a QA system, you can use our QA-specialized models for your purpose:\n\n\n### T5 models\n - UnifiedQA (T5, small) [gs://unifiedqa/models/small](https://console.cloud.google.com/storage/browser/unifiedqa/models/small)\n - UnifiedQA (T5, base) [gs://unifiedqa/models/base](https://console.cloud.google.com/storage/browser/unifiedqa/models/base)\n - UnifiedQA (T5, large) [gs://unifiedqa/models/large](https://console.cloud.google.com/storage/browser/unifiedqa/models/large)\n - UnifiedQA (T5, 3B) [gs://unifiedqa/models/3B](https://console.cloud.google.com/storage/browser/unifiedqa/models/3B)\n - UnifiedQA (T5, 11B) [gs://unifiedqa/models/11B](https://console.cloud.google.com/storage/browser/unifiedqa/models/11B)\n\nNote: In the experiments reported in our paper we always used the checkpoint closest to 100k steps (it usually corresponds to checkpoint 1100500)\n\nYou can use these in two ways:\n- If you don't have any training data, you can use them for [the evaluation](https://github.com/google-research/text-to-text-transfer-transformer#eval).\n- If you training data, you can use them as your initial models and [fine-tune on them](https://github.com/google-research/text-to-text-transfer-transformer#fine-tuning).\n\nFor more details see [the T5 repository](https://github.com/google-research/text-to-text-transfer-transformer).\n\n### BART models\nThe BART models are downloaded from [this link](https://nlp.cs.washington.edu/ambigqa/models/unifiedQA/unifiedQA-bart.zip) (3.6G).\nFor detailed instructions on running the code (training/finetuning/testing), please refer to [here](https://github.com/allenai/unifiedqa/tree/master/bart).\nThe `uncased` models usually gave us better and more robust results.\n\n### v2 T5 models \n\n - UnifiedQA v2 (T5, small) [gs://unifiedqa/models_v2/small](https://console.cloud.google.com/storage/browser/unifiedqa/models_v2/small)\n - UnifiedQA v2 (T5, base) [gs://unifiedqa/models_v2/base](https://console.cloud.google.com/storage/browser/unifiedqa/models_v2/base)\n - UnifiedQA v2 (T5, large) [gs://unifiedqa/models_v2/large](https://console.cloud.google.com/storage/browser/unifiedqa/models_v2/large)\n - UnifiedQA v2 (T5, 3B) [gs://unifiedqa/models_v2/3B](https://console.cloud.google.com/storage/browser/unifiedqa/models_v2/3B)\n - UnifiedQA v2 (T5, 11B) [gs://unifiedqa/models_v2/11B](https://console.cloud.google.com/storage/browser/unifiedqa/models_v2/11B)\n\nNote: In the experiments reported in our paper we always used the checkpoint closest to 250k steps. \n\n\n## FAQ\n**I am not getting the expected results.** An common issue with using UnifiedQA is making sure you use the separator (`\\n`) when encoding encoding your inputs. See [the earlier section](#feeding-data-into-unifiedqa) where we delineate how to encode the inputs.\n\n**Help! I am getting the following error!** See [this discussion](https://github.com/google-research/text-to-text-transfer-transformer/issues/180) if you're getting the following error:\n```bash\nValueError: Configurable 'make_layer_stack' doesn't have a parameter named 'use_universal_transformer'.\n  In file \"gs://danielk-files/t5-models/union_mixture/11B/operative_config.gin\", line 83\n```\n\n\n## How to cite\n\nIf you extend or use this work, please cite the relevant papers:\n```bibtex\n@inproceedings{2020unifiedqa,\n    title={UnifiedQA: Crossing Format Boundaries With a Single QA System},\n    author={D. Khashabi and S. Min and T. Khot and A. Sabhwaral and O. Tafjord and P. Clark and H. Hajishirzi},\n    journal={EMNLP - findings},\n    year={2020}\n}\n@article{khashabi2022unifiedqa,\n    title={UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training},\n    author={Khashabi, Daniel and Kordi, Yeganeh and Hajishirzi, Hannaneh},\n    journal={arXiv preprint arXiv:2202.12359},\n    year={2022}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Funifiedqa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fallenai%2Funifiedqa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Funifiedqa/lists"}