{"id":17141066,"url":"https://github.com/zphang/lrqa","last_synced_at":"2025-06-17T12:33:44.094Z","repository":{"id":46947969,"uuid":"370506618","full_name":"zphang/lrqa","owner":"zphang","description":null,"archived":false,"fork":false,"pushed_at":"2023-03-14T06:06:23.000Z","size":53,"stargazers_count":7,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-02T10:51:23.266Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zphang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-24T23:07:02.000Z","updated_at":"2025-01-11T09:14:24.000Z","dependencies_parsed_at":"2024-12-01T16:47:56.463Z","dependency_job_id":"8de30125-f235-4999-9e4b-3c5b8d805793","html_url":"https://github.com/zphang/lrqa","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zphang/lrqa","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Flrqa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Flrqa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Flrqa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Flrqa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zphang","download_url":"https://codeload.github.com/zphang/lrqa/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Flrqa/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260358277,"owners_count":22996999,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T20:24:15.159Z","updated_at":"2025-06-17T12:33:39.063Z","avatar_url":"https://github.com/zphang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# QuALITY training code (LRQA)\n\nThis LRQA repository contains code for running baselines on the QuALITY data. \n\n## Update\n\n- [Mar 2020]: Updated preprocessing scripts for QuALITY v1.0.1 \n\n## Overview\n\nThis repository contains code to support experiments on long-document QA models. Currently it just works as a multiple-choice experiment repository.\n\n### Tasks\n\nLRQA supports either existing multiple-choice datasets (e.g. Cosmos QA, HellaSwag), or custom task datasets for quick iteration. In all cases, the goal is to map different datasets to a standard format of:\n\n```\n{\n    \"context\": ...\n    \"query\": ...\n    \"option_0\" ...\n      ..\n    \"option_N\" ...\n    \"label\": ...\n}\n```\n\nWhere the data will be formatted into inputs as:\n\n```\n[CLS] context [SEP] query option \n```\nThe strings are concatenated with *no spaces*. Hence, it is recommended to prepend spaces to option and query values. (Formatting may differ based on different tokenizers.)\n\nConverters to this standard format are available for a set of multiple-choice tasks (see: `lrqa/tasks.py`):\n\n- Cosmos QA\n\n\nFor custom tasks, you can provide a path to a folder containing files such as\n```\nconfig.json\ntrain.jsonl\nvalidation.jsonl\ntest.jsonl\n```\n\nThe individual phases are optional. Each phase is contained in a single `jsonl` file, with the keys as specified above. `config.json` specifies some configurations for the task, such as the number of choices. See `lrqa.tasks.CustomJSONLTask` for more details. See `./resources/example_jsonl_task` for an example. \n\n### Models\n\nModels can be broken down into 2 categories:\n\n**Encoder-based** models are based on `transformers.AutoModelForMultipleChoice`. All auto-modals should be compatible. These need to be fine-tuned, though tuned models can be applied across tasks.\n\nNote: Hugging Face's implementation runs `.forward` on all options at once, which can increases ram consumption. Adjust batch sizes and gradient accumulation steps accordingly.\n\n**Generation-based** models are based on `transformers.AutoModelForCausalLM`. All auto-modals should be compatible. These can be run zero-shot, as the LM heads can be used directly to score continuations. \n\nEncoder-Decoder support is coming soon!\n\n## Usage setup\n\nIt is recommended to add the path to this repository to your `PYTHONPATH`, e.g.\n\n```bash\nexport PYTHONPATH=/path/to/lrqa/:${PYTHONPATH} \n```\n\nInstall requirements as necessary from `requirements.txt`. It is also recommended though not necessary to run code from within this folder.\n\n## Sample usage\n\n#### Training + Evaluating an Encoder\n\n```bash\nEXPDIR=/path/to/experiment\npython lrqa/run_lrqa.py \\\n    --model_name_or_path roberta-base \\\n    --model_mode mc \\\n    --task_name cosmosqa \\\n    --output_dir ${EXPDIR} \\\n    --per_device_train_batch_size 8 \\\n    --per_device_eval_batch_size 16 \\\n    --gradient_accumulation_steps 1 \\\n    --do_train \\\n    --do_eval \\\n    --evaluation_strategy steps \\\n    --eval_steps 500 \\\n    --save_strategy no \\\n    --load_best_model_at_end \\\n    --num_train_epochs 1\n```\n\n#### Evaluating a Generator\n\n```bash\nEXPDIR=/path/to/experiment\npython lrqa/run_lrqa.py \\\n    --model_name_or_path gpt2 \\\n    --model_mode generation \\\n    --task_name cosmosqa \\\n    --output_dir ${EXPDIR} \\\n    --per_device_train_batch_size 8 \\\n    --per_device_eval_batch_size 16 \\\n    --gradient_accumulation_steps 1 \\\n    --do_eval \\\n    --evaluation_strategy steps \\\n    --eval_steps 500 \\\n    --save_strategy no \\\n    --load_best_model_at_end \\\n    --num_train_epochs 1\n```\n\n#### Evaluating a Generator on a custom task\n\n```bash\nEXPDIR=/path/to/experiment\npython lrqa/run_lrqa.py \\\n    --model_name_or_path gpt2 \\\n    --model_mode generation \\\n    --task_name custom \\\n    --task_base_path ./resources/example_jsonl_task \\\n    --output_dir ${EXPDIR} \\\n    --per_device_eval_batch_size 16 \\\n    --gradient_accumulation_steps 1 \\\n    --do_eval \\\n    --evaluation_strategy steps \\\n    --eval_steps 500 \\\n    --save_strategy no \\\n    --load_best_model_at_end \\\n    --num_train_epochs 1\n```\n\n\n## Extractive Preprocessing\n\nUse the HTML-cleaned version of the QuALITY data. You can run the preprocessing for generating the extractive baseline inputs as follows:\n\n```bash\npython lrqa/scripts/extraction.py \\\n    --input_base_path /path/to/input/data \\\n    --output_base_path /path/to/output/data \\\n    --scorer rouge \\\n    --query_type question\n```\n\nThis will prepare the data in a format that is suitable for the above `run_lrqa` scripts.\n\nTo use fastText, you will first need to download fastText embeddings from [here](https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip), and then run the following script:\n\n```bash\npython lrqa/scripts/save_fasttext_embeddings.py \\\n    --fasttext_data_path /path/to/crawl-300d-2M.vec \\\n    --output_path /path/to/fasttext_embeddings.p\n```\n\n## Applications\n\n### Applying to BBQ\n\nHere is some sample code for running evaluation on [BBQ](https://arxiv.org/abs/2110.08193).\n\nFirst, we preprocess (just writing to standardized JSONL format) and fine-tune on RACE.\n\n```bash\nEXP_FOL=...\nHF_MODEL_NAME=roberta-base\nBATCH_SIZE=8\nGRAD_ACCUM_STEPS=4\npython lrqa/scripts/race_preproc.py \\\n    --data_path ${EXP_FOL}/race\npython lrqa/run_lrqa.py \\\n    --model_name_or_path ${HF_MODEL_NAME} \\\n    --model_mode mc \\\n    --max_seq_length 512 \\\n    --task_name custom \\\n    --task_base_path ${EXP_FOL}/race \\\n    --output_dir ${EXP_FOL}/race_run \\\n    --learning_rate 1e-5 \\\n    --num_train_epochs 3 \\\n    --warmup_ratio 0.1 \\\n    --eval_steps 1000 \\\n    --save_steps 1000 \\\n    --save_total_limit 5 \\\n    --save_strategy steps \\\n    --evaluation_strategy steps \\\n    --load_best_model_at_end \\\n    --per_device_train_batch_size ${BATCH_SIZE} \\\n    --per_device_eval_batch_size ${BATCH_SIZE} \\\n    --gradient_accumulation_steps ${GRAD_ACCUM_STEPS} \\\n    --do_train --do_eval --do_predict --predict_phases validation\n```\n\nNext we preprocess all the BBQ data (cloned from https://github.com/nyu-mll/BBQ), and run evaluation on each of the categories using fine-tuned models.\n```bash\nBBQ_DATA=...  # clone BBQ repo and point to `data` folder \npython lrqa/scripts/bbq_preproc.py \\\n    --input_data_path=${BBQ_DATA} \\\n    --data_path ${EXP_FOL}/bbq\nfor CATEGORY in Age Disability_status Gender_identity Nationality Physical_appearance Race_ethnicity Race_x_SES Race_x_gender Religion SES Sexual_orientation; do\n    python lrqa/run_lrqa.py \\\n        --model_name_or_path ${EXP_FOL}/race_run/checkpoint-last \\\n        --model_mode mc \\\n        --max_seq_length 512 \\\n        --task_name custom \\\n        --task_base_path ${EXP_FOL}/bbq/${CATEGORY}/ \\\n        --output_dir ${EXP_FOL}/bbq_runs/${CATEGORY}/ \\\n        --per_device_eval_batch_size ${BATCH_SIZE} \\\n        --do_eval --do_predict --predict_phases validation\ndone\n```\n\nThe predictions should be stored in a PyTorch pickle at `${EXP_FOL}/bbq_runs/${CATEGORY}/validation_predictions.p`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzphang%2Flrqa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzphang%2Flrqa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzphang%2Flrqa/lists"}