{"id":13595271,"url":"https://github.com/zhuchen03/FreeLB","last_synced_at":"2025-04-09T10:33:20.586Z","repository":{"id":50305478,"uuid":"240404516","full_name":"zhuchen03/FreeLB","owner":"zhuchen03","description":"Adversarial Training for Natural Language Understanding","archived":false,"fork":false,"pushed_at":"2023-09-06T17:33:04.000Z","size":6118,"stargazers_count":250,"open_issues_count":6,"forks_count":41,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-06T17:46:51.039Z","etag":null,"topics":["adversarial","freelb","iclr2020","natural-language"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhuchen03.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-02-14T01:34:12.000Z","updated_at":"2024-09-26T15:01:18.000Z","dependencies_parsed_at":"2024-01-16T22:19:51.304Z","dependency_job_id":"e1ef5ec5-45e5-4998-805d-fe97396b4cd4","html_url":"https://github.com/zhuchen03/FreeLB","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuchen03%2FFreeLB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuchen03%2FFreeLB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuchen03%2FFreeLB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuchen03%2FFreeLB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhuchen03","download_url":"https://codeload.github.com/zhuchen03/FreeLB/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248020593,"owners_count":21034459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial","freelb","iclr2020","natural-language"],"created_at":"2024-08-01T16:01:46.756Z","updated_at":"2025-04-09T10:33:15.574Z","avatar_url":"https://github.com/zhuchen03.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Introduction\nThis repository contains the implementation for FreeLB on GLUE tasks based on both [fairseq](https://github.com/pytorch/fairseq) and [HuggingFace's transformers](https://github.com/huggingface/transformers) libraries, under `./fairseq-RoBERTa/` and `./huggingface-transformers/` respectively.\nWe also integrated our implementations of vanilla PGD, [FreeAT](https://arxiv.org/abs/1904.12843) and [YOPO](https://arxiv.org/abs/1905.00877) in our fairseq version.\nFreeLB is an adversarial training approach for improving transformer-based language models on Natural Language Understanding tasks.\nIt accumulates the gradient in the ascent steps and updates the parameters with the accumulated gradients, which is approximately equivalent to enlarging the batch size with diversified adversarial examples within different radiuses around the clean input.\nFreeLB improves the performance of BERT and RoBERTa on various Natural Language Understanding tasks including Question Answering, Natural Language Inference, and Sentiment Analysis.\n\nFor technical details and additional experimental results, please refer to our paper:\n\nChen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. [FreeLB: Enhanced Adversarial Training for Language Understanding](https://arxiv.org/abs/1909.11764). In ICLR, 2020.\n\n# What's New\n* Feb 15, 2020: Initial release of FreeLB based on fairseq and HuggingFace's transformers. The first one contains our implementations of FreeLB, FreeAT, YOPO for [RoBERTa](https://arxiv.org/abs/1907.11692), while the latter one is FreeLB for [ALBERT](https://arxiv.org/abs/1909.11942).\n\n* May 16, 2020: Hyperparameters for ALBERT are now available at `huggingface-transformers/launch/run_glue.sh`.\n\n# Prerequisites\nThe code is compatible with PyTorch 1.4.0. \nIn addition, you need to execute the followings in order, to install the prerequisites for fairseq and HuggingFace's transformers:\n```\n# Install apex\ngit clone https://github.com/NVIDIA/apex\ncd apex\npip install -v --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" ./\n\n# Configure fairseq\ncd ../fairseq-RoBERTa\npip install --editable .\n\n# Download and pre-process GLUE data\nwget https://gist.githubusercontent.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e/raw/17b8dd0d724281ed7c3b2aeeda662b92809aadd5/download_glue_data.py\npython download_glue_data.py --data_dir glue_data --tasks all\nsource ./examples/roberta/preprocess_GLUE_tasks.sh glue_data ALL\n\ncd ../huggingface-transformers\npip install --editable .\nmkdir logs\n```\n\n# Launch\nThe launch scripts are under `./fairseq-RoBERTa/launch/` and `./huggingface-transformers/launch/`, where we have included most of the running scripts for RoBERTa and ALBERT on GLUE dev sets. \nWe will release more details in the future.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhuchen03%2FFreeLB","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhuchen03%2FFreeLB","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhuchen03%2FFreeLB/lists"}