{"id":17312931,"url":"https://github.com/andreacossu/continual-pretraining-nlp-vision","last_synced_at":"2025-06-21T07:07:21.273Z","repository":{"id":158087009,"uuid":"487847563","full_name":"AndreaCossu/continual-pretraining-nlp-vision","owner":"AndreaCossu","description":"Code to reproduce experiments from the paper \"Continual Pre-Training Mitigates Forgetting in Language and Vision\" https://arxiv.org/abs/2205.09357","archived":false,"fork":false,"pushed_at":"2023-10-23T08:53:37.000Z","size":893,"stargazers_count":22,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-14T14:55:43.478Z","etag":null,"topics":["continual-learning","forgetting","image-classification","lifelong-learning","masked-image-modeling","masked-language-models","pretrained-models","qnli","self-supervised","senteval","sentiment-analysis","transformers"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AndreaCossu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-05-02T13:05:42.000Z","updated_at":"2025-01-29T15:38:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"9247ce77-0b9e-47f7-9002-797aeafc9ba7","html_url":"https://github.com/AndreaCossu/continual-pretraining-nlp-vision","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AndreaCossu/continual-pretraining-nlp-vision","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreaCossu%2Fcontinual-pretraining-nlp-vision","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreaCossu%2Fcontinual-pretraining-nlp-vision/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreaCossu%2Fcontinual-pretraining-nlp-vision/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreaCossu%2Fcontinual-pretraining-nlp-vision/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AndreaCossu","download_url":"https://codeload.github.com/AndreaCossu/continual-pretraining-nlp-vision/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreaCossu%2Fcontinual-pretraining-nlp-vision/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261080624,"owners_count":23106602,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["continual-learning","forgetting","image-classification","lifelong-learning","masked-image-modeling","masked-language-models","pretrained-models","qnli","self-supervised","senteval","sentiment-analysis","transformers"],"created_at":"2024-10-15T12:45:09.358Z","updated_at":"2025-06-21T07:07:16.258Z","avatar_url":"https://github.com/AndreaCossu.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Continual Pre-Training Mitigates Forgetting in Language and Vision\n\nhttps://arxiv.org/abs/2205.09357\n\n## Prerequisites\n* Install PyTorch (GPU support strongly recommended)\n* Install [Avalanche library](https://github.com/ContinualAI/avalanche/) for continual learning with `pip install avalanche-lib`. Check that version is 0.1.0 (beta). \n* We rely on Huggingface for NLP experiments.\n* We rely on [`torch_cka`](https://pypi.org/project/torch-cka/) package to compute CKA\n* Specify data paths in `utils.py` file. This is used to locate data across all experiments.\n\nWe provide an environment file `env.yml` with which you can build the conda environment we used in the paper.\n\n## NLP environment\n[Download](https://drive.google.com/file/d/18gGyDFJuYkrePX8GOvK3nkqeaI6YWK23/view?usp=sharing) the preprocessed version of the NLP benchmarks.\nThe total size after unpacking should be around `21GB`.\nThis includes tokenized datasets in different versions: for Bert, Roberta and Roberta with expanding vocabulary. \nIn the latter case, you can find one tokenized version of each dataset per experience. \nPut the downloaded file under the same directory. Specify these paths in the `utils.py`.\n\nWhen pretraining or finetuning the original pretrained model, you can pass the Huggingface modelname (e.g., `roberta-base`) to the `modelname` parameter of each script. When finetuning the pretrained model, you should pass\nthe path to the folder where the pretrained model has been saved after pretraining.\n\nThe script `pretraining-nlp.py` applies a *single* step of pretraining/downstream finetuning. The `task_type` parameter distinguishes between the two tasks.. \n\nThe script `continual-pretraining-nlp.py` applies continual pretraining/downstream finetuning (on both scientific abstracts and proxy datasets). The `task_type` parameter distinguishes between the possible tasks.\n\n### SentEval experiments\nClone the [official SentEval repository](https://github.com/facebookresearch/SentEval) and put it in the main project directory under the name `SentEval`.  \nRun the `senteval.py` script.\n\n### Build NLP benchmarks from original data\nIf you do not want to use the version of the dataset we prepared, you can also start from the original data and preprocess them.\nLink to original datasets can be found in the appendix of our paper.\n\nIn particular, `tokenize_and_save.py` prepare the datasets for offline training (no continual learning phase). This is used to study the effect of a single pretraining step on the entire dataset of abstract classification.\nThe script takes the datasets and build the subsampled version we used in our experiments. \nYou need to change the path to the data folders and you can change few options in the first rows, like tokenizer type used in preprocessing and whether to use expanding vocabulary for Roberta or not.\nThe `pretrain` and `finetuning` nomenclature for the scientific abstracts preprocessing distinguish between the splits used for pretraining and the splits used for finetuning.\n\nThe script `tokenize_and_save_cl.py` adopts the same approach but for the continual pretraining stream. It produces a tokenized version of the benchmarks for each experience.\n\n## Computer Vision environment\nCORe50 is available for automatic download through Avalanche. iNaturalist is available through TorchVision (follow instructions from TorchVision API).\n\nThe script `continual-pretraining-cv-supervised.py` performs continual pretraining with supervised image classification protocol.\n\nThe script `continual-pretraining-cv-unsupervised.py` performs continual pretraining with unsupervised masked image modelling protocol.\n\n## CKA analysis\nThe `cka.py` script produces the CKA plots representing similarity across each model layer. \nThe script compares the original pretrained model with the continuously pretrained model.\nWe used the default hyperparameters of the script.\n\n## Traditional continual learning scenario for NLP environment\nWe provide a script (`traditional_cl_nlp.py`) to run the NLP experiments in the traditional continual learning scenario (no continual pretraining), where a model is continuoulsy finetuned on the scientific abstracts dataset.\nThe model is tested on all the experiences of scientific abstracts to measure forgetting. Catastrophic forgetting happens as expected here.\n\n## Utils file\nThe `utils.py` file contains a broad set of utilities used in the experiments. \n\nImportantly, there you can find the list of abstract classes used in our experiments and the \nlist of tokens added to Roberta when its vocabulary is expanding. The list is specified both for the single step pretraining and for the continual pretraining experiment (one list of tokens per experience).\n\nAll the custom Avalanche strategies (e.g., used to adapt Huggingface for Avalanche) are also defined there. \n\nFor iNaturalist, we provide the indices of the few patterns we removed because they did not match the default image format of the dataset.\n\n## Reproducing experiments\nThe folder `experiments/vision` provides files showing the hyperparameter configurations used in our experiments. You can reproduce \nresults by running the main script of the experiment you want with the corresponding hyperparameters values.\nFor NLP experiments hyperparameters are described in the README of the corresponding folder `experiments/nlp`.\n\n## Jupyter notebook\nWe provide a Jupyter notebook to replicate our exploratory analysis of the NLP environment, including the analysis on the number of\nabstracts per class and the most frequent tokens present in the dataset with respect to Roberta tokenizer. \nIn particular, we used this last information to select which tokens to add to Roberta when its vocabulary was expanding. \nThe Jupyter notebook is provided *as is*, since we used it for data exploration and not for the actual execution of experiments.\n\n## How to cite the paper\n\n```\n@article{cossu2022a,\n  title = {Continual {{Pre-Training Mitigates Forgetting}} in {{Language}} and {{Vision}}},\n  author = {Cossu, Andrea and Tuytelaars, Tinne and Carta, Antonio and Passaro, Lucia and Lomonaco, Vincenzo and Bacciu, Davide},\n  year = {2022},\n  url = {https://arxiv.org/abs/2205.09357v1}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreacossu%2Fcontinual-pretraining-nlp-vision","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandreacossu%2Fcontinual-pretraining-nlp-vision","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreacossu%2Fcontinual-pretraining-nlp-vision/lists"}