{"id":13737780,"url":"https://github.com/ZixuanKe/PyContinual","last_synced_at":"2025-05-08T15:31:26.382Z","repository":{"id":50372032,"uuid":"373950964","full_name":"ZixuanKe/PyContinual","owner":"ZixuanKe","description":"PyContinual (An Easy and Extendible Framework for Continual Learning)","archived":false,"fork":false,"pushed_at":"2024-01-29T20:57:15.000Z","size":3147,"stargazers_count":292,"open_issues_count":5,"forks_count":62,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-08-04T03:11:10.217Z","etag":null,"topics":["attention-mechanism","capsule-network","catastrophic-forgetting","continual-learning","knowledge-transfer","natural-language-processing","text-classification","transfer-learning","transformer-architecture"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ZixuanKe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-06-04T20:19:18.000Z","updated_at":"2024-08-02T12:33:56.000Z","dependencies_parsed_at":"2024-01-07T05:59:53.455Z","dependency_job_id":"87b39cd8-eae6-4f44-8b90-1274d971762b","html_url":"https://github.com/ZixuanKe/PyContinual","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZixuanKe%2FPyContinual","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZixuanKe%2FPyContinual/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZixuanKe%2FPyContinual/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZixuanKe%2FPyContinual/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ZixuanKe","download_url":"https://codeload.github.com/ZixuanKe/PyContinual/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224742359,"owners_count":17362229,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-mechanism","capsule-network","catastrophic-forgetting","continual-learning","knowledge-transfer","natural-language-processing","text-classification","transfer-learning","transformer-architecture"],"created_at":"2024-08-03T03:02:00.666Z","updated_at":"2024-11-15T06:30:59.630Z","avatar_url":"https://github.com/ZixuanKe.png","language":"Python","funding_links":[],"categories":["Python","💻 Resources"],"sub_categories":["Tools"],"readme":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n# PyContinual (An Easy and Extendible Framework for Continual Learning)\n\n## News\n🔥 If you're here for the code from our latest [EMNLP23](https://arxiv.org/abs/2310.09436) paper, please check the [developing branch](https://github.com/ZixuanKe/PyContinual/tree/dev/v1.0.0).  We have added [continual_finetune.ipynb](https://github.com/ZixuanKe/PyContinual/tree/main/examples/continual_finetune.ipynb) as a self-contained example of the soft-masking scenario. It runs well without GPUs!         \n🔥 Our works on **continual pre-training of LMs** have now been accepted at [EMNLP22](https://arxiv.org/abs/2210.05549) and [ICLR23](https://arxiv.org/abs/2302.03241). Check out our [repo](https://github.com/UIC-Liu-Lab/ContinualLM) for continual pre-training / post-training!   \n🔥 Our latest **survey on continual learning of NLP** is now in [arXiv](https://arxiv.org/abs/2211.12701). Take a look if you are interested in CL and NLP!  \n🔥 Are you interested in additional baselines, tasks (including extraction and generation), LMs (such as RoBERTa and BART), and efficient training methods (like fp16 and multi-node)? Check out our [developing branch](https://github.com/ZixuanKe/PyContinual/tree/dev/v1.0.0)!\n\n## Easy to Use\n\nYou can sumply change the `baseline`, `backbone` and `task`, and then ready to go.\nHere is an example:\n```python\n\tpython run.py \\  \n\t--bert_model 'bert-base-uncased' \\  \n\t--backbone bert_adapter \\ #or other backbones (bert, w2v...)  \n\t--baseline ctr \\  #or other avilable baselines (classic, ewc...)\n\t--task asc \\  #or other avilable task/dataset (dsc, newsgroup...)\n\t--eval_batch_size 128 \\  \n\t--train_batch_size 32 \\  \n\t--scenario til_classification \\  #or other avilable scenario (dil_classification...)\n\t--idrandom 0  \\ #which random sequence to use\n\t--use_predefine_args #use pre-defined arguments\n```\n\n## Easy to Extend\n\nYou only need to write  your own `./dataloader`, `./networks` and `./approaches`. You are ready to [go](https://github.com/ZixuanKe/PyContinual/blob/main/docs/run_own.md)!\n\n## Performance\n\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://github.com/ZixuanKe/PyContinual\"\u003e\n        \u003cimg src=\"https://github.com/ZixuanKe/PyContinual/blob/main/docs/benchmarks.png\" width=\"500\"/\u003e\n    \u003c/a\u003e    \n    \u003cbr\u003e\n\u003cp\u003e\n\n\n## Introduction\nRecently, continual learning approaches have drawn more and more attention. This repo contains pytorch implementation of a set of (improved) SoTA methods using the same training and evaluation pipeline.\n\nThis repository contains the code for the following papers:\n*  [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://proceedings.neurips.cc/paper/2021/hash/bcd0049c35799cdf57d06eaf2eb3cff6-Abstract.html), Zixuan Ke, [Bing Liu](https://www.cs.uic.edu/~liub/), Nianzu Ma, [Hu Xu](https://howardhsu.github.io/) and [Lei Shu](https://leishu02.github.io/), NeurIPS 2021\n*  [CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks](https://aclanthology.org/2021.emnlp-main.550/), Zixuan Ke, [Bing Liu](https://www.cs.uic.edu/~liub/), [Hu Xu](https://howardhsu.github.io/) and [Lei Shu](https://leishu02.github.io/), EMNLP 2021\n*  [Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks](https://www.aclweb.org/anthology/2021.naacl-main.378.pdf), Zixuan Ke, [Hu Xu](https://howardhsu.github.io/) and [Bing Liu](https://www.cs.uic.edu/~liub/), NAACL 2021\n* [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://proceedings.neurips.cc/paper/2020/file/d7488039246a405baf6a7cbc3613a56f-Paper.pdf), Zixuan Ke, [Bing Liu](https://www.cs.uic.edu/~liub/) and [Xingchang Huang](https://people.mpi-inf.mpg.de/~xhuang/), NeurIPS 2020 (if you only care about this model, you can also check [CAT](https://github.com/ZixuanKe/CAT))\n* [Continual Learning with Knowledge Transfer for Sentiment Classification](https://www.cs.uic.edu/~liub/publications/ECML-PKDD-2020.pdf), Zixuan Ke, [Bing Liu](https://www.cs.uic.edu/~liub/), [Hao Wang](https://cshaowang.github.io/) and [Lei Shu](https://leishu02.github.io/), ECML-PKDD 2020 (if you only care about this model, you can also check [LifelongSentClass](https://github.com/ZixuanKe/LifelongSentClass))\n* **[40+](https://github.com/ZixuanKe/PyContinual/blob/master/docs/baselines.md)** baselines and variants (and keeps \"continually\" growing!)\n\n\n## Features\n* **Datasets**: It currently supports **Language Datasets** (Document/Sentence/Aspect Sentiment Classification, Natural Language Inference, Topic Classification) and **Image Datasets** (CelebA, CIFAR10, CIFAR100, FashionMNIST, F-EMNIST, MNIST, VLCS)\n* **Scenarios**: It currently supports **Task Incremental Learning** and **Domain Incremental Learning**\n* **Training Modes**: It currently supports single-GPU. You can also change it to multi-node distributed training and the mixed precision training.\n\n## Architecture\n`./res`: all results saved in this folder.    \n`./dat`: processed data    \n`./data`: raw data  \n`./dataloader`: contained dataloader for different data  \n`./approaches`: code for training  \n`./networks`: code for network architecture  \n`./data_seq`:  some reference sequences (e.g. asc_random)  \n`./tools`: code for preparing the data\n\n## Setup\n* If you want to run the existing systems, please see [run_exist.md](https://github.com/ZixuanKe/PyContinual/blob/master/docs/run_exist.md)\n* If you want to expand the framework with your own model, please see  [run_own.md](https://github.com/ZixuanKe/PyContinual/blob/master/docs/run_own.md)\n* If you want to see the full list of baselines and variants, please see [baselines.md](https://github.com/ZixuanKe/PyContinual/blob/master/docs/baselines.md)\n\n\n## Reference\nIf using this code, parts of it, or developments from it, please consider cite the references bellow.\n\n\t@inproceedings{ke2021achieve,\n\t  title={Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning},\n\t  author={Ke, Zixuan and Liu, Bing and Ma, Nianzu and Xu, Hu, and Lei Shu},\n\t  booktitle={NeurIPS},\n\t  year={2021}\n\t}\n\t\n\t@inproceedings{ke2021contrast,\n\t  title={CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks},\n\t  author={Ke, Zixuan and Liu, Bing and Xu, Hu, and Lei Shu},\n\t  booktitle={EMNLP},\n\t  year={2021}\n\t}\n\n\t@inproceedings{ke2021adapting,\n\t  title={Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks},\n\t  author={Ke, Zixuan and Xu, Hu and Liu, Bing},\n\t  booktitle={Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},\n\t  pages={4746--4755},\n\t  year={2021}\n\t}\n\n    @inproceedings{ke2020continualmixed,\n    author= {Ke, Zixuan and Liu, Bing and Huang, Xingchang},\n    title= {Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks},\n    booktitle = {Advances in Neural Information Processing Systems},\n    volume={33},\n    year = {2020}}\n\n\t@inproceedings{ke2020continual,\n\tauthor= {Zixuan Ke and Bing Liu and Hao Wang and Lei Shu},\n\ttitle= {Continual Learning with Knowledge Transfer for Sentiment Classification},\n\tbooktitle = {ECML-PKDD},\n\tyear = {2020}}\n\n    \n## Contact\n\n\nPlease drop an email to [Zixuan Ke](mailto:zke4@uic.edu), [Xingchang Huang](mailto:huangxch3@gmail.com) or [Nianzu Ma](mailto:jingyima005@gmail.com) if you have any questions regarding to the code. We thank [Bing Liu](https://www.cs.uic.edu/~liub/), [Hu Xu](https://howardhsu.github.io/) and [Lei Shu](https://leishu02.github.io/) for their valuable comments and opinioins.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZixuanKe%2FPyContinual","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FZixuanKe%2FPyContinual","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZixuanKe%2FPyContinual/lists"}