{"id":13737941,"url":"https://github.com/Mattdl/ContinualEvaluation","last_synced_at":"2025-05-08T15:32:04.825Z","repository":{"id":112388752,"uuid":"496639911","full_name":"Mattdl/ContinualEvaluation","owner":"Mattdl","description":"[Spotlight ICLR 2023 paper] Continual evaluation for lifelong learning with neural networks, identifying the stability gap.","archived":false,"fork":false,"pushed_at":"2023-04-02T09:40:41.000Z","size":510,"stargazers_count":28,"open_issues_count":1,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-14T14:22:23.798Z","etag":null,"topics":["avalanche","continual-evaluation","continual-learning","deep-learning","lifelong-learning","machine-learning","pytorch","stability-gap"],"latest_commit_sha":null,"homepage":"https://openreview.net/forum?id=Zy350cRstc6","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Mattdl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-05-26T13:53:57.000Z","updated_at":"2025-04-01T11:04:32.000Z","dependencies_parsed_at":"2023-05-14T10:15:17.462Z","dependency_job_id":null,"html_url":"https://github.com/Mattdl/ContinualEvaluation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mattdl%2FContinualEvaluation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mattdl%2FContinualEvaluation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mattdl%2FContinualEvaluation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mattdl%2FContinualEvaluation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Mattdl","download_url":"https://codeload.github.com/Mattdl/ContinualEvaluation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253096266,"owners_count":21853567,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avalanche","continual-evaluation","continual-learning","deep-learning","lifelong-learning","machine-learning","pytorch","stability-gap"],"created_at":"2024-08-03T03:02:06.560Z","updated_at":"2025-05-08T15:32:04.391Z","avatar_url":"https://github.com/Mattdl.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Continual Evaluation for Lifelong Learning\n[//]: # (**What is this repo?**)\nThis is a Pytorch and Avalanche based repository to enable finegrained continual evaluation in continual learning, opposed to the standard task transition based evaluation.\nIt is the main codebase for the *Spotlight ICLR 2023 paper*:\n[\"Continual evaluation for lifelong learning: Identifying the stability gap\"](https://openreview.net/forum?id=Zy350cRstc6).\n\n**Why continual evaluation?**\nUsing continual evaluation, [our work](https://arxiv.org/abs/2205.13452) finds a *stability gap*, where representative continual learning\nmethods falter in maintaining robust performance during the learning process.\nMeasuring continual worst-case performance is important to enable continual\nlearners in the real world, especially for safety-critical applications and real-world actuators. \n\n\n**Main features of this repo:**\n- **Continual Eval**: Per-iteration evaluation, including test-set subsampling and adjustable evaluation periodicity\n- **Continual evaluation metrics**: \n  - **New metrics**: Worst-case Accuracy (WC-ACC), Average Minimum Accuracy (Min-ACC), Windowed-Forgetting (WF), Windowed-Plasticity (WP). \n  - **Existing metrics**: Learning Curve Area (LCA), Average Forgetting (FORG), Average Accuracy (ACC).\n- **Extensive tracking**: Track all stats of your continual learning model, e.g. per-iteration feature drift and gradient norms.\n- **7 Continual Learning benchmarks** based on: MNIST, CIFAR10, Mini-Imagenet, Mini-DomainNet, PermutedMNIST, RotatedMNIST, Digits\n\n*Project Status*: Codebase delivered as is, no support available.\n\n## Setup\nThis code uses\n- Python 3.8\n- [Avalanche](https://github.com/ContinualAI/avalanche) 0.1.0 (beta)\n- Pytorch 1.8.1\n\nTo setup your environment, you can use the [install script](./install_script.sh), which automatically creates an Anaconda environment for you. \nThe script defines the default versions used for the paper. \n    \n    ./install_script.sh\n\nYou can also define your own conda environment with the [environment.yml](environment.yml) file.\n\n    conda env create -n CLEVAL_ENV -f environment.yml python=3.8\n    conda activate CLEVAL_ENV\n\n\n## Reproducing results\nAll configs for the experiments can be found in [reproduce/configs](reproduce/configs).\nThe yaml config files enable a structured way to pass arguments to the python script.\n\nTo reproduce an experiment, simply run *./reproduce/run.sh* and pass the yaml filename. For example:\n    \n    ./reproduce/run.sh splitmnist_ER.yaml\n\nNote that we didn't run with deterministic CUDNN backbone for computational efficiency, which might result in small deviations in results.\nWe average all results over 5 initialization seeds, these can be run at once (with n_seeds=5),\nor define the specific seed per run (e.g. seed=0).\n\n## Continual Evaluation Implementation\nThe continual evaluation is integrated in the Avalanche flow of continual learning.\nFor documentation, see [here](https://avalanche.continualai.org/).\n\n\n- [src/eval/continual_eval.py](src/eval/continual_eval.py):  Introduces Continual Evaluation tracking flow after training iterations in Avalanche `after_training_iteration`.\n  The additional phases are defined as:\n\n        # Standard Avalanche CL flow\n        ...                          \n        - before_training_iteration\n        - after_training_iteration\n            - before_tracking        # BEGIN Integrated Continual Evaluation\n            - before_tracking_step\n            - before_tracking_batch\n            - after_tracking_batch\n            - after_tracking_step\n            - after_tracking         # END\n        - after_training_epoch\n        ...\n- [src/eval/continual_eval_metrics.py](src/eval/continual_eval_metrics.py): Contains all the continual evaluation metrics, which all inherit from `TrackerPluginMetric`.\n  This Plugin defines all the Continual Evaluation tracking phases, which each metric can overwrite as appropriate.\n- [main.py](main.py) first passes a list of plugins to Avalanche's `EvaluationPlugin` with `ContinualEvaluationPhasePlugin` first to update the Continual Evaluation metric states on `after_training_iteration`. \n Next in the list, the metric plugins are passed to `EvaluationPlugin` so `after_training_iteration` logs the metrics on each iteration.\n\n## Visualize results\nWe support both tensorboard and WandB.\nTo view results for Tensorboard, run:\n\n    tensorboard --logdir=OUTPUT_DIR\n\n\n## Citing and license\nPlease consider citing us upon using this repo:\n\n    @inproceedings{\n      delange2023continual,\n      title={Continual evaluation for lifelong learning: Identifying the stability gap},\n      author={Matthias De Lange and Gido M van de Ven and Tinne Tuytelaars},\n      booktitle={The Eleventh International Conference on Learning Representations },\n      year={2023},\n      url={https://openreview.net/forum?id=Zy350cRstc6}\n    }\n  \nCode is available under MIT license: A short and simple permissive license with conditions only requiring preservation of copyright and license notices.\nSee [LICENSE](LICENSE) for the full license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMattdl%2FContinualEvaluation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMattdl%2FContinualEvaluation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMattdl%2FContinualEvaluation/lists"}