{"id":13737811,"url":"https://github.com/JosephKJ/ELI","last_synced_at":"2025-05-08T15:31:35.633Z","repository":{"id":52114401,"uuid":"470455415","full_name":"JosephKJ/ELI","owner":"JosephKJ","description":"(CVPR 2022) Energy-based Latent Aligner for Incremental Learning","archived":false,"fork":false,"pushed_at":"2022-12-19T09:07:49.000Z","size":2013,"stargazers_count":48,"open_issues_count":1,"forks_count":9,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-12T18:23:34.770Z","etag":null,"topics":["continual-learning","cvpr","cvpr2022","ebm","energy-based-model","incremental-learning","latent-aligning","pytorch"],"latest_commit_sha":null,"homepage":"https://josephkj.in","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JosephKJ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-16T06:11:54.000Z","updated_at":"2024-09-10T12:04:48.000Z","dependencies_parsed_at":"2023-01-29T21:01:06.723Z","dependency_job_id":null,"html_url":"https://github.com/JosephKJ/ELI","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosephKJ%2FELI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosephKJ%2FELI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosephKJ%2FELI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosephKJ%2FELI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JosephKJ","download_url":"https://codeload.github.com/JosephKJ/ELI/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224742402,"owners_count":17362229,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["continual-learning","cvpr","cvpr2022","ebm","energy-based-model","incremental-learning","latent-aligning","pytorch"],"created_at":"2024-08-03T03:02:01.885Z","updated_at":"2024-11-15T06:31:03.091Z","avatar_url":"https://github.com/JosephKJ.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"## Energy-based Latent Aligner for Incremental Learning [[arXiv](https://arXiv.org/abs/2203.14952) | [Poster](https://github.com/JosephKJ/ELI/files/8820668/ELI_poster_final.pdf) | [Video](https://youtu.be/ohfU9C4lS0s)]\n\n### \n\n### Accepted to CVPR 2022 \n[![paper](https://img.shields.io/badge/arXiv-Paper-\u003cCOLOR\u003e.svg)](https://arxiv.org/abs/2203.14952)\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/4231550/159659561-17bea6a6-5228-42e6-a811-eb18d37c48e9.png\" width=\"500\"/\u003e\n\u003c/p\u003e\n\u003cp align=\"center\" width=\"80%\"\u003e\nWe illustrate an Incremental Learning model trained on a continuum of tasks in the top part of the figure. While learning the current task \u003cimg src=\"https://render.githubusercontent.com/render/math?math=\\tau_t\"\u003e, the latent representation of Task \u003cimg src=\"https://render.githubusercontent.com/render/math?math=\\tau_{t-1}\"\u003e data gets disturbed, as shown by red arrows. ELI learns an energy manifold, and uses it to counteract this inherent representational shift, as illustrated by green arrows, thereby alleviating forgetting.\u003c/p\u003e\n\n### Overview\n\n[//]: # (Deep learning models tend to forget their earlier knowledge while incrementally learning new tasks. This behavior emerges because the parameter updates optimized for the new tasks may not align well with the updates suitable for older tasks. The resulting latent representation mismatch causes forgetting. )\n\nIn this work, we propose ELI: Energy-based Latent Aligner for Incremental Learning, which:\n- Learns an energy manifold for the latent representations such that previous task latents will have low energy and the current task latents have high energy values. \n- This learned manifold is used to counter the representational shift that happens during incremental learning.\n\nThe implicit regularization that is offered by our proposed methodology can be used as a **plug-and-play module** in existing incremental learning methodologies for classification and object-detection. \n\n[//]: # (We validate this through extensive evaluation on CIFAR-100, ImageNet subset, ImageNet 1k and Pascal VOC datasets. We observe consistent improvement when ELI is added to three prominent methodologies in class-incremental learning, across multiple incremental settings. )\n\n[//]: # (Further, when added to the state-of-the-art incremental object detector, ELI provides over 5% improvement in detection accuracy, corroborating its effectiveness and complementary advantage to existing art.)\n\n\n[//]: # (## Methodology)\n\n[//]: # ()\n[//]: # (\u003cp align=\"center\" width=\"100%\"\u003e)\n\n[//]: # (\u003cimg src=\"https://user-images.githubusercontent.com/4231550/159659616-23f6d790-35b3-4be3-b183-c5afda18e9d9.png\" width=\"600\"/\u003e)\n\n[//]: # (\u003c/p\u003e)\n\n\n## Toy Experiment\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/4231550/159659669-be756c6b-1948-4cd1-9ab7-acec9c69030b.png\"/\u003e\n\u003c/p\u003e\n\nA key hypothesis that we base our methodology is that while learning a new task, the latent representations will get disturbed, which will in-turn cause catastrophic forgetting of the previous task, and that an energy manifold can be used to align these latents, such that it alleviates forgetting. \n\nHere, we illustrate a proof-of-concept that our hypothesis is indeed true.\nWe consider a two task experiment on MNIST, where each task contains a subset of classes: \u003cimg src=\"https://render.githubusercontent.com/render/math?math=\\tau_1\"\u003e = {0, 1, 2, 3, 4}, \u003cimg src=\"https://render.githubusercontent.com/render/math?math=\\tau_2\"\u003e = {5, 6, 7, 8, 9}. \n\nAfter learning the second task, the accuracy on \u003cimg src=\"https://render.githubusercontent.com/render/math?math=\\tau_1\"\u003e test set drops to 20.88%, while experimenting with a 32 dimensional latent space.\nThe latent aligner in ELI provides 62.56% improvement in test accuracy to 83.44%.\nThe visualization of a 512 dimensional latent space after learning \u003cimg src=\"https://render.githubusercontent.com/render/math?math=\\tau_2\"\u003e in sub-figure (c), indeed shows cluttering due to representational shift. ELI is able to align the latents as shown in sub-figure (d), which alleviates the drop in accuracy from 89.14% to 99.04%.\n\nThe code for these toy experiments are in:\n- [ELI.ipynb](https://github.com/JosephKJ/ELI/blob/main/ELI.ipynb)\n- [ELI_512.ipynb](https://github.com/JosephKJ/ELI/blob/main/ELI_512.ipynb)\n\n\n## Implicitly Recognizing and Aligning Important Latents\n\nhttps://user-images.githubusercontent.com/4231550/159675403-f2cee8e3-bddb-4e8f-80a1-90cb638b372e.mp4\n\nEach row \u003cimg src=\"https://render.githubusercontent.com/render/math?math=i\"\u003e shows how \u003cimg src=\"https://render.githubusercontent.com/render/math?math=i^th\"\u003e latent dimension is updated by ELI. We see that different dimensions have different degrees of change, which is implicitly decided by our energy-based model.\n\n\n## Classification and Detection Experiments\n\nCode and models for the classification and object detection experiments are inside the respective folders:\n\n- [classification](https://github.com/JosephKJ/ELI/tree/main/classification)\n- [detection](https://github.com/JosephKJ/ELI/tree/main/detection)\n\nEach of these are independent repositories. Please consider them separate. \n\n## Citation\nIf you find our research useful, please consider citing us:\n\n```BibTeX\n\n@inproceedings{joseph2022Energy,\n  title={Energy-based Latent Aligner for Incremental Learning},\n  author={Joseph, KJ and Khan, Salman and Khan, Fahad Shahbaz and Anwar, Rao Muhammad and Balasubramanian, Vineeth},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  year={2022}\n}\n```\n\n\n## Our Related Work\n- Open-world Detection Transformer, CVPR 2022. [Paper](https://arxiv.org/pdf/2112.01513.pdf) | [Code](https://github.com/akshitac8/OW-DETR)\n- Towards Open World Object Detection, CVPR 2021. (Oral) [Paper](https://arxiv.org/abs/2103.02603) | [Code](https://github.com/JosephKJ/OWOD)\n- Incremental Object Detection via Meta-learning, TPAMI 2021. [Paper](https://arxiv.org/abs/2003.08798) | [Code](https://github.com/JosephKJ/iOD)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJosephKJ%2FELI","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJosephKJ%2FELI","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJosephKJ%2FELI/lists"}