{"id":16388514,"url":"https://github.com/arthurdouillard/dytox","last_synced_at":"2025-03-16T16:31:04.244Z","repository":{"id":41083067,"uuid":"430661308","full_name":"arthurdouillard/dytox","owner":"arthurdouillard","description":"Dynamic Token Expansion with Continual Transformers, accepted at CVPR 2022","archived":false,"fork":false,"pushed_at":"2022-06-30T16:31:27.000Z","size":3562,"stargazers_count":137,"open_issues_count":6,"forks_count":17,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-27T11:36:21.128Z","etag":null,"topics":["cifar100","computer-vision","continual-learning","dynamic-network","imagenet","transformer"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2111.11326","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arthurdouillard.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-22T10:28:25.000Z","updated_at":"2025-02-27T03:08:20.000Z","dependencies_parsed_at":"2022-08-15T12:50:09.494Z","dependency_job_id":null,"html_url":"https://github.com/arthurdouillard/dytox","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthurdouillard%2Fdytox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthurdouillard%2Fdytox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthurdouillard%2Fdytox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthurdouillard%2Fdytox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arthurdouillard","download_url":"https://codeload.github.com/arthurdouillard/dytox/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243822283,"owners_count":20353499,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cifar100","computer-vision","continual-learning","dynamic-network","imagenet","transformer"],"created_at":"2024-10-11T04:29:17.058Z","updated_at":"2025-03-16T16:31:03.724Z","avatar_url":"https://github.com/arthurdouillard.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# DyTox\n\n## Transformers for Continual Learning with DYnamic TOken eXpansion\n\n[![Paper](https://img.shields.io/badge/arXiv-2004.13513-brightgreen)](https://arxiv.org/abs/2111.11326)\n![CVPR](https://img.shields.io/badge/CVPR-2022-blue)\n[![Youtube](https://img.shields.io/badge/Youtube-link-red)](https://www.youtube.com/watch?v=O1GNm4WdrNw)\n\n![DyTox main figure](images/dytox.png)\n\nWelcome to DyTox, the first transformer designed explicitly for Continual Learning!\n\u003c/div\u003e\n\n\nWork led by [Arthur Douillard](https://arthurdouillard.com/) and co-authored with [Alexandre Ramé](https://alexrame.github.io/),\n[Guillaume Couairon](https://phazcode.gitlab.io/about/), and [Matthieu Cord](http://webia.lip6.fr/~cord/).\n\nSee our erratum [here](erratum_distributed.md).\n\n# Installation\n\nYou first need to have a working python installation with version \u003e= 3.6.\n\nThen create a conda env, and install the libraries in the `requirements.txt`: it\nincludes pytorch and torchvision for the building blocks of our model. It also\nincludes continuum for data loader made for continual learning, and `timm`.\n\nNote that this code is heavily based on the great codebase of [DeiT](https://github.com/facebookresearch/deit).\n\n# Launching an experiment\n\nCIFAR100 dataset will be auto-downloaded, however you must download yourself\nImageNet.\n\nEach command needs three options files:\n- which dataset you want to run on and in which settings (i.e. how many steps)\n- Which class ordering, by default it'll be 0-\u003eC-1, but we used the class ordering\n  proposed by DER (and which all baselines also follow)\n- Which model version you want (DyTox, DyTox+, and DyTox++ (see supplementary\n  about that last one))\n\nTo launch DyTox on CIFAR100 in the 50 steps setting on the GPUs #0 and #1:\n\n```bash\nbash train.sh 0,1 \\\n    --options options/data/cifar100_2-2.yaml options/data/cifar100_order1.yaml options/model/cifar_dytox.yaml \\\n    --name dytox \\\n    --data-path MY_PATH_TO_DATASET \\\n    --output-basedir PATH_TO_SAVE_CHECKPOINTS \\\n    --memory-size 1000\n```\n\nFolders will be auto-created with the results at\n`logs/cifar/2-2/{DATE}/{DATE}/{DATE}-dytox`.\n\nLikewise, to launch DyTox+ and DyTox++, simply change the options. It's also the\nsame for datasets. Note that we provided 3 different class orders (from DER's\nimplem) for CIFAR100, and we average the results in our paper.\n\nWhen you have a doubt about the options to use, just check what was defined in the\nyaml option files in the folder `./options`.\n\n# Resuming an experiment\n\nSome exp can be slow and you may need to resume it, like for ImageNet1000.\n\nFirst locate the checkpoints folder (by default at `./checkpoints/` if you didn't\nspecify any `output-basedir`) where your experiment first ran. Then run the\nfollowing command (I'm taking ImageNet1000 as an example but you could have\ntaken any models and datasets):\n\n```bash\nbash train.sh 0,1 \\\n    --options options/data/imagenet1000_100-100.yaml options/data/imagenet1000_order1.yaml options/model/imagenet_dytox.yaml \\\n    --name dytox \\\n    --data-path MY_PATH_TO_DATASET \\\n    --resume MY_PATH_TO_CKPT_FOLDER_OF_EXP \\\n    --start-task TASK_ID_STARTING_FROM_0_OF_WHEN_THE_EXP_HAD_STOPPED \\\n    --memory-size 20000\n```\n\n# Results\n\n## ImageNet\n\n![ImageNet figure results](images/imagenet1000.png)\n![ImageNet table results](images/imagenet_table.png)\n\n## CIFAR100\n\n![CIFAR figure results](images/cifar.png)\n![CIFAR table results](images/cifar_table.png)\n\n# Frequenly Asked Questions\n\n\u003e Is DyTox pretrained?\n\n- No! It's trained from scratch for fair comparison with previous SotAs\n\n\u003e Your encoder is made actually of ConVit blocks, can I use something else? Like a MHSA or Swin?\n\n- Yes! I've used ConVit blocks because they trained well from scratch on small datasets like CIFAR\n\n\u003e Can I add a new datasets?\n\n- Yes! You can add any datasets in [continual/datasets.py](https://github.com/arthurdouillard/dytox/blob/main/continual/datasets.py). They just need to be compatible with the [Continuum](https://github.com/Continvvm/continuum) library. But check it out, they have a lot of [implemented datasets](https://continuum.readthedocs.io/en/latest/tutorials/datasets/dataset.html)\n\n\u003e Could I use a convolution-based backbone for the encoder instead of transformer blocks?\n\n- Yes! You'd need to modify the [DyTox module](https://github.com/arthurdouillard/dytox/blob/main/continual/dytox.py). I already provide several [CNNs](https://github.com/arthurdouillard/dytox/tree/main/continual/cnn). Note that for best results, you may want to remove the ultimate block of the CNN and add strides so that the spatial features are big enough at the end to make enough \"tokens\"\n\n\u003e Do I need to install nvidia's apex for the mix precision?\n\n- No! DyTox uses Pytorch native mix precision\n\n\u003e Can I run DyTox on a single GPU instead of two?\n\n- In theory, yes. Although the performance is a [bit lower](https://github.com/arthurdouillard/dytox/issues/2). I'll try to find the root cause of this. But on two GPUs the results are perfectly reproducible.\n\n\u003e What is this finetuning phase?\n\n- New classes data is downsampled to the same amount of old classes data stored in the rehearsal memory. And the encoder is frozen. You can see which modules are frozen in which task in the [options files](https://github.com/arthurdouillard/dytox/blob/main/options/model/cifar_dytox.yaml#L35-L36).\n\n\u003e Memory setting?\n\n- If you use distributed memory (default), use 20/N images per class with N the number of used GPUs. Thus for 2 GPUs, it's `--memory-size 1000` for CIFAR100 and ImageNet100 and `--memory-size 10000` for Imagenet1000. If you use global memory (`--global-memory`), use 20 images per class.\n\n\u003e Distributed memory?\n\n- See [here](erratum_distributed.md).\n\n\u003e Results obtained on \u003e=2 GPUs are slightly different from the first version of the paper?\n\n- See [here](erratum_distributed.md).\n\n\n\n# Citation\n\nIf you compare to this model or use this code for any means, please cite us! Thanks :)\n\n```\n@inproceedings{douillard2021dytox,\n  title     = {DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion},\n  author    = {Douillard, Arthur and Ram\\'e, Alexandre and Couairon, Guillaume and Cord, Matthieu},\n  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\n  year      = {2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthurdouillard%2Fdytox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farthurdouillard%2Fdytox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthurdouillard%2Fdytox/lists"}