{"id":13737624,"url":"https://github.com/GeorgeCazenavette/mtt-distillation","last_synced_at":"2025-05-08T15:30:49.973Z","repository":{"id":37273991,"uuid":"472540119","full_name":"GeorgeCazenavette/mtt-distillation","owner":"GeorgeCazenavette","description":"Official code for our CVPR '22 paper \"Dataset Distillation by Matching Training Trajectories\"","archived":false,"fork":false,"pushed_at":"2024-07-16T12:59:59.000Z","size":40434,"stargazers_count":383,"open_issues_count":9,"forks_count":52,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-08-04T03:10:04.588Z","etag":null,"topics":["artificial-intelligence","computer-vision","machine-learning","synthetic-data"],"latest_commit_sha":null,"homepage":"https://georgecazenavette.github.io/mtt-distillation/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GeorgeCazenavette.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-03-21T22:59:23.000Z","updated_at":"2024-07-31T02:25:54.000Z","dependencies_parsed_at":"2024-01-21T04:23:22.405Z","dependency_job_id":null,"html_url":"https://github.com/GeorgeCazenavette/mtt-distillation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeorgeCazenavette%2Fmtt-distillation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeorgeCazenavette%2Fmtt-distillation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeorgeCazenavette%2Fmtt-distillation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeorgeCazenavette%2Fmtt-distillation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GeorgeCazenavette","download_url":"https://codeload.github.com/GeorgeCazenavette/mtt-distillation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224742086,"owners_count":17362228,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","computer-vision","machine-learning","synthetic-data"],"created_at":"2024-08-03T03:01:55.319Z","updated_at":"2025-05-08T15:30:49.958Z","avatar_url":"https://github.com/GeorgeCazenavette.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Dataset Distillation by Matching Training Trajectories\n\n### [Project Page](https://georgecazenavette.github.io/mtt-distillation/) | [Paper](https://arxiv.org/abs/2203.11932)\n\u003cbr\u003e\n\n![Teaser image](docs/all_grid.png)\n\nThis repo contains code for training expert trajectories and distilling synthetic data from our Dataset Distillation by Matching Training Trajectories paper (CVPR 2022). Please see our [project page](https://georgecazenavette.github.io/mtt-distillation) for more results.\n\n\n\u003e [**Dataset Distillation by Matching Training Trajectories**](https://georgecazenavette.github.io/mtt-distillation/)\u003cbr\u003e\n\u003e [George Cazenavette](https://georgecazenavette.github.io/), [Tongzhou Wang](https://ssnl.github.io/), [Antonio Torralba](https://groups.csail.mit.edu/vision/torralbalab/), [Alexei A. Efros](https://people.eecs.berkeley.edu/~efros/), [Jun-Yan Zhu](https://www.cs.cmu.edu/~junyanz/)\u003cbr\u003e\n\u003e CMU, MIT, UC Berkeley\u003cbr\u003e\n\u003e CVPR 2022 (Oral)\n\nThe task of \"Dataset Distillation\" is to learn a small number of synthetic images such that a model trained on this set alone will have similar test performance as a model trained on the full real dataset.\n\n\u003cimg src='docs/method.gif' width=600\u003e\n\nOur method distills the synthetic dataset by directly optimizing the fake images to induce similar network training dynamics as the full,\nreal dataset. We train \"student\" networks for many iterations on the synthetic data,\nmeasure the error in parameter space between the \"student\" and \"expert\" networks trained on real data,\nand back-propagate through all the student network updates to optimize the synthetic pixels.\n\n\n\n## Wearable ImageNet: Synthesizing Tileable Textures\n\n![Teaser image](docs/texture_teaser.png)\n\nInstead of treating our synthetic data as individual images, we can instead encourage every random crop (with circular padding) on a larger canvas of pixels to induce a good training trajectory. This results in class-based textures that are continuous around their edges.\n\n\u003cimg src='docs/penguins1_horizontal.png' width=600\u003e\n\nGiven these tileable textures, we can apply them to areas that require such properties, such as clothing patterns.\n\n\u003cimg src=\"docs/flamingo_shirt.jpg\" width=\"150\"\u003e\u003cimg src=\"docs/penguin_shirt.jpg\" width=\"150\"\u003e\u003cimg src=\"docs/parrot_dress.jpg\" width=\"150\"\u003e\u003cimg src=\"docs/eagle_jacket.jpg\" width=\"150\"\u003e\n\nVisualizations made using \u003ca href=\"https://tri3d.in/\"\u003eFAB3D\u003c/a\u003e\n\u003cbr\u003e\n\n\n\n### Getting Started\n\nFirst, download our repo:\n```bash\ngit clone https://github.com/GeorgeCazenavette/mtt-distillation.git\ncd mtt-distillation\n```\n\nFor an express instillation, we include ```.yaml``` files.\n\nIf you have an RTX 30XX GPU (or newer), run\n\n```bash\nconda env create -f requirements_11_3.yaml\n```\n\nIf you have an RTX 20XX GPU (or older), run\n\n```bash\nconda env create -f requirements_10_2.yaml\n```\n\nYou can then activate your  conda environment with\n```bash\nconda activate distillation\n```\n##### Quadro Users Take Note:\n```torch.nn.DataParallel``` seems to not work on Quadro A5000 GPUs, and this may extend to other Quadro cards.\n\nIf you experience indefinite hanging during training, try running the process with only 1 GPU by prepending ```CUDA_VISIBLE_DEVICES=0``` to the command.\n\n### Generating Expert Trajectories\nBefore doing any distillation, you'll need to generate some expert trajectories using ```buffer.py```\n\nThe following command will train 100 ConvNet models on CIFAR-100 with ZCA whitening for 50 epochs each:\n```bash\npython buffer.py --dataset=CIFAR100 --model=ConvNet --train_epochs=50 --num_experts=100 --zca --buffer_path={path_to_buffer_storage} --data_path={path_to_dataset}\n```\nWe used 50 epochs with the default learning rate for all of our experts.\nWorse (but still interesting) results can be obtained faster through training fewer experts by changing ```--num_experts```. Note that experts need only be trained once and can be re-used for multiple distillation experiments.\n\n### Distillation by Matching Training Trajectories\nThe following command will then use the buffers we just generated to distill CIFAR-100 down to just 1 image per class:\n```bash\npython distill.py --dataset=CIFAR100 --ipc=1 --syn_steps=20 --expert_epochs=3 --max_start_epoch=20 --zca --lr_img=1000 --lr_lr=1e-05 --lr_teacher=0.01 --buffer_path={path_to_buffer_storage} --data_path={path_to_dataset}\n```\n\n\u003cimg src='docs/animation.gif' width=600\u003e\n\nPlease find a full list of hyper-parameters below:\n\n![image](https://user-images.githubusercontent.com/18726777/184226412-7bd0d577-225b-487c-8c9c-23f6462ca7d0.png)\n\n\n### ImageNet\nOur method can also distill subsets of ```ImageNet``` into low-support synthetic sets.\n\nWhen generating expert trajectories with ```buffer.py``` or distilling the dataset with ```distill.py```, you must designate a named subset of ImageNet with the ```--subset``` flag.\n\nFor example,\n\n```bash\npython distill.py --dataset=ImageNet --subset=imagefruit --model=ConvNetD5 --ipc=1 --res=128 --syn_steps=20 --expert_epochs=2 --max_start_epoch=10 --lr_img=1000 --lr_lr=1e-06 --lr_teacher=0.01 --buffer_path={path_to_buffer_storage} --data_path={path_to_dataset}\n```\nwill distill the ```imagefruit``` subset (at 128x128 resolution) into the following 10 images\n\n\u003cimg src='docs/imagefruit.png' width=600\u003e\n\nTo register your own ImageNet subset, you can add it to the ```Config``` class at the top of ```utils.py```.\n\nSimply create a list with the desired class ID's and add it to the dictionary.\n\n[This gist](https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a) contains a list of all 1k ImageNet classes and their corresponding numbers.\n\n\n### Texture Distillation\nYou can also use the same set of expert trajectories (except those using ZCA) to distill classes into toroidal textures by simply adding the ```--texture``` flag.\n\nFor example,\n\n```bash\npython distill.py --texture --dataset=ImageNet --subset=imagesquawk --model=ConvNetD5 --ipc=1 --res=256 --syn_steps=20 --expert_epochs=2 --max_start_epoch=10 --lr_img=1000 --lr_lr=1e-06 --lr_teacher=0.01 --buffer_path={path_to_buffer_storage} --data_path={path_to_dataset}\n```\nwill distill the ```imagesquawk``` subset (at 256x256 resolution) into the following 10 textures\n\n\u003cimg src='docs/imagesquawk_tex.png' width=600\u003e\n\n## Acknowledgments\nWe would like to thank Alexander Li, Assaf Shocher,  Gokul Swamy, Kangle Deng, Ruihan Gao, Nupur Kumari, Muyang Li, Gaurav Parmar, Chonghyuk Song, Sheng-Yu Wang, and Bingliang Zhang as well as Simon Lucey's Vision Group at the University of Adelaide for their valuable feedback. This work is supported, in part, by the NSF Graduate Research Fellowship under Grant No. DGE1745016 and grants from J.P. Morgan Chase, IBM, and SAP. Our code is adapted from https://github.com/VICO-UoE/DatasetCondensation\n\n## Related Work\n\u003col\u003e\n\u003cli\u003e\n    Tongzhou Wang et al. \u003ca href=\"https://ssnl.github.io/dataset_distillation/\"\u003e\"Dataset Distillation\"\u003c/a\u003e, in arXiv preprint 2018\n\u003c/li\u003e\n\u003cli\u003e\n    Bo Zhao et al. \u003ca href=\"https://arxiv.org/abs/2006.05929\"\u003e\"Dataset Condensation with Gradient Matching\"\u003c/a\u003e, in ICLR 2020\n\u003c/li\u003e\n\u003cli\u003e\n    Bo Zhao and Hakan Bilen. \u003ca href=\"https://arxiv.org/abs/2102.08259\"\u003e\"Dataset Condensation with Differentiable Siamese Augmentation\"\u003c/a\u003e, in ICML 2021\n\u003c/li\u003e\n\u003cli\u003e\n    Timothy Nguyen et al. \u003ca href=\"https://arxiv.org/abs/2011.00050\"\u003e\"Dataset Meta-Learning from Kernel Ridge-Regression\"\u003c/a\u003e, in ICLR 2021\n\u003c/li\u003e\n\u003cli\u003e\n    Timothy Nguyen et al. \u003ca href=\"https://arxiv.org/abs/2107.13034\"\u003e\"Dataset Distillation with Infinitely Wide Convolutional Networks\"\u003c/a\u003e, in NeurIPS 2021\n\u003c/li\u003e\n\u003cli\u003e\n    Bo Zhao and Hakan Bilen. \u003ca href=\"https://arxiv.org/abs/2110.04181\"\u003e\"Dataset Condensation with Distribution Matching\"\u003c/a\u003e, in arXiv preprint 2021\n\u003c/li\u003e\n\u003cli\u003e\n    Kai Wang et al. \u003ca href=\"https://arxiv.org/abs/2203.01531\"\u003e\"CAFE: Learning to Condense Dataset by Aligning Features\"\u003c/a\u003e, in CVPR 2022\n\u003c/li\u003e\n\u003c/ol\u003e\n\n# Reference\nIf you find our code useful for your research, please cite our paper.\n```\n@inproceedings{\ncazenavette2022distillation,\ntitle={Dataset Distillation by Matching Training Trajectories},\nauthor={George Cazenavette and Tongzhou Wang and Antonio Torralba and Alexei A. Efros and Jun-Yan Zhu},\nbooktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\nyear={2022}\n}\n```\n\n```\n@iniroceedings{\ncazenavette2022textures,\ntitle={Wearable ImageNet: Synthesizing Tileable Textures via Dataset Distillation},\nauthor= {George Cazenavette and Tongzhou Wang and Antonio Torralba and Alexei A. Efros and Jun-Yan Zhu},\nbooktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},\nyear={2022},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGeorgeCazenavette%2Fmtt-distillation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FGeorgeCazenavette%2Fmtt-distillation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGeorgeCazenavette%2Fmtt-distillation/lists"}