{"id":13737828,"url":"https://github.com/ryanchankh/mcr2","last_synced_at":"2025-05-08T15:31:35.855Z","repository":{"id":37649903,"uuid":"260047524","full_name":"ryanchankh/mcr2","owner":"ryanchankh","description":"Official Implementation of Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction (2020)","archived":false,"fork":false,"pushed_at":"2022-12-08T10:11:52.000Z","size":639,"stargazers_count":193,"open_issues_count":10,"forks_count":43,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-11-15T06:32:32.159Z","etag":null,"topics":["clustering","deep-learning","information-theory","representation-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ryanchankh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-29T21:15:08.000Z","updated_at":"2024-11-13T08:49:55.000Z","dependencies_parsed_at":"2023-01-24T23:45:16.242Z","dependency_job_id":null,"html_url":"https://github.com/ryanchankh/mcr2","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanchankh%2Fmcr2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanchankh%2Fmcr2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanchankh%2Fmcr2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanchankh%2Fmcr2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ryanchankh","download_url":"https://codeload.github.com/ryanchankh/mcr2/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253096061,"owners_count":21853536,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clustering","deep-learning","information-theory","representation-learning"],"created_at":"2024-08-03T03:02:02.486Z","updated_at":"2025-05-08T15:31:35.504Z","avatar_url":"https://github.com/ryanchankh.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction\nThis repository is the official implementation of [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://arxiv.org/abs/2006.08558) (2020) by [Yaodong Yu](https://yaodongyu.github.io/)* (UC Berkeley), [Kwan Ho Ryan Chan](ryanchankh.github.io)* (UC Berkeley), [Chong You](https://sites.google.com/view/cyou) (UC Berkeley), Chaobing Song (UC Berkeley) and Yi Ma (UC Berkeley).\n\n## What is Maxmial Coding Rate Reduction? \nOur goal is to learn a mapping that maps the high-dimensional data that lies in a low-dimensional manifold to low-dimensional subspaces with the following three properties: \n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"images/teaser.png\" width=\"350\"\\\u003e\u003cbr\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\n1. _Between-Class Discriminative_: Features of samples from different classes/clusters should be highly uncorrelatedand belong to different low-dimensional linear subspaces\n2. _Within-Class Compressible_: Features of samples from the same class/cluster should be relatively correlated in a sense that they belong to a low-dimensional linear subspace\n3. _Maximally Diverse Representation_: Dimension (or variance) of features for each class/cluster should beas large as possibleas long as they stay uncorrelated from the other classes\n\nTo achieve this, we propose an objective function called **Maximal Coding Rate Reduction** (MCR\u003csup\u003e2\u003c/sup\u003e). In our paper, we provide not only theoretical guarantees to the desired properties upon convergence, but also practical properties such as robustness to label corruption and empirical results such as state-of-the-art unsupervised clustering performance. For more details on algorithm design, please refer to our paper.\n\n## Requirements\n- This codebase is written for `python3`.\n- To install necessary python packages, run `pip install -r requirements.txt`.\n\n\n## Training\n### Basics\n- All functions used in training can be found in [`train_func.py`](./train_func.py), which includes: `load_checkpoint(...)`, `load_trainset(...)`, etc. For implementation details please refer to docstring. \n- Code for training are in the following files: [`train_sup.py`](./train_sup.py) and [`train_selfsup.py`](./train_selfsup.py). Each has its own command options. \n- Augmentations is used in unsupervised and contrastive setting. Check [`augmentloader.py`](./augmentloader.py) for implementation details. \n\n### Supervised Setting\n\n```\nusage: train_sup.py [-h] [--arch ARCH] [--fd FD] [--data DATA] [--epo EPO]\n                    [--bs BS] [--lr LR] [--mom MOM] [--wd WD] [--gam1 GAM1]\n                    [--gam2 GAM2] [--eps EPS] [--lcr LCR] [--lcs LCS]\n                    [--tail TAIL] [--transform TRANSFORM]\n                    [--save_dir SAVE_DIR] [--data_dir DATA_DIR]\n                    [--pretrain_dir PRETRAIN_DIR]\n                    [--pretrain_epo PRETRAIN_EPO]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --arch ARCH           architecture for deep neural network (default: resnet18)\n  --fd FD               dimension of feature dimension (default: 128)\n  --data DATA           dataset for training (default: CIFAR10)\n  --epo EPO             number of epochs for training (default: 500)\n  --bs BS               input batch size for training (default: 1000)\n  --lr LR               learning rate (default: 0.0001)\n  --mom MOM             momentum (default: 0.9)\n  --wd WD               weight decay (default: 5e-4)\n  --gam1 GAM1           gamma1 for tuning empirical loss (default: 1.)\n  --gam2 GAM2           gamma2 for tuning empirical loss (default: 1.)\n  --eps EPS             eps squared (default: 0.5)\n  --lcr LCR             label corruption ratio (default: 0)\n  --lcs LCS             label corruption seed for index randomization (default: 10)\n  --tail TAIL           extra information to add to folder name\n  --transform TRANSFORM transform applied to trainset (default: default\n  --save_dir SAVE_DIR   base directory for saving PyTorch model. (default: ./saved_models/)\n  --data_dir DATA_DIR   base directory for saving PyTorch model. (default: ./data/)\n  --pretrain_dir PRETRAIN_DIR load pretrained checkpoint for assigning labels\n  --pretrain_epo PRETRAIN_EPO load pretrained epoch for assigning labels\n```\nExamples at a later section. \n\n### Self-supervised Setting\n\n```\nusage: train_selfsup.py [-h] [--arch ARCH] [--fd FD] [--data DATA] [--epo EPO]\n                        [--bs BS] [--aug AUG] [--lr LR] [--mom MOM] [--wd WD]\n                        [--gam1 GAM1] [--gam2 GAM2] [--eps EPS] [--tail TAIL]\n                        [--transform TRANSFORM] [--sampler SAMPLER]\n                        [--pretrain_dir PRETRAIN_DIR]\n                        [--pretrain_epo PRETRAIN_EPO] [--save_dir SAVE_DIR]\n                        [--data_dir DATA_DIR]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --arch ARCH           architecture for deep neural network (default: resnet18)\n  --fd FD               dimension of feature dimension (default: 32)\n  --data DATA           dataset for training (default: CIFAR10)\n  --epo EPO             number of epochs for training (default: 50)\n  --bs BS               input batch size for training (default: 1000)\n  --aug AUG             number of augmentations per mini-batch (default: 49)\n  --lr LR               learning rate (default: 0.001)\n  --mom MOM             momentum (default: 0.9)\n  --wd WD               weight decay (default: 5e-4)\n  --gam1 GAM1           gamma1 for tuning empirical loss (default: 1.0)\n  --gam2 GAM2           gamma2 for tuning empirical loss (default: 10)\n  --eps EPS             eps squared (default: 2)\n  --tail TAIL           extra information to add to folder name\n  --transform TRANSFORM transform applied to trainset (default: default\n  --sampler SAMPLER     sampler used in augmentloader (default: random\n  --pretrain_dir PRETRAIN_DIR load pretrained checkpoint for assigning labels\n  --pretrain_epo PRETRAIN_EPO load pretrained epoch for assigning labels\n  --save_dir SAVE_DIR   base directory for saving PyTorch model. (default: ./saved_models/)\n  --data_dir DATA_DIR   base directory for saving PyTorch model. (default: ./data/)\n```\nExamples at a later section.\n\n\n## Evaluation\nTesting methods available are: `svm`, `knn`, `nearsub`, `kmeans`, `ensc`. Each method also has options for testing hyperparameters, such as `--k` for top `k` components in kNN. Methods can also be chained. Checkpoint can also be specified using `--epoch` option. Please refer to [`evaluate.py`](./evaluate.py) and [`cluster.py`](./cluster.py) and for more implementation details. \n\n- Command Options\n\n```\nusage: evaluate.py [-h] [--model_dir MODEL_DIR] [--svm] [--knn] [--nearsub]\n                   [--kmeans] [--ensc] [--epoch EPOCH] [--k K] [--n N]\n                   [--gam GAM] [--tau TAU] [--n_comp N_COMP] [--save]\n                   [--data_dir DATA_DIR]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --model_dir MODEL_DIR base directory for saving PyTorch model.\n  --svm                 evaluate using SVM\n  --knn                 evaluate using kNN measuring cosine similarity\n  --nearsub             evaluate using Nearest Subspace\n  --kmeans              evaluate using KMeans\n  --ensc                evaluate using Elastic Net Subspace Clustering\n  --epoch EPOCH         which epoch for evaluation\n  --k K                 top k components for kNN\n  --n N                 number of clusters for cluster (default: 10)\n  --gam GAM             gamma paramter for subspace clustering (default: 100)\n  --tau TAU             tau paramter for subspace clustering (default: 1.0)\n  --n_comp N_COMP       number of components for PCA (default: 30)\n  --save                save labels\n  --data_dir DATA_DIR   path to dataset\n```\n- An example for evaluation:\n\n```\n$ python3 evaluate.py --knn --nearsub --k 10 --model_dir saved_models/sup_resnet18+128_cifar10_epo500_bs1000_lr0.001_mom0.9_wd0.0005_gam11.0_gam210.0_eps0.5_lcr0\n```\n, which runs kNN with top 10 components and nearest subspace on the latest checkpoint in `model_dir`.\n\n\n## Others\n- **(Extracting Features as .zip)** To extract the features, use `extract.py`.\n- **(Plotting figures)** Plot functions are located in `plot.py`. Plots will be saved in `(model_dir)/figures/`\n- **(Reproduce Results in Paper)**\n\n#### Commands for Supervised Learning Setting\n\n```\n$ python3 train_sup.py --arch resnet18 --data cifar10 --fd 128 --epo 500 --bs 1000 --eps 0.5 --gam1 1 --gam2 1 --lr 0.01 --lcr 0.0\n$ python3 train_sup.py --arch resnet18 --data cifar10 --fd 128 --epo 500 --bs 1000 --eps 0.5 --gam1 1 --gam2 1 --lr 0.01 --lcr 0.1\n$ python3 train_sup.py --arch resnet18 --data cifar10 --fd 128 --epo 500 --bs 1000 --eps 0.5 --gam1 1 --gam2 1 --lr 0.01 --lcr 0.2\n$ python3 train_sup.py --arch resnet18 --data cifar10 --fd 128 --epo 500 --bs 1000 --eps 0.5 --gam1 1 --gam2 1 --lr 0.01 --lcr 0.3\n$ python3 train_sup.py --arch resnet18 --data cifar10 --fd 128 --epo 500 --bs 1000 --eps 0.5 --gam1 1 --gam2 1 --lr 0.01 --lcr 0.4\n$ python3 train_sup.py --arch resnet18 --data cifar10 --fd 128 --epo 500 --bs 1000 --eps 0.5 --gam1 1 --gam2 1 --lr 0.01 --lcr 0.5\n```\n\n#### Commands for Self-supervised Learning Setting\n```\n$ python3 train_selfsup.py --arch resnet18ctrl --data cifar10 --fd 128 --epo 100 --bs 1000 --eps 0.5 --gam1 20 --gam2 0.05 --lr 0.1 --aug 50 --transform cifar\n$ python3 train_selfsup.py --arch resnet18ctrl --data cifar100 --fd 128 --epo 100 --bs 1000 --eps 0.5 --gam1 20 --gam2 0.05 --lr 0.1 --aug 50 --transform cifar\n$ python3 train_selfsup.py --arch resnet18stl --data stl10 --fd 128 --epo 100 --bs 1000 --eps 0.5 --gam1 20 --gam2 0.05 --lr 0.1 --aug 50 --transform stl10\n```\n\n## Reference\nFor technical details and full experimental results, please check [the paper](https://arxiv.org/abs/2006.08558). If you have used our work in your own, please consider citing: \n\n```\n@article{yu2020learning,\n  title={Learning diverse and discriminative representations via the principle of maximal coding rate reduction},\n  author={Yu, Yaodong and Chan, Kwan Ho Ryan and You, Chong and Song, Chaobing and Ma, Yi},\n  journal={Advances in Neural Information Processing Systems},\n  volume={33},\n  year={2020}\n}\n\n```\n\n## License and Contributing\n- This README is formatted based on [paperswithcode](https://github.com/paperswithcode/releasing-research-code).\n- Feel free to post issues via Github. \n\n## Contact\nPlease contact [ryanchankh@berkeley.edu](ryanchankh@berkeley.edu) and [yyu@eecs.berkeley.edu](yyu@eecs.berkeley.edu) if you have any question on the codes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanchankh%2Fmcr2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fryanchankh%2Fmcr2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanchankh%2Fmcr2/lists"}