{"id":22354734,"url":"https://github.com/pprp/pytorch-cifar-model-hub","last_synced_at":"2025-07-30T09:31:26.193Z","repository":{"id":42865071,"uuid":"419278481","full_name":"pprp/PyTorch-CIFAR-Model-Hub","owner":"pprp","description":"Implementation of Conv-based and Vit-based networks designed for CIFAR.","archived":false,"fork":false,"pushed_at":"2023-01-02T10:27:38.000Z","size":1678,"stargazers_count":71,"open_issues_count":1,"forks_count":12,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-06T05:32:14.267Z","etag":null,"topics":["cifar10","cifar10-classification","cnn-classification","image-classification","pytorch","tricks","vision-transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pprp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-20T10:09:51.000Z","updated_at":"2025-03-26T08:58:06.000Z","dependencies_parsed_at":"2023-02-01T02:30:46.728Z","dependency_job_id":null,"html_url":"https://github.com/pprp/PyTorch-CIFAR-Model-Hub","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pprp/PyTorch-CIFAR-Model-Hub","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2FPyTorch-CIFAR-Model-Hub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2FPyTorch-CIFAR-Model-Hub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2FPyTorch-CIFAR-Model-Hub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2FPyTorch-CIFAR-Model-Hub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pprp","download_url":"https://codeload.github.com/pprp/PyTorch-CIFAR-Model-Hub/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2FPyTorch-CIFAR-Model-Hub/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267845677,"owners_count":24153774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cifar10","cifar10-classification","cnn-classification","image-classification","pytorch","tricks","vision-transformer"],"created_at":"2024-12-04T13:14:59.858Z","updated_at":"2025-07-30T09:31:25.851Z","avatar_url":"https://github.com/pprp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A PyTorch Implementation of CIFAR Tricks\n\n![](figures/logo.png)\n\nCIFAR10数据集上CNN模型、Transformer模型以及Tricks，数据增强，正则化方法等，并进行了实现。欢迎提issue或者进行PR。\n\n## 0. Requirements\n\n- Python 3.6+\n- torch=1.8.0+cu111\n- torchvision+0.9.0+cu111\n- tqdm=4.26.0\n- PyYAML=6.0\n- einops\n- torchsummary\n\n\n## 1. Implements\n\n### 1.0 Models\n\nvision Transformer:\n\n| Model              | GPU Mem | Top1:train | Top1:val | weight:M |\n| ------------------ | ------- | ---------- | -------- | -------- |\n| vision_transformer | 2869M   | 68.96      | 69.02    | 47.6     |\n| mobilevit_s        | 2009M   | 98.83      | 92.50    | 19.2     |\n| mobilevit_xs       | 1681M   | 98.22      | 91.77    | 7.78     |\n| mobilevit_xxs      | 1175M   | 96.40      | 90.17    | 4.0      |\n| coatnet_0          | 1433M   | 99.94      | 90.15    | 64.9     |\n| coatnet_1          | 2089M   | 99.97      | 90.09    | 123      |\n| coatnet_2          | 2405M   | 99.99      | 90.86    | 208      |\n| cvt                | 2593M   | 94.64      | 84.74    | 75       |\n| swin_t             | 3927M   | 93.24      | 86.09    | 104      |\n| swin_s             | 6707M   | 90.27      | 83.68    | 184      |\n\n\n\n### 1.1 Tricks\n\n- Warmup\n- Cosine LR Decay\n- SAM\n- Label Smooth\n- KD\n- Adabound\n- Xavier Kaiming init\n- lr finder\n\n### 1.2 Augmentation\n\n- Auto Augmentation\n- Cutout\n- Mixup\n- RICAP\n- Random Erase\n- ShakeDrop\n\n\n\n## 2. Training\n\n### 2.1 CIFAR-10训练示例\n\nWideResNet28-10 baseline on CIFAR-10:\n\n```\npython train.py --dataset cifar10\n```\n\nWideResNet28-10 +RICAP on CIFAR-10:\n\n```\npython train.py --dataset cifar10 --ricap True\n```\n\nWideResNet28-10 +Random Erasing on CIFAR-10:\n\n```\npython train.py --dataset cifar10 --random-erase True\n```\n\nWideResNet28-10 +Mixup on CIFAR-10:\n\n```\npython train.py --dataset cifar10 --mixup True\n```\n\n更多脚本可以参考 scripts/run.sh\n\n## 3. Results\n\n### 3.1 原pytorch-ricap的结果\n\n| Model                           |    Error rate     |   Loss    | Error rate (paper) |\n| :------------------------------ | :---------------: | :-------: | :----------------: |\n| WideResNet28-10 baseline        |   3.82（96.18）   |   0.158   |        3.89        |\n| WideResNet28-10 +RICAP          | **2.82（97.18）** |   0.141   |      **2.85**      |\n| WideResNet28-10 +Random Erasing |   3.18（96.82）   | **0.114** |        4.65        |\n| WideResNet28-10 +Mixup          |   3.02（96.98）   |   0.158   |        3.02        |\n\n### 3.2 Reimplementation结果\n\n| Model                           |    Error rate     | Loss  | Error rate (paper) |\n| :------------------------------ | :---------------: | :---: | :----------------: |\n| WideResNet28-10 baseline        |   3.78（96.22）   |       |        3.89        |\n| WideResNet28-10 +RICAP          | **2.81（97.19）** |       |      **2.85**      |\n| WideResNet28-10 +Random Erasing |   3.03（96.97）   | 0.113 |        4.65        |\n| WideResNet28-10 +Mixup          |   2.93（97.07）   | 0.158 |        3.02        |\n\n### 3.3 Half data快速训练验证各网络结构\n\nreimplementation models(no augmentation, half data，epoch200，bs128)\n\n| Model                        |  Error rate   |  Loss  |\n| :--------------------------- | :-----------: | :----: |\n| lenet(cpu爆炸)               |   （70.76）   |        |\n| wideresnet                   | 3.78（96.22） |        |\n| resnet20                     |   （89.72）   |        |\n| senet                        |   （92.34）   |        |\n| resnet18                     |   （92.08）   |        |\n| resnet34                     |   （92.48）   |        |\n| resnet50                     |   （91.72）   |        |\n| regnet                       |   （92.58）   |        |\n| nasnet                       |  out of mem   |        |\n| shake_resnet26_2x32d         |   （93.06）   |        |\n| shake_resnet26_2x64d         |   （94.14）   |        |\n| densenet                     |   （92.06）   |        |\n| dla                          |   （92.58）   |        |\n| googlenet                    |   （91.90）   | 0.2675 |\n| efficientnetb0(利用率低且慢) |   （86.82）   | 0.5024 |\n| mobilenet(利用率低)          |   （89.18）   |        |\n| mobilenetv2                  |   （91.06）   |        |\n| pnasnet                      |   （90.44）   |        |\n| preact_resnet                |   （90.76）   |        |\n| resnext                      |   （92.30）   |        |\n| vgg(cpugpu利用率都高)        |   （88.38）   |        |\n| inceptionv3                  |   （91.84）   |        |\n| inceptionv4                  |   （91.10）   |        |\n| inception_resnet_v2          |   （83.46）   |        |\n| rir                          |   （92.34）   | 0.3932 |\n| squeezenet(CPU利用率高)      |   （89.16）   | 0.4311 |\n| stochastic_depth_resnet18    |   （90.22）   |        |\n| xception                     |               |        |\n| dpn                          |   （92.06）   | 0.3002 |\n| ge_resnext29_8x64d           |   （93.86）   |  巨慢  |\n\n### 3.4 测试cpu gpu影响\n\nTEST: scale/kernel ToyNet\n\n**修改网络的卷积层深度，并进行训练，可以得到以下结论：**\n\n结论：lenet这种卷积量比较少，只有两层的，cpu利用率高，gpu利用率低。在这个基础上增加深度，用vgg那种直筒方式增加深度，发现深度越深，cpu利用率越低，gpu利用率越高。\n\n**修改训练过程的batch size，可以得到以下结论：**\n\n结论：bs会影响收敛效果。\n\n### 3.5 StepLR优化下测试cutout和mixup\n\n| architecture         | epoch | cutout | mixup | C10 test acc (%) |\n| -------------------- | ----- | ------ | ----- | ---------------- |\n| shake_resnet26_2x64d | 200   |        |       | 96.33            |\n| shake_resnet26_2x64d | 200   | √      |       | 96.99            |\n| shake_resnet26_2x64d | 200   |        | √     | 96.60            |\n| shake_resnet26_2x64d | 200   | √      | √     | 96.46            |\n\n### 3.6 测试SAM,ASAM,Cosine,LabelSmooth\n\n| architecture         | epoch | SAM  | ASAM | Cosine LR Decay | LabelSmooth | C10 test acc (%) |\n| -------------------- | ----- | ---- | ---- | --------------- | ----------- | ---------------- |\n| shake_resnet26_2x64d | 200   | √    |      |                 |             | 96.51            |\n| shake_resnet26_2x64d | 200   |      | √    |                 |             | 96.80            |\n| shake_resnet26_2x64d | 200   |      |      | √               |             | 96.61            |\n| shake_resnet26_2x64d | 200   |      |      |                 | √           | 96.57            |\n\nPS:其他库在加长训练过程（epoch=1800）情况下可以实现 `shake_resnet26_2x64d` achieved **97.71%** test accuracy with `cutout` and `mixup`!!\n\n### 3.7 测试cosine lr + shake\n\n| architecture         | epoch | cutout | mixup | C10 test acc (%) |\n| -------------------- | ----- | ------ | ----- | ---------------- |\n| shake_resnet26_2x64d | 300   |        |       | 96.66            |\n| shake_resnet26_2x64d | 300   | √      |       | **97.21**        |\n| shake_resnet26_2x64d | 300   |        | √     | 96.90            |\n| shake_resnet26_2x64d | 300   | √      | √     | 96.73            |\n\n1800 epoch CIFAR ZOO中结果，由于耗时过久，未进行复现。\n\n| architecture         | epoch | cutout | mixup | C10 test acc (%)       |\n| -------------------- | ----- | ------ | ----- | ---------------------- |\n| shake_resnet26_2x64d | 1800  |        |       | 96.94（cifar zoo）     |\n| shake_resnet26_2x64d | 1800  | √      |       | **97.20**（cifar zoo） |\n| shake_resnet26_2x64d | 1800  |        | √     | **97.42**（cifar zoo） |\n| shake_resnet26_2x64d | 1800  | √      | √     | **97.71**（cifar zoo） |\n\n### 3.8 Divide and Co-training方案研究\n\n- lr:\n  - warmup (20 epoch)\n  - cosine lr decay\n  - lr=0.1\n  - total epoch(300 epoch)\n- bs=128\n- aug:\n  - Random Crop and resize\n  - Random left-right flipping\n  - AutoAugment\n  - Normalization\n  - Random Erasing\n  - Mixup\n- weight decay=5e-4 (bias and bn undecayed)\n- kaiming weight init\n- optimizer: nesterov\n\n复现：((**v100:gpu1**)  4min*300/60=20h) top1: **97.59%** 本项目目前最高值。\n\n```bash\npython train.py --model 'pyramidnet272' \\\n                --name 'divide-co-train' \\\n                --autoaugmentation True \\\n                --random-erase True \\\n                --mixup True \\\n                --epochs 300 \\\n                --sched 'warmcosine' \\\n                --optims 'nesterov' \\\n                --bs 128 \\\n                --root '/home/dpj/project/data'\n```\n\n### 3.9 测试多种数据增强\n\n| architecture         | epoch | cutout | mixup | autoaugment | random-erase | C10 test acc (%) |\n| -------------------- | ----- | ------ | ----- | ----------- | ------------ | ---------------- |\n| shake_resnet26_2x64d | 200   |        |       |             |              | 96.42            |\n| shake_resnet26_2x64d | 200   | √      |       |             |              | **96.49**        |\n| shake_resnet26_2x64d | 200   |        | √     |             |              | 96.17            |\n| shake_resnet26_2x64d | 200   |        |       | √           |              | 96.25            |\n| shake_resnet26_2x64d | 200   |        |       |             | √            | 96.20            |\n| shake_resnet26_2x64d | 200   | √      | √     |             |              | 95.82            |\n| shake_resnet26_2x64d | 200   | √      |       | √           |              | 96.02            |\n| shake_resnet26_2x64d | 200   | √      |       |             | √            | 96.00            |\n| shake_resnet26_2x64d | 200   |        | √     | √           |              | 95.83            |\n| shake_resnet26_2x64d | 200   |        | √     |             | √            | 95.89            |\n| shake_resnet26_2x64d | 200   |        |       | √           | √            | 96.25            |\n\n```python\npython train.py --model 'shake_resnet26_2x64d' --name 'ss64_orgin' --bs 64\npython train.py --model 'shake_resnet26_2x64d' --name 'ss64_c' --cutout True --bs 64\npython train.py --model 'shake_resnet26_2x64d' --name 'ss64_m' --mixup True --bs 64\npython train.py --model 'shake_resnet26_2x64d' --name 'ss64_a' --autoaugmentation True  --bs 64\npython train.py --model 'shake_resnet26_2x64d' --name 'ss64_r' --random-erase True  --bs 64\npython train.py --model 'shake_resnet26_2x64d' --name 'ss64_cm'  --cutout True --mixup True --bs 64\npython train.py --model 'shake_resnet26_2x64d' --name 'ss64_ca' --cutout True --autoaugmentation True --bs 64\npython train.py --model 'shake_resnet26_2x64d' --name 'ss64_cr' --cutout True --random-erase True --bs 64\npython train.py --model 'shake_resnet26_2x64d' --name 'ss64_ma' --mixup True --autoaugmentation True --bs 64\npython train.py --model 'shake_resnet26_2x64d' --name 'ss64_mr' --mixup True --random-erase True --bs 64\npython train.py --model 'shake_resnet26_2x64d' --name 'ss64_ar' --autoaugmentation True --random-erase True  --bs 64\n```\n\n### 3.10 测试注意力机制\n\n| Model      | Top1:train | Top1:val | weight:M |\n| ---------- | ---------- | -------- | -------- |\n| spp_d11_pN | 100        | 86.79    | 7.36     |\n| spp_d11_pA | 100        | 85.83    | 7.36     |\n| spp_d11_pB | 100        | 85.66    | 7.36     |\n| spp_d11_pC | 100        | 85.56    | 7.36     |\n| spp_d11_pD | 100        | 85.73    | 7.36     |\n| spp_d20_pN | 100        | 90.59    | 13.4     |\n| spp_d20_pA | 100        | 89.96    | 13.4     |\n| spp_d20_pB | 100        | 89.26    | 13.4     |\n| spp_d20_pC | 100        | 89.69    | 13.4     |\n| spp_d20_pD | 100        | 89.93    | 13.4     |\n| spp_d29_pN | 99.99      | 89.56    | 19.4     |\n| spp_d29_pA | 100        | 90.13    | 19.4     |\n| spp_d29_pB | 100        | 90.16    | 19.4     |\n| spp_d29_pC | 100        | 90.09    | 19.4     |\n| spp_d29_pD | 100        | 90.06    | 19.4     |\n\n\n\n## 4. Reference\n\n[1] https://github.com/BIGBALLON/CIFAR-ZOO\n\n[2] https://github.com/pprp/MutableNAS\n\n[3] https://github.com/clovaai/CutMix-PyTorch\n\n[4] https://github.com/4uiiurz1/pytorch-ricap\n\n[5] https://github.com/NUDTNASLab/pytorch-image-models\n\n[6] https://github.com/facebookresearch/LaMCTS\n\n[7] https://github.com/Alibaba-MIIL/ImageNet21K\n\n[8] https://myrtle.ai/learn/how-to-train-your-resnet/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpprp%2Fpytorch-cifar-model-hub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpprp%2Fpytorch-cifar-model-hub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpprp%2Fpytorch-cifar-model-hub/lists"}