{"id":13643218,"url":"https://github.com/Sharpiless/Yolov3-MobileNet-Distillation","last_synced_at":"2025-04-20T21:33:22.808Z","repository":{"id":112377666,"uuid":"363033238","full_name":"Sharpiless/Yolov3-MobileNet-Distillation","owner":"Sharpiless","description":"在Yolov3-MobileNet上进行模型蒸馏训练","archived":false,"fork":false,"pushed_at":"2021-04-30T05:15:42.000Z","size":483,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-02T01:19:34.379Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Sharpiless.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-04-30T05:12:45.000Z","updated_at":"2023-05-17T12:46:41.000Z","dependencies_parsed_at":"2023-05-13T23:45:37.945Z","dependency_job_id":null,"html_url":"https://github.com/Sharpiless/Yolov3-MobileNet-Distillation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sharpiless%2FYolov3-MobileNet-Distillation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sharpiless%2FYolov3-MobileNet-Distillation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sharpiless%2FYolov3-MobileNet-Distillation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sharpiless%2FYolov3-MobileNet-Distillation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Sharpiless","download_url":"https://codeload.github.com/Sharpiless/Yolov3-MobileNet-Distillation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223839465,"owners_count":17211947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T01:01:44.014Z","updated_at":"2024-11-09T14:32:00.827Z","avatar_url":"https://github.com/Sharpiless.png","language":"Jupyter Notebook","readme":"# PaddleDetection知识蒸馏\r\n\r\n知识蒸馏主要是让让新模型（通常是一个参数量更少的模型）近似原模型(模型即函数)。注意到，在机器学习中，我们常常假定输入到输出有一个潜在的函数关系，这个函数是未知的：从头学习一个新模型就是从有限的数据中近似一个未知的函数。如果让新模型近似原模型，因为原模型的函数是已知的，我们可以使用很多非训练集内的伪数据来训练新模型。\r\n\r\n\r\n# 分类模型蒸馏：\r\n原来我们需要让新模型的softmax分布与真实标签匹配，现在只需要让新模型与原模型在给定输入下的softmax分布匹配了。但是由于softmax函数是一个约等于arg max的近似，它所能描述的知识（对输出的概率描述）非常有限，一种常用的解决方法是直接让新旧模型匹配logits输出，即使用teacher model的logits输出作为student model的回归目标，并使用L2损失作为loss。\r\n\r\n## one-stage检测模型蒸馏：\r\n\r\n**基本思路**\r\n\r\nOne-stage目标检测任务的训练目标难度更大，因为teacher网络会预测出更多的背景bbox，如果直接用teacher的预测输出作为student学习的soft label会有严重的类别不均衡问题。解决这个问题需要引入新的方法。\r\n\r\n主要是《Object detection at 200 Frames Per Second》这篇文章中提出了针对该问题的解决方案，即针对YOLOv3中分类、回归、objectness三个不同的head适配不同的蒸馏损失函数，并对分类和回归的损失函数用objectness分值进行抑制，以解决前景背景类别不均衡问题。\r\n\r\n并且该文章使用未标注数据作为蒸馏损失，跟检测损失（有标注数据）加权求和作为最终的损失函数。\r\n\r\n![](https://github.com/Sharpiless/Yolov3-MobileNet-Pruning/blob/main/model.png)\r\n\r\n\r\n# 关于作者\r\n\u003e B站：[https://space.bilibili.com/470550823](https://space.bilibili.com/470550823)\r\n\r\n\u003e CSDN：[https://blog.csdn.net/weixin_44936889](https://blog.csdn.net/weixin_44936889)\r\n\r\n\u003e AI Studio：[https://aistudio.baidu.com/aistudio/personalcenter/thirdview/67156](https://aistudio.baidu.com/aistudio/personalcenter/thirdview/67156)\r\n\r\n\u003e Github：[https://github.com/Sharpiless](https://github.com/Sharpiless)\r\n\r\n# 实验环境：\r\n\r\n数据集：VOC2012\r\n\r\nGPU：V100*1\r\n\r\nBatch Size：8\r\n\r\nEpoches：70000\r\n\r\n\r\n```python\r\n!rm -rf PaddleDetection/\r\n```\r\n\r\n\r\n```python\r\n!unzip -oq data/data85000/PaddleDetection-master.zip -d ./\r\n```\r\n\r\n\r\n```python\r\n%cd PaddleDetection-master/\r\n```\r\n\r\n    /home/aistudio/PaddleDetection-master\r\n\r\n\r\n\r\n```python\r\n!pip install -r requirements.txt\r\n```\r\n\r\n\r\n```python\r\n!pip install paddleslim\r\n```\r\n\r\n\r\n```python\r\n!mkdir /home/aistudio/PaddleDetection-master/dataset/voc/VOCdevkit\r\n```\r\n\r\n\r\n```python\r\n!unzip -oq ../data/data39480/VOC2012.zip -d dataset/voc/VOCdevkit/\r\n```\r\n\r\n# 划分数据集\r\n\r\n\r\n```python\r\nimport os\r\nfrom tqdm import tqdm\r\nfrom random import shuffle\r\n\r\nbase = 'dataset/voc/'\r\nimg_base = 'VOCdevkit/VOC2012/JPEGImages/'\r\nxml_base = 'VOCdevkit/VOC2012/Annotations/'\r\n\r\nimages_list = os.listdir(os.path.join(base, img_base))\r\nshuffle(images_list)\r\n\r\nsplit_num = int(0.9 * len(images_list))\r\n\r\nwith open(os.path.join(base, 'trainval.txt'), 'w') as f:\r\n    for im in tqdm(images_list[:split_num]):\r\n        img_id = im[:-4]\r\n        line = '{}{}.jpg {}{}.xml\\n'.format(img_base, img_id, xml_base, img_id)\r\n        f.write(line)\r\n\r\nwith open(os.path.join(base, 'test.txt'), 'w') as f:\r\n    for im in tqdm(images_list[split_num:]):\r\n        img_id = im[:-4]\r\n        line = '{}{}.jpg {}{}.xml\\n'.format(img_base, img_id, xml_base, img_id)\r\n        f.write(line)\r\n```\r\n\r\n    100%|██████████| 15412/15412 [00:00\u003c00:00, 915578.85it/s]\r\n    100%|██████████| 1713/1713 [00:00\u003c00:00, 792766.50it/s]\r\n\r\n\r\n## 正常训练：\r\n正常训练反而是收敛最快的。\r\n\r\n\r\n```python\r\n!python tools/train.py \\\r\n    -c configs/yolov3_mobilenet_v1_voc.yml --eval\r\n```\r\n\r\n    2021-04-30 13:02:48,341 - INFO - iter: 29200, lr: 0.001000, 'loss': '19.766043', eta: 2:01:38, batch_cost: 0.17888 sec, ips: 44.72353 images/sec\r\n\r\n\r\n## L2蒸馏损失训练\r\n实验结果：\r\nBest test box ap: 55.303840240819966, in step: 69999\r\n\r\n\r\n```python\r\n!python slim/distillation/distill.py \\\r\n    -c configs/yolov3_mobilenet_v1_voc.yml \\\r\n    -t configs/yolov3_r34_voc.yml \\\r\n    --teacher_pretrained https://paddlemodels.bj.bcebos.com/object_detection/yolov3_r34_voc.tar\r\n```\r\n\r\n    2021-04-29 15:50:48,397 - INFO - step 68500 lr 0.000010, loss 138.481812, distill_loss 120.378197, teacher_loss 15.744522\r\n    2021-04-29 15:51:09,905 - INFO - step 68600 lr 0.000010, loss 160.502563, distill_loss 146.543015, teacher_loss 10.307454\r\n    2021-04-29 15:51:29,715 - INFO - step 68700 lr 0.000010, loss 107.800804, distill_loss 96.591446, teacher_loss 11.338940\r\n    2021-04-29 15:51:50,172 - INFO - step 68800 lr 0.000010, loss 119.590355, distill_loss 106.403137, teacher_loss 14.288318\r\n    2021-04-29 15:52:10,472 - INFO - step 68900 lr 0.000010, loss 122.257706, distill_loss 97.294144, teacher_loss 19.427813\r\n    2021-04-29 15:52:50,220 - INFO - step 69100 lr 0.000010, loss 144.336548, distill_loss 129.560059, teacher_loss 14.828430\r\n    2021-04-29 15:53:09,471 - INFO - step 69200 lr 0.000010, loss 124.607025, distill_loss 112.543617, teacher_loss 12.816269\r\n    2021-04-29 15:53:29,804 - INFO - step 69300 lr 0.000010, loss 123.180389, distill_loss 106.882095, teacher_loss 13.914233\r\n    2021-04-29 15:53:52,323 - INFO - step 69400 lr 0.000010, loss 129.621185, distill_loss 106.162689, teacher_loss 19.629433\r\n    2021-04-29 15:54:12,425 - INFO - step 69500 lr 0.000010, loss 127.870285, distill_loss 112.545990, teacher_loss 12.473295\r\n    2021-04-29 15:54:32,816 - INFO - step 69600 lr 0.000010, loss 194.085114, distill_loss 181.748535, teacher_loss 12.321056\r\n    2021-04-29 15:54:53,872 - INFO - step 69700 lr 0.000010, loss 214.038483, distill_loss 186.034805, teacher_loss 23.179140\r\n    2021-04-29 15:55:13,472 - INFO - step 69800 lr 0.000010, loss 123.184227, distill_loss 106.268097, teacher_loss 16.398609\r\n    2021-04-29 15:55:33,172 - INFO - step 69900 lr 0.000010, loss 91.018211, distill_loss 76.597687, teacher_loss 10.691896\r\n    2021-04-29 15:55:52,595 - INFO - Save model to output/yolov3_mobilenet_v1_voc/model_final.\r\n    2021-04-29 15:55:56,586 - INFO - Test iter 0\r\n    2021-04-29 15:56:05,482 - INFO - Test iter 100\r\n    2021-04-29 15:56:12,581 - INFO - Test iter 200\r\n    2021-04-29 15:56:13,775 - INFO - Test finish iter 215\r\n    2021-04-29 15:56:13,775 - INFO - Total number of images: 1713, inference time: 99.09352407371594 fps.\r\n    2021-04-29 15:56:13,776 - INFO - Start evaluate...\r\n    2021-04-29 15:56:14,063 - INFO - Accumulating evaluatation results...\r\n    2021-04-29 15:56:14,077 - INFO - mAP(0.50, 11point) = 55.30%\r\n    2021-04-29 15:56:14,077 - INFO - Save model to output/yolov3_mobilenet_v1_voc/best_model.\r\n    2021-04-29 15:56:18,180 - INFO - Best test box ap: 55.303840240819966, in step: 69999\r\n\r\n\r\n\r\n```python\r\n!mkdir L2distill\r\n!mv output/yolov3_mobilenet_v1_voc/model_final.* L2distill\r\n```\r\n\r\n## Fine-gained蒸馏损失训练\r\n\r\n《Object detection at 200 Frames Per Second》\r\n好家伙，这么复杂结果还不如上一个L2损失。\r\n\r\n实验结果：\r\nBest test box ap: 44.40395830547559, in step: 66000\r\n\r\n## 注：\r\n该方法需要两个模型的回归目标、分类目标一一对应，因此需要观察模型输出并手动给出对应pair，因此目前只支持yolo。\r\n\r\n\r\n```python\r\n!python slim/distillation/distill.py \\\r\n    -c configs/yolov3_mobilenet_v1_voc.yml -o use_fine_grained_loss=true\\\r\n    -t configs/yolov3_r34_voc.yml \\\r\n    --teacher_pretrained https://paddlemodels.bj.bcebos.com/object_detection/yolov3_r34_voc.tar \\\r\n    -r output/yolov3_mobilenet_v1_voc/38000\r\n```\r\n\r\n    2021-04-30 11:18:53,827 - INFO - step 68400 lr 0.000010, loss 21.284683, distill_loss 11.928459, teacher_loss 9.073975\r\n    2021-04-30 11:19:14,828 - INFO - step 68500 lr 0.000010, loss 27.378477, distill_loss 11.061569, teacher_loss 15.730307\r\n    2021-04-30 11:19:36,127 - INFO - step 68600 lr 0.000010, loss 19.714718, distill_loss 11.936641, teacher_loss 7.292910\r\n    2021-04-30 11:19:57,737 - INFO - step 68700 lr 0.000010, loss 48.085701, distill_loss 31.413290, teacher_loss 13.044544\r\n    2021-04-30 11:20:16,128 - INFO - step 68800 lr 0.000010, loss 51.778694, distill_loss 17.455339, teacher_loss 24.325071\r\n    2021-04-30 11:20:41,966 - INFO - step 68900 lr 0.000010, loss 42.863945, distill_loss 30.657480, teacher_loss 15.654951\r\n    2021-04-30 11:21:03,053 - INFO - step 69000 lr 0.000010, loss 27.961176, distill_loss 12.550332, teacher_loss 14.125258\r\n    2021-04-30 11:21:22,872 - INFO - step 69100 lr 0.000010, loss 43.735428, distill_loss 19.787128, teacher_loss 16.448784\r\n    2021-04-30 11:21:43,249 - INFO - step 69200 lr 0.000010, loss 28.012207, distill_loss 14.070652, teacher_loss 10.715887\r\n    2021-04-30 11:22:05,127 - INFO - step 69300 lr 0.000010, loss 38.618095, distill_loss 18.164053, teacher_loss 17.802208\r\n    2021-04-30 11:22:25,139 - INFO - step 69400 lr 0.000010, loss 48.910019, distill_loss 23.584068, teacher_loss 18.871468\r\n    2021-04-30 11:22:46,030 - INFO - step 69500 lr 0.000010, loss 29.586151, distill_loss 12.643631, teacher_loss 15.415883\r\n    2021-04-30 11:23:07,627 - INFO - step 69600 lr 0.000010, loss 33.945679, distill_loss 17.252405, teacher_loss 14.374951\r\n    2021-04-30 11:23:28,778 - INFO - step 69700 lr 0.000010, loss 33.225529, distill_loss 12.227975, teacher_loss 16.934584\r\n    2021-04-30 11:23:49,270 - INFO - step 69800 lr 0.000010, loss 30.932293, distill_loss 16.380510, teacher_loss 13.527035\r\n    2021-04-30 11:24:10,253 - INFO - step 69900 lr 0.000010, loss 25.630695, distill_loss 13.440517, teacher_loss 9.713081\r\n    2021-04-30 11:24:32,141 - INFO - Save model to output/yolov3_mobilenet_v1_voc/model_final.\r\n    2021-04-30 11:24:35,957 - INFO - Test iter 0\r\n    2021-04-30 11:24:44,138 - INFO - Test iter 100\r\n    2021-04-30 11:24:52,015 - INFO - Test iter 200\r\n    2021-04-30 11:24:52,835 - INFO - Test finish iter 215\r\n    2021-04-30 11:24:52,836 - INFO - Total number of images: 1713, inference time: 100.76612474202045 fps.\r\n    2021-04-30 11:24:52,837 - INFO - Start evaluate...\r\n    2021-04-30 11:24:53,403 - INFO - Accumulating evaluatation results...\r\n    2021-04-30 11:24:53,466 - INFO - mAP(0.50, 11point) = 42.63%\r\n    2021-04-30 11:24:53,471 - INFO - Best test box ap: 44.40395830547559, in step: 66000\r\n\r\n\r\n\r\n```python\r\n!mkdir FineGaineddistill\r\n!mv output/yolov3_mobilenet_v1_voc/model_final.* FineGaineddistill\r\n```\r\n\r\n\r\n```python\r\n!rm -rf output/\r\n```\r\n\r\n# 我的公众号：\r\n\r\n![](https://github.com/Sharpiless/Yolov3-MobileNet-Pruning/blob/main/code.png)\r\n\r\n","funding_links":[],"categories":["Lighter and Deployment Frameworks"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSharpiless%2FYolov3-MobileNet-Distillation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSharpiless%2FYolov3-MobileNet-Distillation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSharpiless%2FYolov3-MobileNet-Distillation/lists"}