{"id":20377544,"url":"https://github.com/reshalfahsi/knowledge-distillation","last_synced_at":"2026-04-15T07:32:29.945Z","repository":{"id":217061184,"uuid":"742853421","full_name":"reshalfahsi/knowledge-distillation","owner":"reshalfahsi","description":"Knowledge Distillation for Skin Lesion Classification","archived":false,"fork":false,"pushed_at":"2024-08-24T10:47:33.000Z","size":4370,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-18T14:48:52.577Z","etag":null,"topics":["efficientnet","image-processing","knowledge-distillation","medical-image-processing","medmnist","pytorch","pytorch-lightning","skin-lesion-classification","squeezenet"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/reshalfahsi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-13T15:10:54.000Z","updated_at":"2024-09-22T09:38:57.000Z","dependencies_parsed_at":"2024-01-14T10:47:32.418Z","dependency_job_id":"6aa765cf-52cc-4dc9-b041-6a98f2298a26","html_url":"https://github.com/reshalfahsi/knowledge-distillation","commit_stats":null,"previous_names":["reshalfahsi/knowledge-distillation"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/reshalfahsi/knowledge-distillation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reshalfahsi%2Fknowledge-distillation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reshalfahsi%2Fknowledge-distillation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reshalfahsi%2Fknowledge-distillation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reshalfahsi%2Fknowledge-distillation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/reshalfahsi","download_url":"https://codeload.github.com/reshalfahsi/knowledge-distillation/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reshalfahsi%2Fknowledge-distillation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281069648,"owners_count":26438668,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-26T02:00:06.575Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["efficientnet","image-processing","knowledge-distillation","medical-image-processing","medmnist","pytorch","pytorch-lightning","skin-lesion-classification","squeezenet"],"created_at":"2024-11-15T01:45:34.302Z","updated_at":"2025-10-26T07:05:59.199Z","avatar_url":"https://github.com/reshalfahsi.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Knowledge Distillation for Skin Lesion Classification\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://colab.research.google.com/github/reshalfahsi/knowledge-distillation/blob/master/Knowledge_Distillation_for_Skin_Lesion_Classification.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"colab\"\u003e\u003c/a\u003e\n    \u003cbr /\u003e\n\u003c/div\u003e\n\n\nThe goal of knowledge distillation is to improve the performance of the half-witted model, which, most of the time, has fewer parameters, by allowing it to learn from the more competent model or the teacher model. The half-witted model, or the student model, excerpts the knowledge from the teacher model by matching its class distribution to the teacher model's. To make the distributions softer (used in the training process as part of the loss function), we can adjust a temperature _T_ to them (this is done by dividing the logits before softmax by the temperature). This project designates EfficientNet-B0 as the teacher and SqueezeNet v1.1 as the student. These models will be experimented on the DermaMNIST dataset of MedMNIST. We will take a look at the performance of the teacher, the student (without knowledge distillation), and the student (with knowledge distillation) in the result section.\n\n\n## Experiment\n\n\nTo witness the distillation in action, please refer to the notebook at the following [link](https://github.com/reshalfahsi/knowledge-distillation/blob/master/Knowledge_Distillation_for_Skin_Lesion_Classification.ipynb).\n\n\n\n## Result\n\n## Quantitative Result\n\nThe quantitative results are delivered below in the form of a table.\n\nModel | Loss | Accuracy |\n------------ | ------------- | ------------- |\nTeacher |  1.935 | 71.61% |\nStudent | 1.932 | 69.02% |\nDistilled | 1.918 | 73.44% |\n\n\n## Accuracy and Loss Curve\n\n### Teacher\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/reshalfahsi/knowledge-distillation/blob/master/assets/teacher_loss_curve.png\" alt=\"teacher_loss_curve\" \u003e \u003cbr /\u003e The loss curve on the train set and the validation set of the teacher model. \u003c/p\u003e\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/reshalfahsi/knowledge-distillation/blob/master/assets/teacher_acc_curve.png\" alt=\"teacher_acc_curve\" \u003e \u003cbr /\u003e The accuracy curve on the train set and the validation set of the teacher model. \u003c/p\u003e\n\n### Student\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/reshalfahsi/knowledge-distillation/blob/master/assets/student_loss_curve.png\" alt=\"student_loss_curve\" \u003e \u003cbr /\u003e The loss curve on the train set and the validation set of the student model. \u003c/p\u003e\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/reshalfahsi/knowledge-distillation/blob/master/assets/student_acc_curve.png\" alt=\"student_acc_curve\" \u003e \u003cbr /\u003e The accuracy curve on the train set and the validation set of the student model. \u003c/p\u003e\n\n\n### Distilled\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/reshalfahsi/knowledge-distillation/blob/master/assets/distilled_loss_curve.png\" alt=\"distilled_loss_curve\" \u003e \u003cbr /\u003e The loss curve on the train set and the validation set of the distilled model. \u003c/p\u003e\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/reshalfahsi/knowledge-distillation/blob/master/assets/distilled_acc_curve.png\" alt=\"distilled_acc_curve\" \u003e \u003cbr /\u003e The accuracy curve on the train set and the validation set of the distilled model. \u003c/p\u003e\n\n\n### Overall Validation Curve\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/reshalfahsi/knowledge-distillation/blob/master/assets/overall_loss.png\" alt=\"overall_loss\" \u003e \u003cbr /\u003e Comparison of loss curves between the teacher model, the student model, and the distilled model on the validation set. \u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/reshalfahsi/knowledge-distillation/blob/master/assets/overall_acc.png\" alt=\"overall_acc\" \u003e \u003cbr /\u003e Comparison of accuracy curves between the teacher model, the student model, and the distilled model on the validation set. \u003c/p\u003e\n\n\n## Qualitative Result\n\nThe qualitative results of the models on the test set are exhibited in the collated form below.\n\n\n### Teacher\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/reshalfahsi/knowledge-distillation/blob/master/assets/teacher_qualitative.png\" alt=\"teacher_qualitative\" \u003e \u003cbr /\u003e The qualitative result of the teacher model. \u003c/p\u003e\n\n\n### Student\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/reshalfahsi/knowledge-distillation/blob/master/assets/student_qualitative.png\" alt=\"student_qualitative\" \u003e \u003cbr /\u003e The qualitative result of the student model. \u003c/p\u003e\n\n\n### Distilled\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/reshalfahsi/knowledge-distillation/blob/master/assets/distilled_qualitative.png\" alt=\"distilled_qualitative\" \u003e \u003cbr /\u003e The qualitative result of the distilled model. \u003c/p\u003e\n\n\n## Credit\n\n- [Knowledge Distillation](https://keras.io/examples/vision/knowledge_distillation/)\n- [Distilling the Knowledge in a Neural Network](https://arxiv.org/pdf/1503.02531.pdf)\n- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://arxiv.org/abs/1905.11946)\n- [SqueezeNet v1.1](https://github.com/forresti/SqueezeNet/tree/master/SqueezeNet_v1.1)\n- [MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://medmnist.com/)\n- [PyTorch Lightning](https://lightning.ai/docs/pytorch/latest/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freshalfahsi%2Fknowledge-distillation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freshalfahsi%2Fknowledge-distillation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freshalfahsi%2Fknowledge-distillation/lists"}