{"id":15646264,"url":"https://github.com/sseung0703/zero-shot_knowledge_distillation","last_synced_at":"2025-04-30T12:07:26.879Z","repository":{"id":119339231,"uuid":"192476646","full_name":"sseung0703/Zero-shot_Knowledge_Distillation","owner":"sseung0703","description":"Zero-Shot Knowledge Distillation in Deep Networks in ICML2019","archived":false,"fork":false,"pushed_at":"2019-06-20T15:17:46.000Z","size":1572,"stargazers_count":48,"open_issues_count":0,"forks_count":9,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-25T07:41:48.124Z","etag":null,"topics":["knowledge-distillation","tensorflow","zero-shot-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sseung0703.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-06-18T06:10:38.000Z","updated_at":"2025-01-17T13:13:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"a607f47d-5d75-43f7-a737-6e4496d12f98","html_url":"https://github.com/sseung0703/Zero-shot_Knowledge_Distillation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sseung0703%2FZero-shot_Knowledge_Distillation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sseung0703%2FZero-shot_Knowledge_Distillation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sseung0703%2FZero-shot_Knowledge_Distillation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sseung0703%2FZero-shot_Knowledge_Distillation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sseung0703","download_url":"https://codeload.github.com/sseung0703/Zero-shot_Knowledge_Distillation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242650895,"owners_count":20163610,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["knowledge-distillation","tensorflow","zero-shot-learning"],"created_at":"2024-10-03T12:12:07.997Z","updated_at":"2025-03-09T05:30:27.644Z","avatar_url":"https://github.com/sseung0703.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Zero-shot Knowledge Distillation\r\nThis is the code for [Zero-shot Knowledge Distillation](https://arxiv.org/abs/1905.08114) which is accepted on ICML2019 oral paper.\r\nNote that it is not exactly the same as the author's algorithm due to a lack of detail in the paper.\r\nSo I had to estimate some of the hyper-parameters and implemented detail.\r\nI got better results than the paper's in case of no augmentation. But I fail to increase performance by augmentation.\r\n\r\n## Abstract\r\nZSKD is the knowledge distillation algorithm which not required real data. So they generate \"Data Impression\" samples which contain the teacher network's knowledge and train student network by it. It means that we don't have to care about privacy or safety. So the pros and cons I think are as follows.\r\n\r\n- Pros\r\n  - Simple but powerful algorithm.\r\n  - Data-free training algorithm\r\n\r\n- Cons\r\n  - Lack of experiments results of a large dataset such as ImageNet\r\n  - Only focused on a classification task\r\n  - The experiment set is not fair. they use augmentation for ZSKD only. So I think all of the rest performance can be increased.\r\n\r\n\r\n\r\n## Requirements\r\nThis code requires\r\n\r\n- Tensorflow  1.13\r\n- Python 3.5\r\n- Opencv 3.2.0\r\n- Scipy 1.1.0\r\n- Numpy 1.15.0\r\n- Matplotlib 3.0.0\r\n\r\neach version is not important except Tensorflow, I think.\r\n\r\n## How to run\r\n\r\nZero-shot knowledge distillation's procedure is composed with \r\n- Training teacher network, \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026rarr; train_w_distill.py\r\n- Generate data impression samples, \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026rarr; Data_Impressions.py\r\n- Training student network by samples\". \u0026rarr; train_w_distill.py\r\n\r\nBut if you want to follow the author's configuration, just run the \r\n\"autotrain.py\".\r\n\r\nWhen you train teacher network and generate some samples, you can visualize \"Concentration matrix\" and generated samples by \r\n\"visualization.py\"\r\nbelow images and tables are my experiment results.\r\n\r\n## Experiment results\r\nI tested all the things by running the autotrain. And all of the numerical value is mean of 5 results.\r\n\r\n|Rate|Teacher|Student|Soft-logits|ZSKD |\r\n| --:|  :-:  |  :-:  |    :-:    | :-: |\r\n| 100| 99.06 | 98.52 |     -     |  -  |\r\n|  40|   -   |   -   |   89.78   |97.92|\r\n|  25|   -   |   -   |   91.64   |97.35|\r\n|  10|   -   |   -   |   93.08   |96.69|\r\n|   5|   -   |   -   |   88.43   |91.31|\r\n|   1|   -   |   -   |   70.02   |92.15|\r\n\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg src=\"concentrate.png\" width=\"400\"\u003e\u003cbr\u003e\r\n  \u003cb\u003eConcentrate matrix of teacher network\u003c/b\u003e\r\n\u003c/p\u003e\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg src=\"samples.png\" width=\"600\"\u003e\u003cbr\u003e\r\n  \u003cb\u003eExpamples of the generated smaples\u003c/b\u003e  \r\n\u003c/p\u003e\r\n \r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsseung0703%2Fzero-shot_knowledge_distillation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsseung0703%2Fzero-shot_knowledge_distillation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsseung0703%2Fzero-shot_knowledge_distillation/lists"}