{"id":17893809,"url":"https://github.com/shreypandit/knowledge_distillation","last_synced_at":"2025-06-28T15:33:17.586Z","repository":{"id":111359071,"uuid":"368493994","full_name":"ShreyPandit/Knowledge_Distillation","owner":"ShreyPandit","description":"Tried out the code for Knowledge Distillation","archived":false,"fork":false,"pushed_at":"2021-05-19T11:30:53.000Z","size":17601,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-08T18:13:58.085Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ShreyPandit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-18T10:48:15.000Z","updated_at":"2021-05-19T11:30:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"e2c3ed2b-adc3-46f4-9281-bf98a483199f","html_url":"https://github.com/ShreyPandit/Knowledge_Distillation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShreyPandit%2FKnowledge_Distillation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShreyPandit%2FKnowledge_Distillation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShreyPandit%2FKnowledge_Distillation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShreyPandit%2FKnowledge_Distillation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ShreyPandit","download_url":"https://codeload.github.com/ShreyPandit/Knowledge_Distillation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246933384,"owners_count":20857055,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-28T14:56:13.009Z","updated_at":"2025-04-03T04:17:36.103Z","avatar_url":"https://github.com/ShreyPandit.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Knowledge_Distillation\nPlatform used - Tensorflow \u003cbr\u003e\n\nKnowledge distillation is a process of training a smaller model using a larger model usually known as a teacher model.\u003cbr\u003e\nIt has various benifits of Transfer learning as the model has lesser parameters than the original Parent (Teacher) model. \u003cbr\u003e\n\nThere are various different hyperparameter used to train here  - \u003cbr\u003e\n1) Temperature - here the softmax function is modified for creating a type of prediction which not have a hard boundary, In the paper it is discussed that knowing the relation between other images in the dataset about how close the given image is from other classes is helpful. So this type of prediction is created by reducing the exponentiated value by a hyper parameter known as temperature. Its values shouldn't be soo high that the predicted values are very close and no conclusion can be drawn \u003cbr\u003e\n2) Alpha - This is the parameter which decided the weightage that needs to be given to the custom defined Loss function, how much weightage should be given to the loss recieved from the parent loss function and the student loss function.\n\nLink for paper - https://arxiv.org/abs/1503.02531\n\nThe code is reffered from the official Keras documentation\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshreypandit%2Fknowledge_distillation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshreypandit%2Fknowledge_distillation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshreypandit%2Fknowledge_distillation/lists"}