{"id":15646280,"url":"https://github.com/ayulockin/losslandscape","last_synced_at":"2025-04-30T12:03:44.092Z","repository":{"id":106646823,"uuid":"279338994","full_name":"ayulockin/LossLandscape","owner":"ayulockin","description":"Explores the ideas presented in Deep Ensembles: A Loss Landscape Perspective (https://arxiv.org/abs/1912.02757) by Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan.","archived":false,"fork":false,"pushed_at":"2020-07-29T02:59:53.000Z","size":2644,"stargazers_count":62,"open_issues_count":0,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-06T03:08:01.683Z","etag":null,"topics":["deep-neural-networks","keras","loss-landscape","neural-networks","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ayulockin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-13T15:19:41.000Z","updated_at":"2024-10-13T03:46:32.000Z","dependencies_parsed_at":"2023-04-26T22:31:22.678Z","dependency_job_id":null,"html_url":"https://github.com/ayulockin/LossLandscape","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ayulockin%2FLossLandscape","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ayulockin%2FLossLandscape/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ayulockin%2FLossLandscape/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ayulockin%2FLossLandscape/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ayulockin","download_url":"https://codeload.github.com/ayulockin/LossLandscape/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233218904,"owners_count":18643296,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-neural-networks","keras","loss-landscape","neural-networks","tensorflow"],"created_at":"2024-10-03T12:12:12.126Z","updated_at":"2025-01-09T16:27:22.555Z","avatar_url":"https://github.com/ayulockin.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LossLandscape\nExplores the ideas presented in [Deep Ensembles: A Loss Landscape Perspective](https://arxiv.org/abs/1912.02757) by Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. \n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"https://i.ibb.co/NrR8KFJ/Untitled.png\"\u003e\u003c/img\u003e\n\u003c/div\u003e\n\nIn the paper, the authors investigate the question - ***why deep ensembles work better than single deep neural networks?*** \n\nIn their investigation the authors figure out:\n\n* Different snapshots (i.e. model from epoch 1, model from epoch 2, and so on) of a same model exhibit functional similarity. Hence, their ensemble is **less likely** to explore the different modes of local minimas in the optimization space. \n* Different solutions of a same model (i.e. trained with different random initializations each time) exhibit functional dissimilarity. Hence, their ensemble is **more likely** to explore the different modes of local minimas in the optimization space. \n\nAlong with these fascinating findings, they present a number of different things that are useful to understand the dynamics of deep neural networks in general. To know more about them check out our report - [Understanding the Effectivity of Ensembles in Deep Learning](https://app.wandb.ai/authors/loss-landscape/reports/Understanding-the-effectivity-of-ensembles-in-deep-learning-(tentative)--VmlldzoxODAxNjA). \n\n## About the notebooks\n- `*_CIFAR10.ipynb`: Shows the training process with three different  architectures (SmallCNN, MediumCNN, and ResNet20v1) as per the paper (with minor modifications).\n- `*_Aug_Val_Acc_Ensembles.ipynb`: Investigates how accuracy can be represented as a function of ensemble size. \n- `Visualizing_Function_Space_Similarity_*.ipynb`: Investigates cosine similarity between weights collected from different snapshots and trajectories, prediction disagreement between different snapshots and trajectories, and presents tSNE visualizations of a how particular solution travels along the optimization landscape (not available for ResNet20v1). \n\n## Model weights\nAvailable [here](https://github.com/ayulockin/LossLandscape/releases/tag/v0.1.0). \n\n## Acknowledgements\n\nThanks to **Yannic Kilcher** for [his amazing explanation video](https://www.youtube.com/watch?v=5IRlUVrEVL8) of the paper which helped us pursue our experiments. \n\nThanks to **Balaji Lakshminarayanan** for providing feedback on the initial draft of the report and also for rectifying our mistake on the tSNE projections. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fayulockin%2Flosslandscape","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fayulockin%2Flosslandscape","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fayulockin%2Flosslandscape/lists"}