{"id":16976129,"url":"https://github.com/eigenfoo/batch-renorm","last_synced_at":"2025-04-12T01:33:09.250Z","repository":{"id":53790332,"uuid":"152452673","full_name":"eigenfoo/batch-renorm","owner":"eigenfoo","description":"A Tensorflow re-implementation of batch renormalization, first introduced by Sergey Ioffe.","archived":false,"fork":false,"pushed_at":"2021-03-15T22:01:45.000Z","size":770,"stargazers_count":13,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-25T21:14:47.960Z","etag":null,"topics":["batch-norm","batch-normalization","batch-renorm","batch-renormalization","deep-learning","sergey-ioffe","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eigenfoo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-10-10T16:10:54.000Z","updated_at":"2023-11-13T22:59:37.000Z","dependencies_parsed_at":"2022-09-24T21:40:15.241Z","dependency_job_id":null,"html_url":"https://github.com/eigenfoo/batch-renorm","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eigenfoo%2Fbatch-renorm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eigenfoo%2Fbatch-renorm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eigenfoo%2Fbatch-renorm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eigenfoo%2Fbatch-renorm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eigenfoo","download_url":"https://codeload.github.com/eigenfoo/batch-renorm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248504368,"owners_count":21115155,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch-norm","batch-normalization","batch-renorm","batch-renormalization","deep-learning","sergey-ioffe","tensorflow"],"created_at":"2024-10-14T01:25:01.662Z","updated_at":"2025-04-12T01:33:09.223Z","avatar_url":"https://github.com/eigenfoo.png","language":"Python","readme":"# Batch Renormalization\n\nA Tensorflow implementation of batch renormalization, first introduced by Sergey\nIoffe.\n\n**Paper:**\n    Batch Renormalization: Towards Reducing Minibatch Dependence in\n        Batch-Normalized Models, Sergey Ioffe\n    https://arxiv.org/abs/1702.03275\n\n**GitHub repository:**\n    https://github.com/eigenfoo/batch-renorm\n\nThe goal of this project is to reproduce the following figure from the paper:\n\n\u003cfigure\u003e\n    \u003ca href=\"https://raw.githubusercontent.com/eigenfoo/batch-renorm/master/docs/paper-figure.png\"\u003e\n    \u003cimg style=\"float: middle\" src=\"https://raw.githubusercontent.com/eigenfoo/batch-renorm/master/docs/paper-figure.png\"\u003e\n    \u003c/a\u003e\n\u003c/figure\u003e\n\nBelow is our reproduction:\n\n\u003cfigure\u003e\n    \u003ca href=\"https://raw.githubusercontent.com/eigenfoo/batch-renorm/master/results/figure.png\"\u003e\n    \u003cimg style=\"float: middle\" src=\"https://raw.githubusercontent.com/eigenfoo/batch-renorm/master/results/figure.png\"\u003e\n    \u003c/a\u003e\n\u003c/figure\u003e\n\n## Description\n\nThere were a few things that we did differently from the paper:\n\n- We used the CIFAR-100 dataset, instead of the ImageNet dataset.\n- We used a plain convolutional network, instead of the Inception-v3\n  architecture.\n- We used the Adam optimizer, instead of the RMSProp optimizer.\n- We split minibatches into 800 microbatches of 2 examples each, instead of 400\n  microbatches of 4 examples each. Note that each minibatch still consists of\n  1600 examples.\n- We trained for a mere 8k training updates, instead of 160k training updates.\n- We ran the training 5 separate times, and averaged the learning curves from\n  all runs. This was not explicitly stated in the paper.\n\nThe reproduced results do not exactly mirror the paper's results: for instance,\nthe learning curves for batch norm and batch renorm do not converge to the same\nvalue, and the learning curve for batch norm even appears to be curving down\ntowards the end of training.\n\nWe suspect that these discrepancies are due to two factors:\n\n1. Not training for long enough time (8k training steps is nothing compared to\n   160k), and\n2. Using a different architecture/dataset to reproduce the same results. While\n   the behavior should still be the same, it may be the case that certain\n   hyperparameters are ill-chosen.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feigenfoo%2Fbatch-renorm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feigenfoo%2Fbatch-renorm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feigenfoo%2Fbatch-renorm/lists"}