{"id":16858911,"url":"https://github.com/titu1994/batchrenormalization","last_synced_at":"2025-03-17T05:32:32.610Z","repository":{"id":150946667,"uuid":"82606313","full_name":"titu1994/BatchRenormalization","owner":"titu1994","description":"Batch Renormalization algorithm implementation in Keras","archived":false,"fork":false,"pushed_at":"2019-02-03T16:23:38.000Z","size":40417,"stargazers_count":98,"open_issues_count":3,"forks_count":27,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-02-27T18:27:11.338Z","etag":null,"topics":["batch-renormalization","deep-learning","keras","keras-layer"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/titu1994.png","metadata":{"files":{"readme":"README.md","changelog":"history/batch_renorm_history.txt","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-20T21:51:04.000Z","updated_at":"2024-10-21T06:13:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"f6b50b35-f80a-40c6-8c8f-e61319fd158b","html_url":"https://github.com/titu1994/BatchRenormalization","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/titu1994%2FBatchRenormalization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/titu1994%2FBatchRenormalization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/titu1994%2FBatchRenormalization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/titu1994%2FBatchRenormalization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/titu1994","download_url":"https://codeload.github.com/titu1994/BatchRenormalization/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243846984,"owners_count":20357297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch-renormalization","deep-learning","keras","keras-layer"],"created_at":"2024-10-13T14:15:33.353Z","updated_at":"2025-03-17T05:32:27.596Z","avatar_url":"https://github.com/titu1994.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Batch Renormalization\nBatch Renormalization algorithm implementation in Keras 2.0+. Original paper by Sergey Ioffe, [Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models](https://arxiv.org/pdf/1702.03275.pdf).\n\n# NOTE:\n\nThis implementation of BatchRenormalization is inconsistent with the original paper and therefore results may not be similar !\n\nFor discussion on the inconsistency of this implementation, refer here : https://github.com/keras-team/keras-contrib/issues/17\n\n# Usage\nAdd the `batch_renorm.py` script into your repository, and import the BatchRenormalization layer.\n\nEg. You can replace Keras BatchNormalization layers with BatchRenormalization layers. \n```\nfrom batch_renorm import BatchRenormalization\n```\n\n# Performance\nUsing BatchRenormalization layers requires slightly more time than the simpler BatchNormalization layer. \n\nObserved speed differences in WRN-16-4 with respect to BatchNormalization on a 980M GPU:\n\n1) **Batch Normalization** : 137 seconds per epoch.\n\n2) **Batch Renormalization (Mode 0)** : 152 seconds per epoch.\n\n3) **Batch Renormalization (Mode 2)** : 142 seconds per epoch.\n\n# Results\nThe following graph is from training a Wide Residual Network (WRN-16-4) on the CIFAR 10 dataset, with no data augmentation and no dropout. Therefore all models clearly overfit. \n\nHowever, the graphs compare WRN-16-4 model with Keras BatchNormalization (mode 0) with BatchRenormalization (mode 0 and mode 2). All other parameters are kept constant.\n\n![Training curve](https://github.com/titu1994/BatchRenormalization/blob/master/plots/batchnorm_vs_renorm.png?raw=true)\n\n# Parameters\nThere are several parameters that are present in addition to the parameters in BatchNormalization layers.\n\n```\nr_max_value: The clipped maximum value that the internal parameter 'r' can take. The value of r will be clipped in the range\n             (1 / r_max_value, r_max_value) after a sufficient number of iterations. \n             The paper suggests a default value of 3.\n             \nd_max_value: The clipped maximum value that the internal parameter 'd' can take. The value of d will be clipped in the range\n             (-d_max_value, d_max_value) after a sufficient number of iterations. \n             The paper suggests a default value of 5.\n             \nt_delta:     This parameter determines in how many iterations the internal r_max and d_max values will become equal to \n             r_max_value and d_max_value. \n             \n             Default setting is 1, which means that in 5 iterations the internal parameters \n             will become their maximum value.\n             \n             Values larger than 1 can cause gradient explosion, and prevent learning of anything useful.\n             \n             Using very small values will lead to slower learning, but eventually will lead to the same result as using \n             t_delta = 1. \n             \n             Sugggested t_delta values = 1 to 1e-3.\n```\n\n# Requirements\nKeras 1.2.1 (will be updated when Keras 2 launches)\n\nTheano / Tensorflow\n\nh5py\n\nseaborn (optional, for plotting training graph)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftitu1994%2Fbatchrenormalization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftitu1994%2Fbatchrenormalization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftitu1994%2Fbatchrenormalization/lists"}