{"id":17139737,"url":"https://github.com/hwalsuklee/tensorflow-mnist-mlp-batch_normalization-weight_initializers","last_synced_at":"2025-04-13T10:03:20.208Z","repository":{"id":52955837,"uuid":"77834728","full_name":"hwalsuklee/tensorflow-mnist-MLP-batch_normalization-weight_initializers","owner":"hwalsuklee","description":"MNIST classification using Multi-Layer Perceptron (MLP) with 2 hidden layers. Some weight-initializers and batch-normalization are implemented.","archived":false,"fork":false,"pushed_at":"2017-01-20T11:18:35.000Z","size":1701,"stargazers_count":55,"open_issues_count":0,"forks_count":17,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-27T01:23:00.262Z","etag":null,"topics":["batch-normalization","he","he-initializer","mnist","mnist-classification","tensorflow","weight-initializers","xavier","xavier-initializer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hwalsuklee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-01-02T13:18:47.000Z","updated_at":"2025-03-14T06:23:50.000Z","dependencies_parsed_at":"2022-08-29T19:12:15.224Z","dependency_job_id":null,"html_url":"https://github.com/hwalsuklee/tensorflow-mnist-MLP-batch_normalization-weight_initializers","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hwalsuklee%2Ftensorflow-mnist-MLP-batch_normalization-weight_initializers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hwalsuklee%2Ftensorflow-mnist-MLP-batch_normalization-weight_initializers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hwalsuklee%2Ftensorflow-mnist-MLP-batch_normalization-weight_initializers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hwalsuklee%2Ftensorflow-mnist-MLP-batch_normalization-weight_initializers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hwalsuklee","download_url":"https://codeload.github.com/hwalsuklee/tensorflow-mnist-MLP-batch_normalization-weight_initializers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248695332,"owners_count":21146954,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch-normalization","he","he-initializer","mnist","mnist-classification","tensorflow","weight-initializers","xavier","xavier-initializer"],"created_at":"2024-10-14T20:12:56.029Z","updated_at":"2025-04-13T10:03:20.178Z","avatar_url":"https://github.com/hwalsuklee.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Various initializers and batch normalization\n\nAn implementation of weight/bias initializers and batch normalization in Tensorflow.\n\nMNIST database is used to show performance-comparison\n\n## Network architecture\n\nIn order to examine the effect of initializers and batch normalization, a simple network architecture called multilayer perceptrons (MLP) is employed.\n\nMLP has following architecture.\n\n+ input layer : 784 nodes (MNIST images size)\n+ first hidden layer : 256 nodes\n+ second hidden layer : 256 nodes\n+ output layer : 10 nodes (number of class for MNIST)\n\n## Various initializers\n\nThe following initializers for weights/biases of network are considered.\n\n+ Weight Initializer\n  * normal, trucated normal\n  * xaiver : [Understanding the difficulty of training deep feedforward neural networks](http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf)\n  * he : [Delving Deep into Rectifiers](http://arxiv.org/pdf/1502.01852v1.pdf)\n+ Bias Initializer\n  * normal, constant\n  \n### Simulation results\n+ Weight Initializer : he \u003e trauncated normal = xaiver \u003e normal\n+ Bias Initilaizer : zero \u003e normal\n\nSample results are following.\n\n![weight_init](fig/results_weight_initializers.jpg)\n\n|Index|Weight Initializer|Bias Initializer|Accuracy|\n|:---:|:---:|:---:|:---:|\n|normal_w_normal_b_0.9451|normal|normal|0.9451|\n|normal_w_zero_b_0.9485|normal|zero|0.9485|\n|truncated_normal_w_normal_b_0.9788|truncated_normal|normal|0.9788|\n|truncated_normal_w_zero_b_0.9790|truncated_normal|zero|0.9790|\n|xavier_w_normal_b_0.9800|xavier|normal|0.9800|\n|xavier_w_zero_b_0.9806|xavier|zero|0.9806|\n|he_w_normal_b_0.9798|he|normal|0.9798|\n|he_w_zero_b_0.9811|he|zero|0.9811|\n\n## Batch normalization\nBatch normalization improves performance of network in terms of final accuracy and convergence rate.\u003c/br\u003e\nIn this simulation, bias initalizer was zero-constant initializer and weight initializers were xavier / he.\n\nSample results are following.\n\n![batch_norm](fig/results_batch_normalization.jpg)\n\n|Index|Weight Initializer|Batch Normalization|Accuracy|\n|:---:|:---:|:---:|:---:|\n|xavier_woBN_0.9806|xavier|Unused|0.9806|\n|xavier_withBN_0.9812|xavier|Used|0.9812|\n|he_woBN_0.9811|he|Unused|0.9811|\n|he_withBN_0.9837|he|Used|0.9837|\n\n## Usage\n\n`python run_main.py --weight-init \u003cweight initializer\u003e --bias-init \u003cbias initializer\u003e --batch-norm \u003cTrue or False\u003e`\n\n`\u003cweight initializer\u003e` must be selected in [normal, truncated_normal, xavier, he].\u003c/br\u003e\n`\u003cbias initializer\u003e` must be selected in [normal, zero].\n\nYou may command like `python run_main.py --weight-init xavier --bias-init zero --batch-norm True`.\n\n## Acknowledgement\n\nThis implementation has been tested on Tensorflow r0.12.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhwalsuklee%2Ftensorflow-mnist-mlp-batch_normalization-weight_initializers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhwalsuklee%2Ftensorflow-mnist-mlp-batch_normalization-weight_initializers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhwalsuklee%2Ftensorflow-mnist-mlp-batch_normalization-weight_initializers/lists"}