{"id":21857745,"url":"https://github.com/amitkumarj441/nn_prune","last_synced_at":"2026-05-19T15:06:44.527Z","repository":{"id":70230003,"uuid":"171044498","full_name":"amitkumarj441/nn_prune","owner":"amitkumarj441","description":"Weight/Unit pruning in Tensorflow","archived":false,"fork":false,"pushed_at":"2019-02-16T20:31:36.000Z","size":316,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-28T02:35:00.705Z","etag":null,"topics":["mnist-dataset","neural-networks","pruning","tensorflow"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amitkumarj441.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-16T19:44:38.000Z","updated_at":"2019-02-16T20:31:37.000Z","dependencies_parsed_at":"2023-05-11T01:45:31.573Z","dependency_job_id":null,"html_url":"https://github.com/amitkumarj441/nn_prune","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitkumarj441%2Fnn_prune","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitkumarj441%2Fnn_prune/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitkumarj441%2Fnn_prune/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitkumarj441%2Fnn_prune/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amitkumarj441","download_url":"https://codeload.github.com/amitkumarj441/nn_prune/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235712020,"owners_count":19033575,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mnist-dataset","neural-networks","pruning","tensorflow"],"created_at":"2024-11-28T02:31:24.355Z","updated_at":"2026-05-19T15:06:39.443Z","avatar_url":"https://github.com/amitkumarj441.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pruning in neural network\nWeight/Unit pruning in Tensorflow\n\nGiven a layer of a neural network ReLU(xW) are two well-known ways to prune it:\n\n  - Weight pruning: Set individual weights in the weight matrix to zero. This corresponds to deleting connections as in the figure above. Here, to achieve sparsity of k% I rank the individual weights in weight matrix W according to their magnitude (absolute value) , and then set to zero the smallest k%.\n\n  - Unit/Neuron pruning: Set entire columns to zero in the weight matrix to zero, in effect deleting the corresponding output neuron. Here to achieve sparsity of k% we rank the the columns of a weight matrix according to their L2-norm and delete the smallest k%.\n  \n###### References\n[EXPLOITING SPARSENESS IN DEEP NEURAL NETWORKS FOR LARGE VOCABULARY SPEECH RECOGNITION](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6288897\u0026tag=1)\n\n## Conclusion\nAfter implementing the weight pruning and neuron pruning methods, I noticed that to have good performance after pruning the final distributions of the weights plays a crucial role. The overall dependence of final distribution is on the learning rate. The initial learning rate used was 1e-5. In this case, the final accuracy, after training over 50 epochs was ~98%. But, I can distinguish that the distributions of the weights had a flat top. In this case, the test accuracy dropped very early (smaller values of k) in the case of weight as well as unit pruning. The learning rate was then increased to 0.001 and the final accuracy after training was ~98%. In this case it was noticed that the distributions of the weight matrices were proper gaussians. In this case the the accuracies didn't drop for early values of k (accuracies did not degrade rapidly with increasing k) and the performance was much better.\n\nIt was also noticed that the test accuracy initially increases a little bit in weight pruning. This shows that the network overfits the training set, which depicts that pruning has a regularizing effect and it decreases the generalization error. Though fundamentally different, L1 regularization also leads to a sparse solution. I believe the main intuition behind pruning is that weights with small magnitudes do not have much effect on the outputs. The weight distribution above also shows that the majority of the weights have extremely small values hence zeroing them does not\nhave much effect. The above referenced paper shows that ~70% of the weights of the neural network are small and can be removed.\n\nAs per the experiemnt, I see unit weight pruning performance much better than unit pruning, in unit pruning we sets the entire columns to zero, which results in zeroing out of the output neuron. We are necessarily removing conclusive features which are necessary for classification. First, we remove those features whose performance does not decrease i.e., features which are not useful. As k increases, we keep on removing useful features and performance degrades more quickly. As we see that the cloumn norms have lesser values with lower magnitudes hence its performance degrades faster. Apart from this, we believe that the network learns redundant features, thus removing some features (or weights) won't have any effect. Although, since we are removing features based on magnitude, this may be one of those things that fairly contributed to achieve the observed result.\n\nHere we see that the dense evaluation is slower than sparse evaluation for both weight and unit/neuron pruning. As we see that increasing the sparsity decreases evaluation time of sparse evaluation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famitkumarj441%2Fnn_prune","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famitkumarj441%2Fnn_prune","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famitkumarj441%2Fnn_prune/lists"}