{"id":15653871,"url":"https://github.com/astorfi/attention-guided-sparsity","last_synced_at":"2025-04-30T22:22:24.906Z","repository":{"id":108139568,"uuid":"120543193","full_name":"astorfi/attention-guided-sparsity","owner":"astorfi","description":"Attention-Based Guided Structured Sparsity of Deep Neural Networks","archived":false,"fork":false,"pushed_at":"2020-03-22T21:55:48.000Z","size":977,"stargazers_count":29,"open_issues_count":1,"forks_count":8,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-30T20:33:10.557Z","etag":null,"topics":["attention-mechanism","convolutional-neural-networks","deep-learning","sparsity"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/astorfi.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-07T00:52:32.000Z","updated_at":"2024-07-25T02:06:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"a0ad0bd8-fd0b-49ac-9c30-a2937d519d06","html_url":"https://github.com/astorfi/attention-guided-sparsity","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astorfi%2Fattention-guided-sparsity","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astorfi%2Fattention-guided-sparsity/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astorfi%2Fattention-guided-sparsity/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astorfi%2Fattention-guided-sparsity/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/astorfi","download_url":"https://codeload.github.com/astorfi/attention-guided-sparsity/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251790213,"owners_count":21644182,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-mechanism","convolutional-neural-networks","deep-learning","sparsity"],"created_at":"2024-10-03T12:47:33.173Z","updated_at":"2025-04-30T22:22:24.881Z","avatar_url":"https://github.com/astorfi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"===================================================================================\nAttention-Based Guided Structured Sparsity of Deep Neural Networks\n===================================================================================\n.. image:: https://travis-ci.org/astorfi/attention-guided-sparsity.svg?branch=master\n    :target: https://travis-ci.org/astorfi/attention-guided-sparsity\n.. image:: https://coveralls.io/repos/github/astorfi/attention-guided-sparsity/badge.svg?branch=master\n    :target: https://coveralls.io/github/astorfi/attention-guided-sparsity?branch=master\n.. image:: https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat\n    :target: https://github.com/astorfi/attention-guided-sparsity/pulls\n.. image:: https://badges.frapsoft.com/os/v2/open-source.svg?v=102\n    :target: https://github.com/ellerbrock/open-source-badge/\n.. image:: http://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat\n    :target: https://github.com/astorfi/attention-guided-sparsity/blob/master/LICENSE\n.. image:: http://www.repostatus.org/badges/latest/active.svg\n   :alt: Project Status: Active – The project has reached a stable, usable state and is being actively developed.\n   :target: http://www.repostatus.org/#active\n.. image:: https://zenodo.org/badge/120543193.svg\n   :target: https://zenodo.org/badge/latestdoi/120543193\n\nThis repository contains the code developed by TensorFlow_ for our paper:\n\n\n| `Attention-Based Guided Structured Sparsity of Deep Neural Networks`_,\n| by: `Amirsina Torfi`_ and `Rouzbeh Asghari Shirvani`_\n\n.. _Attention-Based Guided Structured Sparsity of Deep Neural Networks: https://arxiv.org/abs/1901.01939\n.. _TensorFlow: https://www.tensorflow.org/\n.. _Amirsina Torfi: https://astorfi.github.io/\n.. _Rouzbeh Asghari Shirvani: https://www.linkedin.com/in/rozbeh/\n\n#################\nTable of Contents\n#################\n.. contents::\n  :local:\n  :depth: 3\n\n\n-----------------\nGoal and Outcome\n-----------------\n\nNetwork pruning is aimed at imposing sparsity in a neural network architecture\nby increasing the portion of zero-valued weights for reducing its size energy efficiency\nconsideration and increasing evaluation speed. In most of the conducted\nresearch efforts, the sparsity is enforced for network pruning without any attention\nto the internal network characteristics such as unbalanced outputs of the neurons or\nmore specifically the distribution of the weights and outputs of the neurons. That\nmay cause severe accuracy drop due to uncontrolled sparsity. In this work, we\npropose an attention mechanism that simultaneously controls the sparsity intensity\nand supervised network pruning by keeping important information bottlenecks of\nthe network to be active. On CIFAR-10, *the proposed method outperforms the\nbest baseline method by 6% and reduced the accuracy drop by 2.6× at the same\nlevel of sparsity.*\n\n-------------------\nScope of the works\n-------------------\n\nIn this work, we proposed a controller mechanism for network pruning with the goal of (1) model\ncompression for having few active parameters by enforcing group sparsity, (2) preventing the accuracy\ndrop by controlling the sparsity of the network using an additional loss function by forcing a\nportion of the output neurons to stay alive in each layer of the network, and (3) capability of being\nincorporated for any layer type\n\n\n.. |im| image:: _img/varianceloss.gif\n\n|im|\n\n\n-------------\nRequirements\n-------------\n\n~~~~~~~~~~~\nTensorFLow\n~~~~~~~~~~~\n\nThis code is written in Python and requires **TensorFlow** as the framework. For installation on *Ubuntu*, installing\nTensorFlow with *GPU support* can be as follows:\n\n.. code:: shell\n\n    sudo apt-get install python3-pip python3-dev # for Python 3.n\n    pip3 install tensorflow-gpu\n\nPlease refer to `Official TensorFLow installation guideline`_ for further details considering your specific system architecture.\n\n.. _Official TensorFLow installation guideline: https://openreview.net/pdf?id=S1dGIXVUz\n\n--------------------\nCode Implementation\n--------------------\n\n~~~~~~~~\ndataset\n~~~~~~~~\nFor this repository, the experiments are performed on `MNIST dataset`_ which is available online.\n*MNIST* can directly be downloaded using the following script supported by *TensorFLow*:\n\n.. code:: python\n\n    from tensorflow.examples.tutorials.mnist import input_data\n    mnist = input_data.read_data_sets(FLAGS.data_dir, fake_data=FLAGS.fake_data)\n\nFor which the **FLAGS** are predefined by *argument parser*.\n\n.. _MNIST dataset: http://yann.lecun.com/exdb/mnist/\n\n\n~~~~~~~~~~~~\nArchitecture\n~~~~~~~~~~~~\n\nIn the experiment on MNIST dataset, an architecture similar to **LeNet** has been utilized as a baseline for\ninvestigation of our proposed method with no data augmentation. The baseline architecture has been defined as below:\n\n.. code:: python\n\n    def net(x,training_status):\n\n        with tf.name_scope('reshape'):\n            x_image = tf.reshape(x, [-1, 28, 28, 1])\n\n        h_conv1 = nn_conv_layer(x_image, [5, 5, 1, 64], [64], 'conv1', \\\n                                training_status=training_status, act=tf.nn.relu)\n\n        with tf.name_scope('pool1'):\n            h_pool1 = max_pool_2x2(h_conv1)\n\n        h_conv2 = nn_conv_layer(h_pool1, [5, 5, 64, 128], [128], 'conv2',\\\n                                training_status=training_status, act=tf.nn.relu)\n\n        # Second pooling layer.\n        with tf.name_scope('pool2'):\n            h_pool2 = max_pool_2x2(h_conv2)\n\n        h_pool2_flat = tf.reshape(h_pool2, [-1, 7 * 7 * 128])\n\n        h_fc1 = nn_layer(h_pool2_flat, 7 * 7 * 128, 512, 'fc1', \\\n                         training_status=training_status, act=tf.nn.relu)\n        dropped_h_fc1 = tf.nn.dropout(h_fc1, keep_prob)\n\n        h_fc2 = nn_layer(dropped_h_fc1, 512, 256, 'fc2', \\\n                         training_status=training_status, act=tf.nn.relu)\n        dropped_h_fc2 = tf.nn.dropout(h_fc2, keep_prob)\n\n        # Do not apply softmax activation yet, see below.\n        output = nn_layer(dropped_h_fc2, 256, 10, 'softmax', \\\n                          training_status=training_status, act=tf.identity)\n\n        return output, keep_prob\n\n\n----------------------\nTraining / Evaluation\n----------------------\n\n.. \u003chtml\u003e\n.. \u003chead\u003e\n..   \u003clink rel=\"stylesheet\" type=\"text/css\" href=\"demo/asciinema-player.css\" /\u003e\n.. \u003c/head\u003e\n.. \u003cbody\u003e\n..   \u003casciinema-player src=\"demo/162175.json\" cols=\"80\" rows=\"24\"\u003e\u003c/asciinema-player\u003e\n..   ...\n..   \u003cscript src=\"demo/asciinema-player.js\"\u003e\u003c/script\u003e\n.. \u003c/body\u003e\n.. \u003c/html\u003e\n\n~~~~~~~~\nDemo\n~~~~~~~~\n|speakerrecognition|\n\n.. |speakerrecognition| image:: demo/demo_snapshot.png\n    :target: https://asciinema.org/a/162175\n\n~~~~~~~~~~~~\nDescription\n~~~~~~~~~~~~\n\nAt first, clone the repository. Then, cd to the dedicated directory:\n\n.. code:: shell\n\n    cd python\n\nThen, execute the ``main.py``:\n\n.. code:: shell\n\n    python main.py --max_steps=100000\n\nUsing the above script, the code does the following:\n\n  * Automatically download the dataset\n  * Starts training\n  * Does the evaluation while training is running.\n  * Continue training up to 100000 steps.\n\n**NOTE:** *If you are using a virtual environment which contains TensorFLow, make sure to activate it before running the model.*\n\n--------\nResults\n--------\n\nThe below figure depicts a comparison at different levels of sparsity. As it can be observed from the figure, our\nmethod demonstrates its superiority in higher levels of sparsity. We named our proposed method as **Guided** **Structured**\n**Sparsity** (**GSS**).\n\n.. |imcomp| image:: _img/comparison.png\n\n|imcomp|\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastorfi%2Fattention-guided-sparsity","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fastorfi%2Fattention-guided-sparsity","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastorfi%2Fattention-guided-sparsity/lists"}