{"id":16739613,"url":"https://github.com/abhilash1910/miniattention","last_synced_at":"2025-10-30T05:45:42.491Z","repository":{"id":57441749,"uuid":"285554122","full_name":"abhilash1910/MiniAttention","owner":"abhilash1910","description":"Hierarchical Attention Layer for Keras for document classification.","archived":false,"fork":false,"pushed_at":"2023-02-02T14:15:47.000Z","size":113,"stargazers_count":7,"open_issues_count":1,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-04-26T14:41:42.960Z","etag":null,"topics":["attention","attention-mechanism","bilstm","hierarchicalattention","keras-layers","keras-tensorflow","lstm-neural-networks","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abhilash1910.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.TXT","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-06T11:33:20.000Z","updated_at":"2024-03-13T14:10:09.000Z","dependencies_parsed_at":"2023-02-17T20:46:06.493Z","dependency_job_id":null,"html_url":"https://github.com/abhilash1910/MiniAttention","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhilash1910%2FMiniAttention","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhilash1910%2FMiniAttention/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhilash1910%2FMiniAttention/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhilash1910%2FMiniAttention/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abhilash1910","download_url":"https://codeload.github.com/abhilash1910/MiniAttention/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248225652,"owners_count":21068078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","attention-mechanism","bilstm","hierarchicalattention","keras-layers","keras-tensorflow","lstm-neural-networks","tensorflow"],"created_at":"2024-10-13T00:52:26.728Z","updated_at":"2025-10-30T05:45:42.413Z","avatar_url":"https://github.com/abhilash1910.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mini-Attention\n\n## A Keras Hirarchical Attention Layer for Document Classification in NLP :robot:\n\nThis library is an implementation of Heirarchical Attention Networks for Document Classification (Yang etal,2015).[Link](https://www.cs.cmu.edu/~./hovy/papers/16HLT-hierarchical-attention-networks.pdf0). This is compatible with Keras and Tensorflow (keras version \u003e=2.0.6). As the paper suggests, it uses hierarchical attention mechanism and capabilities of Word Encoder (including bi-directional Recurrent unit- GRU) ,Sentence Attention and Document Classification are addressed.\n\n## Dependencies\n\n\u003ca href=\"https://www.tensorflow.org/\"\u003eTensorflow\u003c/a\u003e\n\n\n\u003ca href=\"https://keras.io/\"\u003eKeras\u003c/a\u003e\n\n\n## Usability\n\nThe library or the Layer is compatible with Tensorflow and Keras. Installation is carried out using the pip command as follows:\n\n```python\npip install MiniAttention==0.1\n```\n\nFor using inside the Jupyter Notebook or Python IDE (along with Keras layers):\n\n```python\nimport MiniAttention.MiniAttention as MA\n```\n\nThe Layer takes as input a 3D Tensor with dimensions: (sample_size,steps,features)\nThe Layer as an output produces a 2D Tensor with dimensions: (sample_size,features)\nThis Layer can be used after the Keras.Embedding() Layer to provide a global attention using the features and Embedding weights.Additionally Embedding libraries like (Glove,Word2Vec,ELMO) can be used in the Embedding Layer. It can also be used in between and before LSTM/Bidirectional LSTM/ GRU and recurrent layers . In this context, it has a capability to cater to Sequential and Model (functional) types of Model architectures(specific for Keras). The functional (keras.models.Model) version is provided as follows:\n\n```python\n\ninp=Input(shape=(inp_shape,))\nz=Embedding(max_features,256)(inp)\nz=MA.MiniAttentionBlock(keras.initializers.he_uniform,None,None,keras.regularizers.L2(l2=0.02),None,None,None,None,None)(z)\nz=tf.keras.layers.Bidirectional(LSTM(128,recurrent_activation=\"relu\",return_sequences=True))(z)\nz=tf.keras.layers.Bidirectional(LSTM(64,recurrent_activation=\"relu\",return_sequences=True))(z)\nz=MA.MiniAttentionBlock(keras.initializers.he_uniform,None,None,keras.regularizers.L2(l2=0.02),None,None,None,None,None)(z)\nz=keras.layers.Dense(64,activation=\"relu\")(z)\nz=keras.layers.Dense(64,activation=\"relu\")(z)\nz=keras.layers.Dense(1,activation=\"sigmoid\")(z)\nmodel=keras.models.Model(inputs=inp,outputs=z)\nmodel.compile(loss=\"binary_crossentropy\",metrics=['accuracy'],optimizer=keras.optimizers.Adagrad(learning_rate=1e-3))\nmodel.summary()\n```\n\nFor Sequential Model (keras.models.Sequential):\n\n```python\nmodel = Sequential()\nmodel.add(Embedding(max_features,128,input_shape=(100,)))\nmodel.add(MA.MiniAttentionBlock(None,None,None,None,None,None,None,None,None))\nmodel.add(LSTM(128))\nmodel.add(Dense(8,activation='relu'))\nmodel.add(Dense(4,activation='sigmoid'))\nmodel.compile(loss='binary_crossentropy',metrics=['accuracy'],optimizer='Adagrad')\nmodel.summary()\n```\n\nThe arguments for the MiniAttentionBlock class include:\n\n```\n1.W_init: Weight Initializer - Compatible with keras.initializers\n2.b_init: Bias Initializer - Compatible with keras.initializers\n3.u_init: Output Initializer -Compatible with keras.initializers\n4.W_reg: Weight Regularizer - Compatible with keras.regularizers\n5.b_reg: Bias Regularizer - Compatible with keras.regularizers\n6.u_reg: Output Regularizer -Compatible with keras.regularizers\n7.W_const: Weight Constraint - Compatible with keras.constraints\n8.b_const: Bias Constraint - Compatible with keras.constraints\n9.u_const: Output Constraint -Compatible with keras.constraints\n10.bias: Boolean - True/False- Whether to use bias (Optional)\n```\n\nThere are 3 main functions inside the MiniAttentionBlock class. The \"**init**\" method is used for initializeing the weight , bias tensors for computation. The \"attention_block\" method is used for assigning the variables (tensors) and checking for the input tensor size. The \"build_nomask\" method is used for computing the attention modules.Uses tanh as the internal activation function with exponential normalization.Masking has not been added to the library yet.\n\n## Example\n\nFor reference on how to use the library, a Jupyter Notebook sample is present in the repository: \"MiniAttention_on_IMDB.ipynb\".\n\nThis is a sample which uses this Layer with Keras.layers.Embedding() Layer in IMDB binary classification.It uses the default keras embedding which is followed from the official tutorial by [Keras docs](https://keras.io/examples/nlp/bidirectional_lstm_imdb). Alternately,\"Tensorboard-tfds-IMDB.py\" contains a tensorboard demonstration.\n\n\n## Contributing\n\nPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhilash1910%2Fminiattention","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabhilash1910%2Fminiattention","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhilash1910%2Fminiattention/lists"}