{"id":13790942,"url":"https://github.com/batzner/indrnn","last_synced_at":"2025-04-05T14:06:47.045Z","repository":{"id":54350785,"uuid":"125218947","full_name":"batzner/indrnn","owner":"batzner","description":"TensorFlow implementation of Independently Recurrent Neural Networks","archived":false,"fork":false,"pushed_at":"2021-08-19T20:04:03.000Z","size":609,"stargazers_count":514,"open_issues_count":0,"forks_count":128,"subscribers_count":37,"default_branch":"master","last_synced_at":"2025-03-29T13:06:55.130Z","etag":null,"topics":["indrnn","paper-implementations","rnn","tensorflow"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1803.04831","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/batzner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-14T13:44:17.000Z","updated_at":"2025-01-21T03:50:17.000Z","dependencies_parsed_at":"2022-08-13T13:00:40.724Z","dependency_job_id":null,"html_url":"https://github.com/batzner/indrnn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batzner%2Findrnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batzner%2Findrnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batzner%2Findrnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batzner%2Findrnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/batzner","download_url":"https://codeload.github.com/batzner/indrnn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247345852,"owners_count":20924102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["indrnn","paper-implementations","rnn","tensorflow"],"created_at":"2024-08-03T22:00:53.231Z","updated_at":"2025-04-05T14:06:47.014Z","avatar_url":"https://github.com/batzner.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":[],"readme":"# Independently Recurrent Neural Networks\n\nSimple TensorFlow implementation of [Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN](https://arxiv.org/pdf/1803.04831.pdf) by Shuai Li et al. The author's original implementation in Theano and Lasagne can be found in [Sunnydreamrain/IndRNN_Theano_Lasagne](https://github.com/Sunnydreamrain/IndRNN_Theano_Lasagne).\n\n[![Build Status](https://travis-ci.com/batzner/indrnn.svg?branch=master)](https://travis-ci.com/batzner/indrnn)\n\n## Summary\n\nIn IndRNNs, neurons in recurrent layers are independent from each other. The basic RNN calculates the hidden state `h` with `h = act(W * input + U * state + b)`. IndRNNs use an element-wise vector multiplication `u * state` meaning each neuron has a single recurrent weight connected to its last hidden state. \n\nThe IndRNN \n- can be used efficiently with ReLU activation functions making it easier to stack multiple recurrent layers without saturating gradients\n- allows for better interpretability, as neurons in the same layer are independent from each other\n- prevents vanishing and exploding gradients by regulating each neuron's recurrent weight\n\n## Usage\n\nCopy [ind_rnn_cell.py](https://github.com/batzner/indrnn/blob/master/ind_rnn_cell.py) into your project.\n\n```python\nfrom ind_rnn_cell import IndRNNCell\n\n# Regulate each neuron's recurrent weight as recommended in the paper\nrecurrent_max = pow(2, 1 / TIME_STEPS)\n\ncell = MultiRNNCell([IndRNNCell(128, recurrent_max_abs=recurrent_max),\n                     IndRNNCell(128, recurrent_max_abs=recurrent_max)])\noutput, state = tf.nn.dynamic_rnn(cell, input_data, dtype=tf.float32)\n...\n```\n## Experiments in the paper\n\n### Addition Problem\n\nSee [examples/addition_rnn.py](https://github.com/batzner/indrnn/blob/master/examples/addition_rnn.py) for a script reproducing the \"Adding Problem\" from the paper. Below are the results reproduced with the `addition_rnn.py` code.\n\n![https://github.com/batzner/indrnn/raw/master/img/addition/TAll.png](https://github.com/batzner/indrnn/raw/master/img/addition/TAll.png)\n \n### Sequential MNIST\n\nSee [examples/sequential_mnist.py](https://github.com/batzner/indrnn/blob/master/examples/sequential_mnist.py) for a script reproducing the Sequential MNIST experiment. I let it run for two days and stopped it after 60,000 training steps with a\n\n- Training error rate of 0.7%\n- Validation error rate of 1.1%\n- **Test error rate of 1.1%**\n\n![https://github.com/batzner/indrnn/raw/master/img/sequential_mnist/errors.png](https://github.com/batzner/indrnn/raw/master/img/sequential_mnist/errors.png)\n\n\n## Requirements\n- Python 3.4+\n- TensorFlow 1.5+\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbatzner%2Findrnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbatzner%2Findrnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbatzner%2Findrnn/lists"}