{"id":14959135,"url":"https://github.com/philipperemy/tensorflow-phased-lstm","last_synced_at":"2025-05-02T12:31:26.050Z","repository":{"id":70249862,"uuid":"75755400","full_name":"philipperemy/tensorflow-phased-lstm","owner":"philipperemy","description":"Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (NIPS 2016) - Tensorflow 1.0","archived":false,"fork":false,"pushed_at":"2018-12-19T07:37:18.000Z","size":1066,"stargazers_count":128,"open_issues_count":1,"forks_count":24,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-04-07T01:53:12.408Z","etag":null,"topics":["deep-learning","lstm-model","machine-learning","mnist-dataset","recurrent-networks","tensorflow","tensorflow-1-0"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philipperemy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-12-06T17:35:27.000Z","updated_at":"2025-03-21T16:06:13.000Z","dependencies_parsed_at":"2023-03-04T00:01:00.480Z","dependency_job_id":null,"html_url":"https://github.com/philipperemy/tensorflow-phased-lstm","commit_stats":{"total_commits":58,"total_committers":4,"mean_commits":14.5,"dds":"0.39655172413793105","last_synced_commit":"c2867162a91b5013f064cda96d7bd3651275f029"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-phased-lstm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-phased-lstm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-phased-lstm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-phased-lstm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philipperemy","download_url":"https://codeload.github.com/philipperemy/tensorflow-phased-lstm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252038155,"owners_count":21684632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","lstm-model","machine-learning","mnist-dataset","recurrent-networks","tensorflow","tensorflow-1-0"],"created_at":"2024-09-24T13:18:54.454Z","updated_at":"2025-05-02T12:31:25.301Z","avatar_url":"https://github.com/philipperemy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (NIPS 2016)\n[![license](https://img.shields.io/badge/License-MIT-brightgreen.svg)](https://github.com/philipperemy/keras-attention-mechanism/blob/master/LICENSE) [![dep1](https://img.shields.io/badge/Tensorflow-1.2+-blue.svg)](https://www.tensorflow.org/)\n\nTensorflow has released an official version of the Phased LSTM. I wrote a script to show how to use it: \n\nhttps://github.com/philipperemy/tensorflow-phased-lstm/blob/master/official_tensorflow_phased_lstm.py\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"fig/official_tensorflow_impl.png\" width=\"500\"\u003e\u003cbr\u003e\u003cbr\u003e\n  \u003cp\u003e\u003ci\u003eTraining on the classification of MNIST digits with Phased LSTM and Basic LSTM (official TF implementation)\u003c/i\u003e\u003c/p\u003e\n\u003c/div\u003e\n\n\u003chr/\u003e\n\n**NOTE: You can still use this alternative implementation (tested on Tensorflow v1.10). The code is very similar and it's as fast as the official one (0.8 seconds for a forward-backward pass on a Titan X Maxwell GPU).**\n\n## How to use it?\n```bash\ngit clone git@github.com:philipperemy/tensorflow-phased-lstm.git plstm\ncd plstm\nsudo pip install -r requirements.txt\n# make sure at least Tensorflow 1.2.0 is installed.\n# To reproduce the results of Phased LSTM on MNIST dataset.\npython mnist_phased_lstm.py -m BasicLSTMCell\npython mnist_phased_lstm.py -m PhasedLSTMCell\n```\n\n# Phased LSTM\n\nThe Phased LSTM model extends the LSTM model by adding a new time gate, kt (Fig. 1(b)). The\nopening and closing of this gate is controlled by an independent rhythmic oscillation specified by\nthree parameters; updates to the cell state ct and ht are permitted only when the gate is open. The\nfirst parameter, τ , controls the real-time period of the oscillation. The second, ron, controls the ratio\nof the duration of the “open” phase to the full period. The third, s, controls the phase shift of the\noscillation to each Phased LSTM cell.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"fig/fig1.png\"\u003e\u003cbr\u003e\u003cbr\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"fig/fig2.png\"\u003e\u003cbr\u003e\u003cbr\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"fig/fig3.png\"\u003e\u003cbr\u003e\u003cbr\u003e\n\u003c/div\u003e\n\n\n# Resuts on MNIST dataset\n\nHere are the results on the MNIST dataset on the training set. We consider MNIST as long sequences. Clearly with 32 cells, the Basic LSTM implementation cannot learn whereas Phased LSTM does pretty well.\n\n## Training Accuracy\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"fig/mnist_acc.png\" width=\"400\"\u003e\u003cbr\u003e\u003cbr\u003e\n\u003c/div\u003e\n\n## Training Loss\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"fig/mnist_ce.png\" width=\"400\"\u003e\u003cbr\u003e\u003cbr\u003e\n\u003c/div\u003e\n\nThe Phased LSTM has many surprising advantages. With its rhythmic periodicity, it acts like a\nlearnable, gated Fourier transform on its input, permitting very fine timing discrimination. Alternatively,\nthe rhythmic periodicity can be viewed as a kind of persistent dropout that preserves state [27],\nenhancing model diversity. The rhythmic inactivation can even be viewed as a shortcut to the past\nfor gradient backpropagation, accelerating training. The presented results support these interpretations,\ndemonstrating the ability to discriminate rhythmic signals and to learn long memory traces.\nImportantly, in all experiments, Phased LSTM converges more quickly and theoretically requires\nonly 5% of the computes at runtime, while often improving in accuracy compared to standard LSTM.\nThe presented methods can also easily be extended to GRUs [6], and it is likely that even simpler\nmodels, such as ones that use a square-wave-like oscillation, will perform well, thereby making even\nmore efficient and encouraging alternative Phased LSTM formulations. An inspiration for using\noscillations in recurrent networks comes from computational neuroscience [3], where rhythms have\nbeen shown to play important roles for synchronization and plasticity [22]. Phased LSTMs were\nnot designed as biologically plausible models, but may help explain some of the advantages and\nrobustness of learning in large spiking recurrent networks.\n\nFrom: https://arxiv.org/pdf/1610.09513v1.pdf\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipperemy%2Ftensorflow-phased-lstm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilipperemy%2Ftensorflow-phased-lstm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipperemy%2Ftensorflow-phased-lstm/lists"}