{"id":16162446,"url":"https://github.com/sunsided/tensorflow-lstm-sin","last_synced_at":"2025-08-19T22:06:41.187Z","repository":{"id":50228305,"uuid":"74840601","full_name":"sunsided/tensorflow-lstm-sin","owner":"sunsided","description":"TensorFlow 1.3 experiment with LSTM (and GRU) RNNs for sine prediction","archived":false,"fork":false,"pushed_at":"2017-09-24T02:03:16.000Z","size":920,"stargazers_count":54,"open_issues_count":0,"forks_count":25,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-02-28T12:33:11.022Z","etag":null,"topics":["artificial-intelligence","experiment","gru","lstm","neural-network","prediction","recurrent-neural-networks","signal-processing","tensorflow","timeseries"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sunsided.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-11-26T17:33:41.000Z","updated_at":"2024-03-29T16:54:36.000Z","dependencies_parsed_at":"2022-09-24T09:02:10.817Z","dependency_job_id":null,"html_url":"https://github.com/sunsided/tensorflow-lstm-sin","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunsided%2Ftensorflow-lstm-sin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunsided%2Ftensorflow-lstm-sin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunsided%2Ftensorflow-lstm-sin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunsided%2Ftensorflow-lstm-sin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sunsided","download_url":"https://codeload.github.com/sunsided/tensorflow-lstm-sin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243950927,"owners_count":20373664,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","experiment","gru","lstm","neural-network","prediction","recurrent-neural-networks","signal-processing","tensorflow","timeseries"],"created_at":"2024-10-10T02:30:09.596Z","updated_at":"2025-03-18T22:31:08.590Z","avatar_url":"https://github.com/sunsided.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TensorFlow LSTM sin(t) Example\n\nSingle- and multilayer LSTM networks with no additional output nonlinearity based on \n[aymericdamien's TensorFlow examples](https://github.com/aymericdamien/TensorFlow-Examples/)\nand [Sequence prediction using recurrent neural networks](http://mourafiq.com/2016/05/15/predicting-sequences-using-rnn-in-tensorflow.html).\n\nExperiments with varying numbers of hidden units, LSTM cells and techniques like gradient clipping were conducted using `static_rnn` and `dynamic_rnn`. All networks have been optimized using [Adam](https://arxiv.org/abs/1412.6980) on the MSE loss function.\n\n## Experiment 1\n\nGiven a single LSTM cell with `100` hidden states, predict the next `50` timesteps \ngiven the last `100` timesteps. \n\nThe network is trained on a sine of `1 Hz` only using random shifts, thus fails on\ngeneralizing to higher frequencies (`2 Hz` and `3 Hz` in the image); in addition, the\nnetwork should be able to simply memoize the shape of the input.\nIt was optimized with a learning rate of `0.001` for `200000` iterations and \nbatches of `50` examples.\n\n![](images/tf-recurrent-sin.jpg)\n\n## Experiment 2\n\nGiven a single LSTM cell with `150` hidden states, predict the next `50` timesteps \ngiven the last `100` timesteps. \n\nThe network is trained on sines of random frequencies between `0.5 .. 4 Hz` using \nrandom shifts. Prediction quality is worse than for the `1 Hz` only experiment above,\nbut it generalizes to the `2 Hz` and `3 Hz` tests.\nIt was optimized with a learning rate of `0.001` for `300000` iterations and \nbatches of `50` examples.\n\nAt loss `0.614914`, the prediction looks like this:\n\n![](images/tf-recurrent-sin-2.jpg)\n\n## Experiment 3\n\nGiven a single LSTM cell with `150` hidden states, predict the next `50` timesteps \ngiven the last `25` timesteps. \n\nThe network is trained on sines of random frequencies between `0.5 Hz` and `4 Hz` using \nrandom shifts. Prediction quality is worse than for the `1 Hz` only experiment above,\nbut it generalizes to the `2 Hz` and `3 Hz` tests.\nIt was optimized with a learning rate of `0.0005` for `500000` iterations and \nbatches of `50` examples.\n\nThe following image shows the output at loss `0.177742`:\n\n![](images/tf-recurrent-sin-3.jpg)\n\nThe following is the network trained to predict the next `100` timesteps\ngiven the previous `25` timesteps; the parameters are otherwise unchanged.\n\nThis is the result at loss `0.257725`:\n\n![](images/tf-recurrent-sin-3.1.jpg)\n\n## Experiment 4\n\nSame as the last experiment, however using `500` hidden states and gradient clipping\nfor the optimizer as described [here](http://stackoverflow.com/a/36501922/195651):\n\n```python\nadam = tf.train.AdamOptimizer(learning_rate=learning_rate)\ngradients = adam.compute_gradients(loss)\nclipped_gradients = [(tf.clip_by_value(grad, -0.5, 0.5), var) for grad, var in gradients]\noptimizer = adam.apply_gradients(clipped_gradients)\n```\n\nLosses get as low as `0.069027` within the given iterations, but vary wildly.\nThis is at loss `0.422188`:\n\n![](images/tf-recurrent-sin-4.jpg)\n\n## Experiment 5\n\nThis time, the `dynamic_rnn()` function is used instead of `rnn()`, drastically improving the \nstartup time. In addition, the single LSTM cell has been replaced with `4` stacked \nLSTM cells of `32` hidden states each.\n\n```python\nlstm_cells = [rnn.LSTMCell(n_hidden, forget_bias=1.0) \n              for _ in range(n_layers)]\nstacked_lstm = rnn.MultiRNNCell(lstm_cells)\noutputs, states = tf.nn.dynamic_rnn(stacked_lstm, x, dtype=tf.float32, time_major=False)\n```\n\nThe output still uses linear regression:\n\n```python\noutput = tf.transpose(outputs, [1, 0, 2])\npred = tf.matmul(output[-1], weights['out']) + biases['out']\n```\n\nThe network is trained with learning rate `0.001` for at least `300000` iterations\n(with the additional criterion that the loss should be below `0.15`).\n\nThe following picture shows the performance at loss `0.138701` at iteration `375000`.\n\n![](images/tf-recurrent-sin-5.jpg)\n\nWhen using only `10` hidden states, training takes much longer given a learning rate of `0.001`\nand reaches a loss of about `0.5` after `~1200000` iterations, where convergence effectively stops.\n\nThe following used `10` hidden states and a base learning rate of `0.005` in combination with a \nstep policy that reduced the learning rate by a factor of `0.1` every `250000` iterations.\nSimilar to the previous experiment, optimization was stopped after at least `300000` iterations \nhave passed and the loss was below `0.2`.\n\nThe picture shows the outcome after `510000` iterations at a loss of `0.180995`:\n\n![](images/tf-recurrent-sin-5.1.jpg)\n\n## Experiment 6\n\nMuch like the last experiment, this one uses `10` hidden states per layer\nin a `4` layer deep recurrent structure. Instead of using LSTM layers, however,\nthis one uses GRUs.\n\nBecause the loss did not go below `0.3`, training was stopped after `1000000`\niterations.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsunsided%2Ftensorflow-lstm-sin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsunsided%2Ftensorflow-lstm-sin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsunsided%2Ftensorflow-lstm-sin/lists"}