{"id":13934679,"url":"https://github.com/nicklashansen/rnn_lstm_from_scratch","last_synced_at":"2025-07-19T19:30:57.584Z","repository":{"id":132520690,"uuid":"213749834","full_name":"nicklashansen/rnn_lstm_from_scratch","owner":"nicklashansen","description":"How to build RNNs and LSTMs from scratch with NumPy.","archived":false,"fork":false,"pushed_at":"2020-10-04T08:39:49.000Z","size":223,"stargazers_count":252,"open_issues_count":2,"forks_count":70,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-11-27T01:32:36.790Z","etag":null,"topics":["deep-learning","lstm-neural-networks","natural-language-processing","numpy","pytorch","recurrent-neural-networks"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nicklashansen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-10-08T20:42:22.000Z","updated_at":"2024-11-19T08:57:09.000Z","dependencies_parsed_at":"2023-05-16T20:30:44.892Z","dependency_job_id":null,"html_url":"https://github.com/nicklashansen/rnn_lstm_from_scratch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nicklashansen/rnn_lstm_from_scratch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicklashansen%2Frnn_lstm_from_scratch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicklashansen%2Frnn_lstm_from_scratch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicklashansen%2Frnn_lstm_from_scratch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicklashansen%2Frnn_lstm_from_scratch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nicklashansen","download_url":"https://codeload.github.com/nicklashansen/rnn_lstm_from_scratch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicklashansen%2Frnn_lstm_from_scratch/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265998438,"owners_count":23862107,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","lstm-neural-networks","natural-language-processing","numpy","pytorch","recurrent-neural-networks"],"created_at":"2024-08-07T23:01:10.153Z","updated_at":"2025-07-19T19:30:57.320Z","avatar_url":"https://github.com/nicklashansen.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"# How to build RNNs and LSTMs from scratch with NumPy\n\n**\\[Update 08/18/2020\\] Improvements to dataset; exercises and descriptions have been made more clear.**\n\nOriginally developed by me (Nicklas Hansen), Peter E. Christensen and Alexander R. Johansen as educational material for the graduate deep learning course at the Technical University of Denmark (DTU). You can access the full course material [here](https://github.com/DeepLearningDTU/02456-deep-learning-with-PyTorch). Inspired by the great [Andrej Karpathy](https://karpathy.ai/).\n____\n\nIn this lab we will introduce different ways of learning from sequential data.\nAs an example, we will train a neural network to do language modelling, i.e. predict the next token in a sentence. In the context of natural language processing a token could be a character or a word, but mind you that the concepts introduced here apply to all kinds of sequential data, such as e.g. protein sequences, weather measurements, audio signals or monetary transaction history, just to name a few.\n\nTo really get a grasp of what is going on inside the recurrent neural networks that we are about to teach you, we will carry out a substantial part of this exercise in NumPy rather than PyTorch. Once you get a hold of it, we will proceed to the PyTorch implementation.\n\nIn this notebook we will show you:\n* How to represent categorical variables in networks\n* How to build a recurrent neural network (RNN) from scratch\n* How to build a LSTM network from scratch\n* How to build a LSTM network in PyTorch\n\n\n## Dataset\n\nFor this exercise we will create a simple dataset that we can learn from. We generate sequences of the form:\n\n`a b EOS`,\n\n`a a b b EOS`,\n\n`a a a a a b b b b b EOS`\n\nwhere `EOS` is a special character denoting the end of a sequence. The task is to predict the next token `t_n`, i.e. `a`, `b`, `EOS` or the unknown token `UNK` given the sequence of tokens `t_1`, `t_2`, `...`, `t_n-1` and we are to process sequences in a sequential manner. As such, the network will need to learn that e.g. 5 `b`s and an `EOS` token will follow 5 `a`s.\n\n## Results\n\nThe RNN takes considerable effort to converge to a nice solution:\n\n![RNN loss](https://i.imgur.com/aSeXXI5.png)\n\nThe LSTM learns much faster than the RNN:\n\n![LSTM loss](https://i.imgur.com/MioBMl2.png)\n\nAnd finally, the PyTorch LSTM learns even faster and converges to a better local minimum:\n\n![PyTorch LSTM loss](https://i.imgur.com/6HrHHRp.png)\n\nAfter working your way through these exercises, you should have a better understanding of how RNNs work, how to train them, and what they can be used for. And the conclusion? - use PyTorch.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicklashansen%2Frnn_lstm_from_scratch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnicklashansen%2Frnn_lstm_from_scratch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicklashansen%2Frnn_lstm_from_scratch/lists"}