{"id":13659864,"url":"https://github.com/jcjohnson/torch-rnn","last_synced_at":"2025-05-15T07:06:44.556Z","repository":{"id":37733260,"uuid":"51682497","full_name":"jcjohnson/torch-rnn","owner":"jcjohnson","description":"Efficient, reusable RNNs and LSTMs for torch","archived":false,"fork":false,"pushed_at":"2022-06-21T21:10:11.000Z","size":850,"stargazers_count":2522,"open_issues_count":111,"forks_count":508,"subscribers_count":87,"default_branch":"master","last_synced_at":"2025-04-14T12:59:09.141Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jcjohnson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-02-14T06:14:36.000Z","updated_at":"2025-04-13T15:40:28.000Z","dependencies_parsed_at":"2022-09-16T06:11:02.030Z","dependency_job_id":null,"html_url":"https://github.com/jcjohnson/torch-rnn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcjohnson%2Ftorch-rnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcjohnson%2Ftorch-rnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcjohnson%2Ftorch-rnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcjohnson%2Ftorch-rnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jcjohnson","download_url":"https://codeload.github.com/jcjohnson/torch-rnn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254292042,"owners_count":22046426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T05:01:13.083Z","updated_at":"2025-05-15T07:06:39.538Z","avatar_url":"https://github.com/jcjohnson.png","language":"Lua","funding_links":[],"categories":["Lua","Codes"],"sub_categories":[],"readme":"# torch-rnn\ntorch-rnn provides high-performance, reusable RNN and LSTM modules for torch7, and uses these modules for character-level\nlanguage modeling similar to [char-rnn](https://github.com/karpathy/char-rnn).\n\nYou can find documentation for the RNN and LSTM modules [here](doc/modules.md); they have no dependencies other than `torch`\nand `nn`, so they should be easy to integrate into existing projects.\n\nCompared to char-rnn, torch-rnn is up to **1.9x faster** and uses up to **7x less memory**. For more details see \nthe [Benchmark](#benchmarks) section below.\n\n\n# Installation\n\n## Docker Images\nCristian Baldi has prepared Docker images for both CPU-only mode and GPU mode;\nyou can [find them here](https://github.com/crisbal/docker-torch-rnn).\n\n## System setup\nYou'll need to install the header files for Python 2.7 and the HDF5 library. On Ubuntu you should be able to install\nlike this:\n\n```bash\nsudo apt-get -y install python2.7-dev\nsudo apt-get install libhdf5-dev\n```\n\n## Python setup\nThe preprocessing script is written in Python 2.7; its dependencies are in the file `requirements.txt`.\nYou can install these dependencies in a virtual environment like this:\n\n```bash\nvirtualenv .env                  # Create the virtual environment\nsource .env/bin/activate         # Activate the virtual environment\npip install -r requirements.txt  # Install Python dependencies\n# Work for a while ...\ndeactivate                       # Exit the virtual environment\n```\n\n## Lua setup\nThe main modeling code is written in Lua using [torch](http://torch.ch); you can find installation instructions\n[here](http://torch.ch/docs/getting-started.html#_). You'll need the following Lua packages:\n\n- [torch/torch7](https://github.com/torch/torch7)\n- [torch/nn](https://github.com/torch/nn)\n- [torch/optim](https://github.com/torch/optim)\n- [lua-cjson](https://luarocks.org/modules/luarocks/lua-cjson)\n- [torch-hdf5](https://github.com/deepmind/torch-hdf5)\n\nAfter installing torch, you can install / update these packages by running the following:\n\n```bash\n# Install most things using luarocks\nluarocks install torch\nluarocks install nn\nluarocks install optim\nluarocks install lua-cjson\n\n# We need to install torch-hdf5 from GitHub\ngit clone https://github.com/deepmind/torch-hdf5\ncd torch-hdf5\nluarocks make hdf5-0-0.rockspec\n```\n\n### CUDA support (Optional)\nTo enable GPU acceleration with CUDA, you'll need to install CUDA 6.5 or higher and the following Lua packages:\n- [torch/cutorch](https://github.com/torch/cutorch)\n- [torch/cunn](https://github.com/torch/cunn)\n\nYou can install / update them by running:\n\n```bash\nluarocks install cutorch\nluarocks install cunn\n```\n\n## OpenCL support (Optional)\nTo enable GPU acceleration with OpenCL, you'll need to install the following Lua packages:\n- [cltorch](https://github.com/hughperkins/cltorch)\n- [clnn](https://github.com/hughperkins/clnn)\n\nYou can install / update them by running:\n\n```bash\nluarocks install cltorch\nluarocks install clnn\n```\n\n## OSX Installation\nJeff Thompson has written a very detailed installation guide for OSX that you [can find here](http://www.jeffreythompson.org/blog/2016/03/25/torch-rnn-mac-install/).\n\n# Usage\nTo train a model and use it to generate new text, you'll need to follow three simple steps:\n\n## Step 1: Preprocess the data\nYou can use any text file for training models. Before training, you'll need to preprocess the data using the script\n`scripts/preprocess.py`; this will generate an HDF5 file and JSON file containing a preprocessed version of the data.\n\nIf you have training data stored in `my_data.txt`, you can run the script like this:\n\n```bash\npython scripts/preprocess.py \\\n  --input_txt my_data.txt \\\n  --output_h5 my_data.h5 \\\n  --output_json my_data.json\n```\n\nThis will produce files `my_data.h5` and `my_data.json` that will be passed to the training script.\n\nThere are a few more flags you can use to configure preprocessing; [read about them here](doc/flags.md#preprocessing)\n\n## Step 2: Train the model\nAfter preprocessing the data, you'll need to train the model using the `train.lua` script. This will be the slowest step.\nYou can run the training script like this:\n\n```bash\nth train.lua -input_h5 my_data.h5 -input_json my_data.json\n```\n\nThis will read the data stored in `my_data.h5` and `my_data.json`, run for a while, and save checkpoints to files with \nnames like `cv/checkpoint_1000.t7`.\n\nYou can change the RNN model type, hidden state size, and number of RNN layers like this:\n\n```bash\nth train.lua -input_h5 my_data.h5 -input_json my_data.json -model_type rnn -num_layers 3 -rnn_size 256\n```\n\nBy default this will run in GPU mode using CUDA; to run in CPU-only mode, add the flag `-gpu -1`.\n\nTo run with OpenCL, add the flag `-gpu_backend opencl`.\n\nThere are many more flags you can use to configure training; [read about them here](doc/flags.md#training).\n\n## Step 3: Sample from the model\nAfter training a model, you can generate new text by sampling from it using the script `sample.lua`. Run it like this:\n\n```bash\nth sample.lua -checkpoint cv/checkpoint_10000.t7 -length 2000\n```\n\nThis will load the trained checkpoint `cv/checkpoint_10000.t7` from the previous step, sample 2000 characters from it,\nand print the results to the console.\n\nBy default the sampling script will run in GPU mode using CUDA; to run in CPU-only mode add the flag `-gpu -1` and\nto run in OpenCL mode add the flag `-gpu_backend opencl`.\n\nThere are more flags you can use to configure sampling; [read about them here](doc/flags.md#sampling).\n\n# Benchmarks\nTo benchmark `torch-rnn` against `char-rnn`, we use each to train LSTM language models for the tiny-shakespeare dataset\nwith 1, 2 or 3 layers and with an RNN size of 64, 128, 256, or 512. For each we use a minibatch size of 50, a sequence \nlength of 50, and no dropout. For each model size and for both implementations, we record the forward/backward times and \nGPU memory usage over the first 100 training iterations, and use these measurements to compute the mean time and memory \nusage.\n\nAll benchmarks were run on a machine with an Intel i7-4790k CPU, 32 GB main memory, and a Titan X GPU.\n\nBelow we show the forward/backward times for both implementations, as well as the mean speedup of `torch-rnn` over \n`char-rnn`. We see that `torch-rnn` is faster than `char-rnn` at all model sizes, with smaller models giving a larger\nspeedup; for a single-layer LSTM with 128 hidden units, we achieve a **1.9x speedup**; for larger models we achieve about\na 1.4x speedup.\n\n\u003cimg src='imgs/lstm_time_benchmark.png' width=\"800px\"\u003e\n\nBelow we show the GPU memory usage for both implementations, as well as the mean memory saving of `torch-rnn` over\n`char-rnn`. Again `torch-rnn` outperforms `char-rnn` at all model sizes, but here the savings become more significant for\nlarger models: for models with 512 hidden units, we use **7x less memory** than `char-rnn`.\n\n\u003cimg src='imgs/lstm_memory_benchmark.png' width=\"800px\"\u003e\n\n\n# TODOs\n- Get rid of Python / JSON / HDF5 dependencies?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcjohnson%2Ftorch-rnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjcjohnson%2Ftorch-rnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcjohnson%2Ftorch-rnn/lists"}