{"id":15654054,"url":"https://github.com/mattdangerw/keras-text-generation","last_synced_at":"2025-04-30T22:26:40.862Z","repository":{"id":27201503,"uuid":"94829643","full_name":"mattdangerw/keras-text-generation","owner":"mattdangerw","description":"RNN text generation using Keras for word and character level models.","archived":false,"fork":false,"pushed_at":"2023-03-25T00:21:21.000Z","size":794,"stargazers_count":27,"open_issues_count":3,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-30T20:33:52.859Z","etag":null,"topics":["deep-learning","keras","machine-learning","python","recurrent-neural-networks","text-generation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mattdangerw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-19T23:33:23.000Z","updated_at":"2021-03-23T17:18:22.000Z","dependencies_parsed_at":"2024-10-23T03:46:21.730Z","dependency_job_id":"54cca335-cde4-40f9-95ae-82c2e8f68fa7","html_url":"https://github.com/mattdangerw/keras-text-generation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattdangerw%2Fkeras-text-generation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattdangerw%2Fkeras-text-generation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattdangerw%2Fkeras-text-generation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattdangerw%2Fkeras-text-generation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mattdangerw","download_url":"https://codeload.github.com/mattdangerw/keras-text-generation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251790831,"owners_count":21644298,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","keras","machine-learning","python","recurrent-neural-networks","text-generation"],"created_at":"2024-10-03T12:49:20.902Z","updated_at":"2025-04-30T22:26:40.844Z","avatar_url":"https://github.com/mattdangerw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Keras Text Generation\n=====================\n\nRecurrent neural network (RNN) text generation using Keras. Generating text with\nneural networks [is fun](http://karpathy.github.io/2015/05/21/rnn-effectiveness/),\nand there are a ton of projects and standalone scripts to do it.\n\nThis project does not provide any groundbreaking features over what it already\nout there, but attempts to be a good, well documented place to start playing\nwith text generation within the Keras framework. It handles the nitty-gritty\ndetails of loading a text corpus and feeding it into a Keras model.\n\nSupports both a character-level model and a word-level model (with\ntokenization). Supports saving a model and model metadata to disk for later\nsampling. Supports using a validation set. Uses stateful RNNs within Keras for\nmore efficient sampling.\n\nRequirements\n------------\n\n- Keras 2.0\n- Colorama 0.3\n\nQuick start\n-----------\n\n```shell\npip install tensorflow-gpu # Or tensorflow or Theano\npip install keras colorama\n# Train on the included Shakespeare corpus with default parameters\npython train.py\n# Sample the included Shakespeare corpus with default parameters\npython samply.py\n# Train with long samples, more layers, more epochs, and live sampling\npython train.py --seq-length 100 --num-layers 4 --num-epochs 100 --live-sample\n# Sample with a random seed for 500 characters and more random output\npython sample.py --length 500 --temperature 2.0\n# Train on a new dataset with a word level model and larger embedding\npython train.py --data-dir ~/datasets/twain --word-tokens --embedding-size 128\n# Sample new dataset with a custom seed\npython sample.py --data-dir ~/datasets/twain --seed \"History doesn't repeat itself, but\"\n```\n\nUsage\n-----\n\nThere are two invokable scripts, `train.py` and `sample.py`, which should be run\nin succession. Each operates on a data directory whose contents are as follows:\n\n- **input.txt**, input text corpora. Required by `train.py`\n- **validate.txt**, optional validation text corpora. Used in `train.py`\n- **model.h5**, keras model weights. Created by `train.py` and required by `sample.py`\n- **model.pkl**, model metadata. Created by `train.py` and required by `sample.py`\n\nThe `input.txt` file should contain whatever texts you would like to train the\nRNN on, concatenated into a single file. The text processing is by default\nnewline aware, so if you files contain hard wrapped prose, you may want to\nremove the wrapping newlines. The `validate.txt` file should be formatted\nsimilarly to the input.txt. It is totally optional, but useful to monitor\nfor overfitting, etc.\n\nThere are two main modes to process the input--a character-level model and\na word-level model. Under the character level model, we will simply lowercase\nthe input text and feed it into the RNN character by character. Under the word\nlevel model, the input text will be split into individual word tokens and each\ntoken will be given a separate value before being fed into the RNN. Word will be\ntokenized roughly following the Penn Treebank approach. By default we will\nheuristically attempt to \"detokenize\" the text after sampling, but this can be\ndisabled with `--pristine-output`.\n\n### train.py\n\n- **--data-dir**, type=string, default=data/tinyshakespeare. The data directory\n  containing an `input.txt` file.\n- **--live-sample**, type=flag. Sample the model after every epoch. Very\n  useful if you want to quickly iterate on a new approach.\n- **--word-tokens**, type=flag. Whether to model the RNN at a word level or a\n  a character level.\n- **--pristine-input**, type=flag. For character models, do not lowercase the\n  text corpora before feeding it into the RNN. For word models, do not attempt\n  fancy tokenization. You can pass this this to use your own tokenizer as a\n  preprocessing step. Implies `--pristine-output`.\n- **--pristine-output**, type=flag. For word models, do not attempt to\n  \"detokenize\" the output.\n- **--embedding-size**, type=int, default=64. Size of the embedding layer.\n  This can be much lower when using the character level model, and bigger under\n  the word level model.\n- **--rnn-size**, type=int, default=128. Number of LSTM cells in each RNN\n  layer.\n- **--num-layers**, type=int, default=2. Number of layers in the RNN.\n- **--batch-size**, type=int, default=32. Batch size, i.e. how many samples\n  to process in parallel during training.\n- **--seq-length**, type=int, default=50. We will split the input text into\n  into individual samples of length `--seq-length` before feeding them into the\n  RNN. The model will be bad at learning dependencies in the text longer then\n  `--seq-length`.\n- **--seq-step**, type=int, default=25. We grab samples from the input text\n  semi redundantly every `--seq-step` characters (or words). For example, a\n  `--seq-length` of 50 and `--seq-step` of 25 would pull each character in the\n  text into two separate samples offset from each other by 25 characters.\n- **--num-epochs**, type=int, default=50. Number of epochs, i.e. passes over\n  the entire dataset during training.\n\n### sample.py\n\n- **--data-dir**, type=string, default=data/tinyshakespeare. The data directory\n  containing `model.h5` and `model.pkl` files generated by `train.py`.\n- **--seed**, type=string, default=None. Seed string for sampling. If no seed\n  is supplied we will grab a random seed from the input text.\n- **--length**, type=int, default=1000. Length of the sample to generate. For\n  the word level model this is length in words.\n- **--diversity**, type=float, default=1.0. Sampling diversity. A diversity\n  of \u003c 1.0 will make take conservative guesses from the RNN when generating\n  text. A diversity of \u003e 1.0 will make riskier choices when generating text.\n\nFAQ\n---\n\n**Why not just use [char-rnn-tensorflow](https://github.com/sherjilozair/char-rnn-tensorflow) or [word-rnn-tensorflow](https://github.com/hunkim/word-rnn-tensorflow)?**\n\nIf your goal is just computational speed or low memory footprint, go with those\nprojects! Pretty much the appeal here is using Keras. If you want an easy\ndeclarative framework to try new approaches, this is a good place to start.\n\nThere are also a few additional features here, such as fancier word tokenization\nand support for a hold out validation set, that may be of use depending on your\napplication.\n\n**Can we add a command line flag for a different optimizer, RNN cell, etc.?**\n\nMost of command line flags exposed are to work with different datasets of\nvarying sizes. If you want to change the structure of the RNN, just change the\ncode. That's where Keras excels.\n\n**Can I use a different tokenization scheme for my word level model?**\n\nYep! Pass the `--pristine-input` flag and use a fancier tokenizer as a\npreprocessing step. Tokens will be formed by calling `text.split()` on the\ninput.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmattdangerw%2Fkeras-text-generation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmattdangerw%2Fkeras-text-generation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmattdangerw%2Fkeras-text-generation/lists"}