{"id":13487110,"url":"https://github.com/batzner/tensorlm","last_synced_at":"2025-04-19T15:18:22.909Z","repository":{"id":24359783,"uuid":"100861317","full_name":"batzner/tensorlm","owner":"batzner","description":"Wrapper library for text generation / language models at char and word level with RNN in TensorFlow","archived":false,"fork":false,"pushed_at":"2022-06-21T21:07:11.000Z","size":81,"stargazers_count":61,"open_issues_count":2,"forks_count":30,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-04-24T01:02:40.094Z","etag":null,"topics":["char-lm","char-rnn","language-model","nlp","tensorflow","tensorflow-library"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/batzner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-20T12:54:14.000Z","updated_at":"2023-08-09T05:54:52.000Z","dependencies_parsed_at":"2022-09-12T21:01:33.572Z","dependency_job_id":null,"html_url":"https://github.com/batzner/tensorlm","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batzner%2Ftensorlm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batzner%2Ftensorlm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batzner%2Ftensorlm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batzner%2Ftensorlm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/batzner","download_url":"https://codeload.github.com/batzner/tensorlm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226451043,"owners_count":17627209,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["char-lm","char-rnn","language-model","nlp","tensorflow","tensorflow-library"],"created_at":"2024-07-31T18:00:55.493Z","updated_at":"2024-11-28T21:06:05.953Z","avatar_url":"https://github.com/batzner.png","language":"Python","readme":"# tensorlm\n\nGenerate Shakespeare poems with 4 lines of code.\n\n\u003ca href=\"https://theblog.github.io/post/character-language-model-lstm-tensorflow/\" target=\"_blank\"\u003e[![showcase of the package](http://i.cubeupload.com/8Cm5RQ.gif)](http://theblog.github.io/post/character-language-model-lstm-tensorflow/)\u003c/a\u003e\n\n## Installation\n\n`tensorlm` is written in / for Python 3.4+ and TensorFlow 1.1+\n\n    pip3 install tensorlm\n    \n## Basic Usage\n\nUse the `CharLM` or `WordLM` class:\n```python\nimport tensorflow as tf\nfrom tensorlm import CharLM\n    \nwith tf.Session() as session:\n    \n    # Create a new model. You can also use WordLM\n    model = CharLM(session, \"datasets/sherlock/tinytrain.txt\", max_vocab_size=96,\n                   neurons_per_layer=100, num_layers=3, num_timesteps=15)\n    \n    # Train it \n    model.train(session, max_epochs=10, max_steps=500)\n    \n    # Let it generate a text\n    generated = model.sample(session, \"The \", num_steps=100)\n    print(\"The \" + generated)\n```\n\nThis should output something like:\n\n    The  ee e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e \n  \n## Command Line Usage\n\n**Train:** \n`python3 -m tensorlm.cli --train=True --level=char --train_text_path=datasets/sherlock/tinytrain.txt --max_vocab_size=96 --neurons_per_layer=100 --num_layers=2 --batch_size=10 --num_timesteps=15 --save_dir=out/model --max_epochs=300 --save_interval_hours=0.5`\n\n**Sample:**\n`python3 -m tensorlm.cli --sample=True --level=char --neurons_per_layer=400 --num_layers=3 --num_timesteps=160 --save_dir=out/model`\n\n**Evaluate:**\n`python3 -m tensorlm.cli --evaluate=True --level=char --evaluate_text_path=datasets/sherlock/tinyvalid.txt --neurons_per_layer=400 --num_layers=3 --batch_size=10 --num_timesteps=160 --save_dir=out/model`\n\n\nSee `python3 -m tensorlm.cli --help` for all options.\n\n## Advanced Usage\n\n### Custom Input Data\n\nThe inputs and targets don't have to be text. `GeneratingLSTM` only expects token ids, so you can use any data type for the sequences, as long as you can encode the data to integer ids.\n```python\n# We use integer ids from 0 to 19, so the vocab size is 20. The range of ids must always start\n# at zero.\nbatch_inputs = np.array([[1, 2, 3, 4], [15, 16, 17, 18]])  # 2 batches, 4 time steps each\nbatch_targets = np.array([[2, 3, 4, 5], [16, 17, 18, 19]])\n\n# Create the model in a TensorFlow graph\nmodel = GeneratingLSTM(vocab_size=20, neurons_per_layer=10, num_layers=2, max_batch_size=2)\n\n# Initialize all defined TF Variables\nsession.run(tf.global_variables_initializer())\n\nfor _ in range(5000):\n    model.train_step(session, batch_inputs, batch_targets)\n\nsampled = model.sample_ids(session, [15], num_steps=3)\nprint(\"Sampled: \" + str(sampled))\n```\n\nThis should output something like:\n\n    Sampled: [16, 18, 19]\n\n### Custom Training, Dropout etc.\n\nUse the `GeneratingLSTM` class directly. This class is agnostic to the dataset type. It expects integer ids and returns integer ids.\n\n```python\nimport tensorflow as tf\nfrom tensorlm import Vocabulary, Dataset, GeneratingLSTM\n\nBATCH_SIZE = 20\nNUM_TIMESTEPS = 15\n\nwith tf.Session() as session:\n    # Generate a token -\u003e id vocabulary based on the text\n    vocab = Vocabulary.create_from_text(\"datasets/sherlock/tinytrain.txt\", max_vocab_size=96,\n                                        level=\"char\")\n\n    # Obtain input and target batches from the text file\n    dataset = Dataset(\"datasets/sherlock/tinytrain.txt\", vocab, BATCH_SIZE, NUM_TIMESTEPS)\n\n    # Create the model in a TensorFlow graph\n    model = GeneratingLSTM(vocab_size=vocab.get_size(), neurons_per_layer=100, num_layers=2,\n                           max_batch_size=BATCH_SIZE, output_keep_prob=0.5)\n\n    # Initialize all defined TF Variables\n    session.run(tf.global_variables_initializer())\n\n    # Do the training\n    epoch = 1\n    step = 1\n    for epoch in range(20):\n        for inputs, targets in dataset:\n            loss = model.train_step(session, inputs, targets)\n\n            if step % 100 == 0:\n                # Evaluate from time to time\n                dev_dataset = Dataset(\"datasets/sherlock/tinyvalid.txt\", vocab,\n                                      batch_size=BATCH_SIZE, num_timesteps=NUM_TIMESTEPS)\n                dev_loss = model.evaluate(session, dev_dataset)\n                print(\"Epoch: %d, Step: %d, Train Loss: %f, Dev Loss: %f\" % (\n                    epoch, step, loss, dev_loss))\n\n                # Sample from the model from time to time\n                print(\"Sampled: \\\"The \" + model.sample_text(session, vocab, \"The \") + \"\\\"\")\n\n            step += 1\n\n```\n\nThis should output something like:\n\n    Epoch: 3, Step: 100, Train Loss: 3.824941, Dev Loss: 3.778008\n    Sampled: \"The                                                                                                     \"\n    Epoch: 7, Step: 200, Train Loss: 2.832825, Dev Loss: 2.896187\n    Sampled: \"The                                                                                                     \"\n    Epoch: 11, Step: 300, Train Loss: 2.778579, Dev Loss: 2.830176\n    Sampled: \"The         eee                                                                                         \"\n    Epoch: 15, Step: 400, Train Loss: 2.655153, Dev Loss: 2.684828\n    Sampled: \"The        ee    e  e   e  e  e  e  e  e  e   e  e  e   e  e  e   e  e  e   e  e  e   e  e  e   e  e  e \"\n    Epoch: 19, Step: 500, Train Loss: 2.444502, Dev Loss: 2.479753\n    Sampled: \"The    an  an  an  on  on  on  on  on  on  on  on  on  on  on  on  on  on  on  on  on  on  on  on  on  o\"\n","funding_links":[],"categories":["The Data Science Toolbox","Tensor Flow","Deep Learning"],"sub_categories":["Deep Learning Packages","Automated Machine Learning","TensorFlow"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbatzner%2Ftensorlm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbatzner%2Ftensorlm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbatzner%2Ftensorlm/lists"}