{"id":16843475,"url":"https://github.com/lopuhin/transformer-lm","last_synced_at":"2025-07-23T02:05:37.372Z","repository":{"id":138539560,"uuid":"176069746","full_name":"lopuhin/transformer-lm","owner":"lopuhin","description":"Transformer language model (GPT-2) with sentencepiece tokenizer","archived":false,"fork":false,"pushed_at":"2021-02-27T15:23:53.000Z","size":368,"stargazers_count":164,"open_issues_count":10,"forks_count":46,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-06-15T14:09:10.605Z","etag":null,"topics":["gpt-2","language-model","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lopuhin.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-17T07:30:07.000Z","updated_at":"2025-03-05T08:43:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"cdaa93d7-21e3-4525-bcca-3af389e1fa66","html_url":"https://github.com/lopuhin/transformer-lm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lopuhin/transformer-lm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lopuhin%2Ftransformer-lm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lopuhin%2Ftransformer-lm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lopuhin%2Ftransformer-lm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lopuhin%2Ftransformer-lm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lopuhin","download_url":"https://codeload.github.com/lopuhin/transformer-lm/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lopuhin%2Ftransformer-lm/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266604009,"owners_count":23954725,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpt-2","language-model","tensorflow"],"created_at":"2024-10-13T12:51:23.424Z","updated_at":"2025-07-23T02:05:37.350Z","avatar_url":"https://github.com/lopuhin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Training GPT-2 transformer language model with sentencepiece tokenizer\n======================================================================\n\n.. image:: https://img.shields.io/travis/lopuhin/transformer-lm/master.svg\n   :target: https://travis-ci.org/lopuhin/transformer-lm\n   :alt: Build Status\n\nTraining GPT-2 transformer language model on your own corpora\nwith `sentencepiece \u003chttps://github.com/google/sentencepiece\u003e`_ tokenization.\n\nThis repo contains a PyTorch implementation of GPT-2, which support multi-GPU\ntraining.\nIt also contains a TensorFlow implementation in ``lm/gpt_2_tf``,\nbut it is not developed any more. They share the same data preparation scripts.\nTF training command is ``gpt-2-tf-train`` and needs TensorFlow 1.13.\nDocumentation below is for PyTorch version.\n\n.. contents::\n\nInstallation\n------------\n\nPython 3.6+ is required with torch nightly or 1.6.0+.\nWorking in a virtualenv is assumed below.\n`Install \u003chttps://pytorch.org/get-started/locally/\u003e`__\nappropriate version of pytorch first, and then::\n\n    pip install -r requirements.txt\n    python setup.py develop\n\n\nUsage\n-----\n\nInstructions are below. See also ``test/test_shakespeare.sh``\nfor a complete pipeline demo on a small corpus (takes a minute on a CPU).\n\nPrepare data for training\n+++++++++++++++++++++++++\n\nCorpus format: a directory with top-level ``train``, ``valid`` and ``test``\nfolders. Each top-level folder may contain sub-folders. Inside them,\nthere must be utf-8 encoded text files with ``.txt`` extension.\n\nThe commands to train sentencepiece model and encode the corpus support\nmultiple corpora,\nin below examples we assume they can be listed as ``data/corpora-*``.\n\n1. Train sentencepiece model (``sp-text.txt`` can be removed after running).\n   This can consume a large amount of memory, adjust sentencepiece arguments\n   as advised if needed\n   (this is not supported in the ``sp-train`` command directly)::\n\n    sp-train data/corpora-* sp-text.txt sp-model\n\n2. Encode corpora, producing numpy files::\n\n    sp-encode data/corpora-* sp-model.model data/encoded\n\n\nTraining\n++++++++\n\nExample command::\n\n    gpt-2 run-root data/encoded sp-model.model\n\n``run-root`` would contain model checkpoints and json-lines logs,\nwhich can be plotted in a jupyter notebook with\n``json_log_plots.plot(\"run-root\")``, with number of tokens seen on the X axis.\n\nDefault hyperparameters correspond to released \"small\" GPT-2 model.\n\nWhen multiple GPUs are available, they would be used for training with the\nhelp of ``torch.distributed``.\n\nIf the path exists and ``--clean`` key is NOT passed, training would be resumed.\nNote that all parameters still need to be specified and\nmodel parameters need to match.\n\nNotes on training parameters:\n\n- ``--batch-size`` is per-GPU, so you don't need to re-tune it when changing\n  number of GPUs, just use max that fits into memory.\n- ``--g-accum-gradients`` is the global number of gradient accumulations,\n  it must be divisible by the number of GPUs. Effective global batch size is\n  always ``batch_size * g_accum_gradients``.\n- ``--lr`` does not need to be changed when changing\n  ``--batch-size`` or ``--g-accum-gradients`` or number of GPUs\n  or ``--n-ctx``: loss is already scaled appropriately.\n\n\nInference\n+++++++++\n\nExample command::\n\n    gpt-2-gen run-root \"Artificial intelligence\"\n\n``run-root`` would contain model checkpoints\n``\"Artificial intelligence\"`` is the text prefix used as a starting point for generating tokens\n\nNotes on inference parameters:\n\n- ``--tokens-to-generate``: number of tokens to generate, default is 42\n- ``--top-k``: number of token candidates to generate for each position (beam width), default is 8.\n\n\nLicense \u0026 credits\n-----------------\n\nLicense is MIT.\n\nTensorFlow GPT-2 model is taken from\nhttps://github.com/openai/gpt-2/blob/master/src/model.py\nand TensorFlow GPT-2 training code is based on\nhttps://github.com/nshepperd/gpt-2/blob/finetuning/train.py\n\nPyTorch port is based on original OpenAI code.\n\nTest Shakespeare corpus under ``tests/shakespeare``\nis from http://shakespeare.mit.edu under public domain.\n\nSee also OpenAI GPT-2\n`paper \u003chttps://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf\u003e`_\nand `blog \u003chttps://openai.com/blog/better-language-models/\u003e`_.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flopuhin%2Ftransformer-lm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flopuhin%2Ftransformer-lm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flopuhin%2Ftransformer-lm/lists"}