{"id":13564871,"url":"https://github.com/huggingface/pytorch-openai-transformer-lm","last_synced_at":"2025-05-16T01:06:10.670Z","repository":{"id":44427973,"uuid":"137223715","full_name":"huggingface/pytorch-openai-transformer-lm","owner":"huggingface","description":"🐥A PyTorch implementation of OpenAI's finetuned transformer language model with a script to import the weights pre-trained by OpenAI","archived":false,"fork":false,"pushed_at":"2021-08-09T16:17:12.000Z","size":273,"stargazers_count":1509,"open_issues_count":24,"forks_count":285,"subscribers_count":90,"default_branch":"master","last_synced_at":"2025-05-16T00:07:31.300Z","etag":null,"topics":["language-model","neural-networks","openai","pytorch","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-13T14:02:41.000Z","updated_at":"2025-05-12T01:07:03.000Z","dependencies_parsed_at":"2022-08-23T23:20:14.351Z","dependency_job_id":null,"html_url":"https://github.com/huggingface/pytorch-openai-transformer-lm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fpytorch-openai-transformer-lm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fpytorch-openai-transformer-lm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fpytorch-openai-transformer-lm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fpytorch-openai-transformer-lm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/pytorch-openai-transformer-lm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254448579,"owners_count":22072764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["language-model","neural-networks","openai","pytorch","transformer"],"created_at":"2024-08-01T13:01:37.325Z","updated_at":"2025-05-16T01:06:05.660Z","avatar_url":"https://github.com/huggingface.png","language":"Python","funding_links":[],"categories":["Python","Paper implementations｜论文实现","Paper implementations","Deep Learning Projects"],"sub_categories":["Other libraries｜其他库:","Other libraries:"],"readme":"# PyTorch implementation of OpenAI's Finetuned Transformer Language Model\n\nThis is a PyTorch implementation of the [TensorFlow code](https://github.com/openai/finetune-transformer-lm) provided with OpenAI's paper [\"Improving Language Understanding by Generative Pre-Training\"](https://blog.openai.com/language-unsupervised/) by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.\n\nThis implementation comprises **a script to load in the PyTorch model the weights pre-trained by the authors** with the TensorFlow implementation.\n\n![Transformer Language Model](assets/ftlm.png)\n\nThe model classes and loading script are located in [model_pytorch.py](model_pytorch.py).\n\nThe names of the modules in the PyTorch model follow the names of the Variable in the TensorFlow implementation. This implementation tries to follow the original code as closely as possible to minimize the discrepancies.\n\nThis implementation thus also comprises a modified Adam optimization algorithm as used in OpenAI's paper with:\n- fixed weights decay following the work of [Loshchilov et al.](https://arxiv.org/abs/1711.05101), and\n- scheduled learning rate as [commonly used for Transformers](http://nlp.seas.harvard.edu/2018/04/03/attention.html#optimizer).\n\n## Requirements\nTo use the model it-self by importing [model_pytorch.py](model_pytorch.py), you just need:\n- PyTorch (version \u003e=0.4)\n\nTo run the classifier training script in [train.py](train.py) you will need in addition:\n- tqdm\n- sklearn\n- spacy\n- ftfy\n- pandas\n\nYou can download the weights of the OpenAI pre-trained version by cloning [Alec Radford's repo](https://github.com/openai/finetune-transformer-lm) and placing the `model` folder containing the pre-trained weights in the present repo.\n\n## Using the pre-trained model as a Transformer Language Model\nThe model can be used as a transformer language model with OpenAI's pre-trained weights as follow:\n```python\nfrom model_pytorch import TransformerModel, load_openai_pretrained_model, DEFAULT_CONFIG\n\nargs = DEFAULT_CONFIG\nmodel = TransformerModel(args)\nload_openai_pretrained_model(model)\n```\n\nThis model generates Transformer's hidden states. You can use the `LMHead` class in [model_pytorch.py](model_pytorch.py) to add a decoder tied with the weights of the encoder and get a full language model. You can also use the `ClfHead` class in [model_pytorch.py](model_pytorch.py) to add a classifier on top of the transformer and get a classifier as described in OpenAI's publication. (see an example of both in the `__main__` function of [train.py](train.py))\n\nTo use the positional encoder of the transformer, you should encode your dataset using the `encode_dataset()` function of [utils.py](utils.py). Please refer to the beginning of the `__main__` function in [train.py](train.py) to see how to properly define the vocabulary and encode your dataset.\n\n## Fine-tuning the pre-trained model on a classification task\nThis model can also be integrated in a classifier as detailed in [OpenAI's paper](https://blog.openai.com/language-unsupervised/). An example of fine-tuning on the ROCStories Cloze task is included with the training code in [train.py](train.py)\n\nThe ROCStories dataset can be downloaded from the associated [website](http://cs.rochester.edu/nlp/rocstories/).\n\nAs with the [TensorFlow code](https://github.com/openai/finetune-transformer-lm), this code implements the ROCStories Cloze Test result reported in the paper which can be reproduced by running:\n\n```bash\npython -m spacy download en\npython train.py --dataset rocstories --desc rocstories --submit --analysis --data_dir [path to data here]\n```\n\n#### First experiments on the ROCStories test set\nFinetuning the PyTorch model for 3 Epochs on ROCStories takes 10 minutes to run on a single NVidia K-80.\n\nThe single run test accuracy of this PyTorch version is 85.84%, while the authors reports a median accuracy with the TensorFlow code of 85.8% and the paper reports a best single run accuracy of 86.5%.\n\nThe authors implementations uses 8 GPU and can thus accomodate a batch of 64 samples while the present implementation is single GPU and is in consequence limited to 20 instances on a K80 for memory reasons. In our test, increasing the batch size from 8 to 20 samples increased the test accuracy by 2.5 points. A better accuracy may be obtained by using a multi-GPU setting (not tried yet).\n\nThe previous SOTA on the ROCStories dataset is 77.6% (\"Hidden Coherence Model\" of Chaturvedi et al. published in \"Story Comprehension for Predicting What Happens Next\" EMNLP 2017, which is a very nice paper too!)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fpytorch-openai-transformer-lm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Fpytorch-openai-transformer-lm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fpytorch-openai-transformer-lm/lists"}