{"id":13650087,"url":"https://github.com/FranxYao/Language-Model-Pretraining-for-Text-Generation","last_synced_at":"2025-04-22T18:30:49.193Z","repository":{"id":143212705,"uuid":"182930900","full_name":"FranxYao/Language-Model-Pretraining-for-Text-Generation","owner":"FranxYao","description":"LM pretraining for generation, reading list, resources, conference mappings. ","archived":false,"fork":false,"pushed_at":"2020-02-25T06:54:30.000Z","size":41,"stargazers_count":20,"open_issues_count":1,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-28T11:44:06.969Z","etag":null,"topics":["bert","bert-model","gpt","language-generation","language-model","pretrained-models","text-generation"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FranxYao.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-04-23T04:13:58.000Z","updated_at":"2023-07-21T06:59:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"847d53ed-de15-4e79-a7ac-5944895cdc2e","html_url":"https://github.com/FranxYao/Language-Model-Pretraining-for-Text-Generation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FranxYao%2FLanguage-Model-Pretraining-for-Text-Generation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FranxYao%2FLanguage-Model-Pretraining-for-Text-Generation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FranxYao%2FLanguage-Model-Pretraining-for-Text-Generation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FranxYao%2FLanguage-Model-Pretraining-for-Text-Generation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FranxYao","download_url":"https://codeload.github.com/FranxYao/Language-Model-Pretraining-for-Text-Generation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250297160,"owners_count":21407160,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","bert-model","gpt","language-generation","language-model","pretrained-models","text-generation"],"created_at":"2024-08-02T02:00:33.567Z","updated_at":"2025-04-22T18:30:48.841Z","avatar_url":"https://github.com/FranxYao.png","language":null,"funding_links":[],"categories":["Natural Language Processing"],"sub_categories":[],"readme":"\n# Language-Model-Pretraining-for-Text-Generation\n\nLM pretraining for generation, reading list, resources, conference mappings. \n\n![](cover.jpeg)\n\n----\n\n* Deep contextualized word representations (ELMo). NAACL 18 \n  * I just like ELMo and AI2. \n\n* BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL 19 \n\n* BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. \n  * Current SOTA on summarization \n\n* Language Models are Unsupervised Multitask Learners (GPT2)\n\n* Improving Language Understanding by Generative Pre-Training (GPT)\n\n* Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)\n\n* Unified Language Model Pre-training for Natural Language Understanding and Generation (UniLM), NeurIPS 19\n  * Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon. MSR\n  \n* Text Summarization with Pretrained Encoders. ACL 19\n  \n* Unsupervised Pretraining for Sequence to Sequence Learning, EMNLP 17 \n  * Prajit Ramachandran and Peter J. Liu and Quoc V. Le, Google Brain \n  * The LM pretraining idea, before BERT. Effective on MT and Summarization. \n\n* MASS: Masked Sequence to Sequence Pre-training for Language Generation, ICML 19 \n  * Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu, NJU and MSRA\n\n* Pre-trained Language Model Representations for Language Generation, NAACL 19\n  * Sergey Edunov, Alexei Baevski, Michael Auli, FAIR\n  * LM pretraining fine tuned on different size of MT data. The smaller the more performance gain. \n\n* Semi-Supervised Sequence Modeling with Cross-View Training. (CVT) EMNLP 18  \n  * Kevin Clark, Minh-Thang Luong, Christopher D. Manning, Quoc V. Le\n  * Semi-supervised learning with multi-task learning. MT is one application. Also improvements on other tasks \n\n* Few-shot NLG with Pre-trained Language Model\n  * Zhiyu Chen, Harini Eavani, Yinyin Liu, and William Yang Wang\n  * Extreme few-shot supervision: **50 training instances get 26 BLEU**\n\n* Cross-lingual Language Model Pretraining, Arxiv 19 \n  * Guillaume Lample, Alexis Conneau, FAIR \n  * Application to Cross-lingual classification, Unsupervised machine translation, Supervised machine translation (low-resource), Low-resource language model \n\n* BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model, Arxiv 19 \n  * Alex Wang, Kyunghyun Cho, NYU \n  * Directly sample from BERT, not as effective. \n\n* Constant-Time Machine Translation with Conditional Masked Language Models ([link](https://arxiv.org/pdf/1904.09324.pdf))\n  * Marjan Ghazvininejad, Omer Levy, Yinhan Liu and Luke Zettlemoyer\n  * Iterative decoding: generate with BERT, then polish with BERT, then continue polish\n  \n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFranxYao%2FLanguage-Model-Pretraining-for-Text-Generation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFranxYao%2FLanguage-Model-Pretraining-for-Text-Generation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFranxYao%2FLanguage-Model-Pretraining-for-Text-Generation/lists"}