{"id":27344438,"url":"https://github.com/ML-GSAI/LLaDA","last_synced_at":"2025-04-12T17:06:23.208Z","repository":{"id":277488706,"uuid":"932356580","full_name":"ML-GSAI/LLaDA","owner":"ML-GSAI","description":"Official PyTorch implementation for \"Large Language Diffusion Models\"","archived":false,"fork":false,"pushed_at":"2025-04-07T09:01:03.000Z","size":19616,"stargazers_count":1405,"open_issues_count":39,"forks_count":106,"subscribers_count":34,"default_branch":"main","last_synced_at":"2025-04-07T10:21:49.657Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ML-GSAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-13T19:28:36.000Z","updated_at":"2025-04-07T09:11:07.000Z","dependencies_parsed_at":"2025-02-14T07:29:47.435Z","dependency_job_id":"df4fe1e2-129c-419e-a4b4-5b395c361f91","html_url":"https://github.com/ML-GSAI/LLaDA","commit_stats":null,"previous_names":["ml-gsai/llada"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ML-GSAI%2FLLaDA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ML-GSAI%2FLLaDA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ML-GSAI%2FLLaDA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ML-GSAI%2FLLaDA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ML-GSAI","download_url":"https://codeload.github.com/ML-GSAI/LLaDA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248602306,"owners_count":21131615,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-12T17:02:13.366Z","updated_at":"2025-04-12T17:06:23.190Z","avatar_url":"https://github.com/ML-GSAI.png","language":"Python","funding_links":[],"categories":["others","A01_文本生成_文本对话"],"sub_categories":["大语言对话模型及数据"],"readme":"# Large Language Diffusion Models\n[![arXiv](https://img.shields.io/badge/arXiv-2502.09992-red.svg)](https://arxiv.org/abs/2502.09992)\n[![deploy](https://img.shields.io/badge/Hugging%20Face%20-LLaDA_Base%20-FFEB3B)](https://huggingface.co/GSAI-ML/LLaDA-8B-Base)\n[![deploy](https://img.shields.io/badge/Hugging%20Face%20-LLaDA_Instruct%20-FFEB3B)](https://huggingface.co/GSAI-ML/LLaDA-8B-Instruct)\n[![deploy](https://img.shields.io/badge/🤗%20Hugging%20Face%20-Spaces%20demo%20-blue)](https://huggingface.co/spaces/multimodalart/LLaDA)\n[![deploy](https://img.shields.io/badge/Zhihu-知乎-blue)](https://zhuanlan.zhihu.com/p/24214732238)\n\nWe introduce LLaDA (\u003cb\u003eL\u003c/b\u003earge \u003cb\u003eLa\u003c/b\u003enguage \u003cb\u003eD\u003c/b\u003eiffusion with m\u003cb\u003eA\u003c/b\u003esking), a diffusion model with an unprecedented 8B scale, trained entirely from scratch, \nrivaling LLaMA3 8B in performance.\n\n\u003cdiv style=\"display: flex; justify-content: center; flex-wrap: wrap;\"\u003e\n    \u003cimg src=\"./imgs/LLaDA_vs_LLaMA.svg\" style=\"width: 45%\" /\u003e\n    \u003cimg src=\"./imgs/LLaDA_vs_LLaMA_chat.svg\" style=\"width: 46%\" /\u003e\n\u003c/div\u003e\n\n\n## Inference\nThe [LLaDA-8B-Base](https://huggingface.co/GSAI-ML/LLaDA-8B-Base) and [LLaDA-8B-Instruct](https://huggingface.co/GSAI-ML/LLaDA-8B-Instruct) are uploaded\nin Huggingface. Please first install `transformers==4.38.2` and employ the [transformers](https://huggingface.co/docs/transformers/index) to load.\n\n```angular2html\nfrom transformers import AutoModel, AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained('GSAI-ML/LLaDA-8B-Base', trust_remote_code=True)\nmodel = AutoModel.from_pretrained('GSAI-ML/LLaDA-8B-Base', trust_remote_code=True, torch_dtype=torch.bfloat16)\n```\n\nWe provide `get_log_likelihood()` and `generate()` functions in `get_log_likelihood.py` \nand `generate.py` respectively, for conditional likelihood evaluation and conditional generation.\n\nYou can directly run `python chat.py` to have multi-round conversations with LLaDA-8B-Instruct.\n\nIn addition, please refer to our paper and [GUIDELINES.md](GUIDELINES.md) for more details about the inference methods.\n\n## Gradio demo \nThank you very much to [apolinário](https://github.com/apolinario) for helping us create this amazing demo!\n\nFirst, install [Gradio](https://www.gradio.app) `pip install gradio`, and then you can directly run `python app.py`\n\n\u003cdiv style=\"display: flex; justify-content: center; flex-wrap: wrap;\"\u003e\n    \u003cimg src=\"./imgs/example_gradio.gif\" style=\"width: 80%\" /\u003e\n\u003c/div\u003e\n\n## Pre-training and Supervised Fine-Tuning\n\nWe will not provide the training framework and data as most open-source LLMs do.\n\nHowever, the pre-training and Supervised Fine-Tuning of LLaDA are straightforward. If \nyou have a codebase for training an autoregressive model, you can modify it to \nadapt to LLaDA with just a few lines of code.\n\nWe provide guidelines for the pre-training and SFT of LLaDA in [GUIDELINES.md](GUIDELINES.md). \nYou can also refer to [SMDM](https://github.com/ML-GSAI/SMDM), which has a similar training process to LLaDA \nand has open-sourced the training framework.\n\n## Evaluation\n\nWe use two evaluation methods: conditional likelihood estimation and conditional generation. For the base model, conditional likelihood estimation is applied to specific metrics and conditional generation to the rest. For the Instruct model, conditional generation is used for all metrics.\n\nWe implement conditional likelihood estimation using the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) library, while conditional generation is performed with an internal library, as lm-evaluation-harness lacks support for certain metrics (i.e., HumanEval-FIM). \n\nPlease refer to Appendix B.5. of our [paper](https://arxiv.org/abs/2502.09992) for all evaluation details.\n\nWe provide the code for evaluation using the open-source library lm-evaluation-harness. To begin, please install `lm_eval==0.4.5` and refer to `evaluation/eval.sh` for the specific commands.\n\n\n## FAQ\nHere, we address some common questions about LLaDA.\n\n### 0. How do I train my own LLaDA?\nPlease refer to [GUIDELINES.md](GUIDELINES.md) for the guidelines. \nYou can also refer to [SMDM](https://github.com/ML-GSAI/SMDM), which follows the same training \nprocess as LLaDA and has open-sourced its code.\n\n\n### 1. What is the difference between LLaDA and BERT?\n\nOur motivation is not to improve BERT, nor to apply image generation methods like [MaskGIT](https://arxiv.org/abs/2202.04200) \nto text. **Our goal is to explore a theoretically complete language modeling approach — masked diffusion models.** \nDuring this process, we simplified the approach and discovered that the loss function of masked diffusion models \nis related to the loss functions of BERT and MaskGIT. You can find our theoretical research process in Question 7.\n\nSpecifically, LLaDA employs a masking ratio that varies randomly between 0 and 1, while BERT uses \na fixed ratio. This subtle difference has significant implications. **The training\nobjective of LLaDA is an upper bound on the negative log-likelihood of the model \ndistribution, making LLaDA a generative model.** This enables LLaDA to naturally \nperform in-context learning, instruction-following, and ensures Fisher consistency \nfor scalability with large datasets and models. You can also find a direct answer \nto this question in Section 2.1 of our paper.\n\n\n### 2. What is the relationship between LLaDA and Transformer?\nNetwork structure and probabilistic modeling are two distinct approaches that collectively form the \nfoundation of language models. LLaDA, like GPT, adopts the \nTransformer architecture. The key difference lies in the probabilistic modeling approach: GPT \nutilizes an autoregressive next-token prediction method, \nwhile LLaDA employs a diffusion model for probabilistic modeling.\n\n\n### 3. What is the sampling efficiency of LLaDA?\nCurrently, LLaDA's sampling speed is slower than the autoregressive baseline for three reasons: \n1. LLaDA samples with a fixed context length;\n2. LLaDA cannot yet leverage techniques like KV-Cache;\n3. LLaDA achieves optimal performance when the number of sampling steps equals the response length.\nReducing the number of sampling steps leads to a decrease in performance, as detailed in Appendix B.4 \nand Appendix B.6 of our paper.\n\nIn this work, we aim to explore the upper limits of LLaDA's capabilities, **challenging the assumption \nthat the key LLM abilities are inherently tied to autoregressive models**. We will continue \nto optimize its efficiency in the future. We believe this research approach is reasonable, \nas verifying the upper limits of diffusion language models' capabilities will provide us with\nmore resources and sufficient motivation to optimize efficiency.\n\nRecall the development of diffusion models for images, from [DDPM](https://arxiv.org/abs/2006.11239) \nto the [Consistency model](https://arxiv.org/pdf/2410.11081), where sampling speed accelerated nearly \n1000 times over the course of 4 years. **We believe there is significant room for optimization in LLaDA's \nsampling efficiency as well**. Current solutions, including semi-autoregressive sampling (as \ndetailed in [GUIDELINES.md](GUIDELINES.md)), can mitigate the fixed context length issue, and \n[consistency distillation](https://arxiv.org/pdf/2502.05415) can reduce the number of sampling steps.\n\n\n### 4. What is the training stability of LLaDA?\nFor details on the pre-training process of LLaDA, please refer to Section 2.2 of our paper. \nDuring the total pre-training on 2.3T tokens, we encountered a training crash (loss becoming NaN) \nonly once at 1.2T tokens. Our solution was to resume checkpoint and reduce \nthe learning rate from 4e-4 to 1e-4.\n\n\n### 5. Why is the final answer \"72\" generated earlier than the intermediate calculation step (e.g., 12 × 4 = 48) in Tab4?\n\n**The mask predictor has successfully predicted the reasoning process. However, during the \nremasking process, the reasoning steps are masked out again.** As shown in the figure \nbelow, the non-white background represents the model's generation process, while the \nwhite-background boxes indicate the predictions made by the mask predictor at each step. \nWe adopt a randomly remasking strategy.\n\n\u003cdiv style=\"display: flex; justify-content: center; flex-wrap: wrap;\"\u003e\n    \u003cimg src=\"./imgs/diff_remask.gif\" style=\"width: 80%\" /\u003e\n\u003c/div\u003e\n\n### 6. Why does LLaDA answer 'Bailing' when asked 'Who are you'?\nThis is because our pre-training and SFT data were designed for training an autoregressive model, \nwhereas LLaDA directly utilizes data that contains identity markers.\n\n\n### 7. Our journey in developing LLaDA?\nLLaDA is built upon our two prior works, [RADD](https://arxiv.org/abs/2406.03736) and \n[SMDM](https://arxiv.org/abs/2410.18514). \n\nRADD demonstrated that the **training objective of LLaDA serves as an upper bound on the negative \nlog-likelihood** of the model’s distribution, a conclusion also supported by [MD4](https://arxiv.org/abs/2406.04329) \nand [MDLM](https://arxiv.org/abs/2406.07524). \nFurthermore, RADD was the first to theoretically prove that **masked diffusion models do not require time t \nas an input to Transformer**. This insight provides the theoretical \njustification for LLaDA’s unmodified use of the Transformer architecture. Lastly, \nRADD showed that **the training objective of masked diffusion models is equivalent to that of \nany-order autoregressive models**, offering valuable insights into how masked diffusion models can \novercome the reversal curse.\n\nSMDM introduces the first **scaling law** for masked diffusion models and demonstrates that, with the \nsame model size and training data, masked diffusion models can achieve downstream benchmark results \non par with those of autoregressive models. Additionally, SMDM presents a simple, **unsupervised \nclassifier-free guidance** method that greatly improves downstream benchmark performance, which has \nbeen adopted by LLaDA.\n\n\n## Citation\n\n```bibtex\n@article{nie2025large,\n  title={Large Language Diffusion Models},\n  author={Nie, Shen and Zhu, Fengqi and You, Zebin and Zhang, Xiaolu and Ou, Jingyang and Hu, Jun and Zhou, Jun and Lin, Yankai and Wen, Ji-Rong and Li, Chongxuan},\n  journal={arXiv preprint arXiv:2502.09992},\n  year={2025}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FML-GSAI%2FLLaDA","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FML-GSAI%2FLLaDA","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FML-GSAI%2FLLaDA/lists"}