{"id":13935799,"url":"https://github.com/PaddlePaddle/PALM","last_synced_at":"2025-07-19T21:30:40.995Z","repository":{"id":53482375,"uuid":"210520127","full_name":"PaddlePaddle/PALM","owner":"PaddlePaddle","description":"a Fast, Flexible, Extensible and Easy-to-use NLP Large-scale Pretraining and Multi-task Learning Framework.","archived":true,"fork":false,"pushed_at":"2021-03-29T11:40:54.000Z","size":2320,"stargazers_count":183,"open_issues_count":11,"forks_count":30,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-06-17T12:03:12.570Z","etag":null,"topics":["baidu","multi-task-learning","nlp","paddlepaddle","pretrain-model","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PaddlePaddle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-09-24T05:37:22.000Z","updated_at":"2025-02-23T11:44:50.000Z","dependencies_parsed_at":"2022-08-19T04:50:30.031Z","dependency_job_id":null,"html_url":"https://github.com/PaddlePaddle/PALM","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/PaddlePaddle/PALM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaddlePaddle%2FPALM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaddlePaddle%2FPALM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaddlePaddle%2FPALM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaddlePaddle%2FPALM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PaddlePaddle","download_url":"https://codeload.github.com/PaddlePaddle/PALM/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaddlePaddle%2FPALM/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266019657,"owners_count":23864916,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["baidu","multi-task-learning","nlp","paddlepaddle","pretrain-model","transformers"],"created_at":"2024-08-07T23:02:06.275Z","updated_at":"2025-07-19T21:30:40.989Z","avatar_url":"https://github.com/PaddlePaddle.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# PaddlePALM\n\nEnglish | [简体中文](./README_zh.md)\n\nPaddlePALM (PArallel Learning from Multi-tasks) is a fast, flexible, extensible and easy-to-use NLP large-scale pretraining and multi-task learning framework. PaddlePALM is a high level framework **aiming at fastly developing high-performance NLP models**. \n\nWith PaddlePALM, it is easy to achieve effecient exploration of robust learning of NLP models with multiple auxilary tasks. For example, based on PaddlePALM, the produced robust MRC model, [D-Net](https://github.com/PaddlePaddle/models/tree/develop/PaddleNLP/Research/MRQA2019-D-NET), has achieved **the 1st place** in [EMNLP2019 MRQA](https://mrqa.github.io) track.\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"https://tva1.sinaimg.cn/large/006tNbRwly1gbjkuuwrmlj30hs0hzdh2.jpg\" alt=\"Sample\"  width=\"300\" height=\"333\"\u003e\n\t\u003cp align=\"center\"\u003e\n\t\t\u003cem\u003eMRQA2019 Leaderboard\u003c/em\u003e\n\t\u003c/p\u003e\n\u003c/p\u003e\n\nBeyond the research scope, PaddlePALM has been applied on **Baidu Search Engine** to seek for more accurate user query understanding and answer mining, which implies the high reliability and performance of PaddlePALM.\n\n#### Features:\n\n- **Easy-to-use:** with PALM, *8 steps* to achieve a typical NLP task. Moreover, all basic components (e.g., the model backbone, dataset reader, task output head, optimizer...) have been decoupled, which allows the replacement of any component to other candidates with quite minor changes of your code. \n- **Built-in Popular NLP Backbones and Pre-trained models:** multiple state-of-the-art general purpose model architectures and pretrained models (e.g., BERT,ERNIE,RoBERTa,...) are built-in. \n- **Easy to play Multi-task Learning:** only one API is needed for jointly training of several tasks with parameters reusement.\n- **Support train/eval with Multi-GPUs:** automatically recognize and adapt to multiple gpus mode to accelerate training and inference.\n- **Pre-training friendly:** self-supervised tasks (e.g., mask language model) are built-in to facilitate pre-training. Easy to train from scratch.\n- **Easy to Customize:** support customized development of any component (e.g, backbone, task head, reader and optimizer) with reusement of pre-defined ones, which gives developers high flexibility and effeciency to adapt for diverse NLP scenes. \n\nYou can easily re-produce following competitive results with minor codes, which covers most of NLP tasks such as classification, matching, sequence labeling, reading comprehension, dialogue understanding and so on. More details can be found in `examples`.\n\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e\u003cstrong\u003eDataset\u003c/strong\u003e\n        \u003cbr\u003e\u003c/th\u003e\n      \u003cth colspan=\"2\"\u003e\u003ccenter\u003e\u003cstrong\u003echnsenticorp\u003c/strong\u003e\u003c/center\u003e\u003c/th\u003e\n      \u003cth colspan=\"2\"\u003e\u003ccenter\u003e\u003cstrong\u003eQuora Question Pairs matching\u003c/strong\u003e\u003ccenter\u003e\u003c/th\u003e\n      \u003cth colspan=\"1\"\u003e\u003cstrong\u003eMSRA-NER\u003cbr\u003e(SIGHAN2006)\u003c/strong\u003e\u003c/th\u003e\n      \u003cth colspan=\"2\"\u003e\u003cstrong\u003eCMRC2018\u003c/strong\u003e\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd rowspan=\"2\"\u003e\n        \u003cp\u003e\n          \u003cstrong\u003eMetric\u003c/strong\u003e\n          \u003cbr\u003e\u003c/p\u003e\n      \u003c/td\u003e\n      \u003ctd colspan=\"1\"\u003e\n        \u003ccenter\u003e\u003cstrong\u003eaccuracy\u003c/strong\u003e\u003c/center\u003e\n        \u003cbr\u003e\u003c/td\u003e\n      \u003ctd colspan=\"1\"\u003e\n        \u003cstrong\u003ef1-score\u003c/strong\u003e\n        \u003cstrong\u003e\u003c/strong\u003e\n        \u003cbr\u003e\u003c/td\u003e\n      \u003ctd colspan=\"1\"\u003e\n        \u003ccenter\u003e\u003cstrong\u003eaccuracy\u003c/strong\u003e\u003c/center\u003e\n        \u003cbr\u003e\u003c/td\u003e\n      \u003ctd colspan=\"1\"\u003e\n        \u003cstrong\u003ef1-score\u003c/strong\u003e\n        \u003cstrong\u003e\u003c/strong\u003e\n        \u003cbr\u003e\u003c/td\u003e\n      \u003ctd colspan=\"1\"\u003e\n        \u003cstrong\u003ef1-score\u003c/strong\u003e\n        \u003cstrong\u003e\u003c/strong\u003e\n        \u003cbr\u003e\u003c/td\u003e\n      \u003ctd colspan=\"1\"\u003e\n        \u003cstrong\u003eem\u003c/strong\u003e\n        \u003cbr\u003e\u003c/td\u003e\n      \u003ctd colspan=\"1\"\u003e\n        \u003cstrong\u003ef1-score\u003c/strong\u003e\n        \u003cbr\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd colspan=\"2\" width=\"\"\u003e\n        \u003cstrong\u003etest\u003c/strong\u003e\n        \u003cbr\u003e\u003c/td\u003e\n      \u003ctd colspan=\"2\" width=\"\"\u003e\n        \u003cstrong\u003etest\u003c/strong\u003e\n        \u003cbr\u003e\u003c/td\u003e\n      \u003ctd colspan=\"1\" width=\"\"\u003e\n        \u003cstrong\u003etest\u003c/strong\u003e\n        \u003cbr\u003e\u003c/td\u003e\n      \u003ctd colspan=\"2\" width=\"\"\u003e\n        \u003cstrong\u003edev\u003c/strong\u003e\n        \u003cbr\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003e\u003cstrong\u003eERNIE Base\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd\u003e95.8\u003c/td\u003e\n      \u003ctd\u003e95.8\u003c/td\u003e\n      \u003ctd\u003e86.2\u003c/td\u003e\n      \u003ctd\u003e82.2\u003c/td\u003e\n      \u003ctd\u003e99.2\u003c/td\u003e\n      \u003ctd\u003e64.3\u003c/td\u003e\n      \u003ctd\u003e85.2\u003c/td\u003e\n    \u003c/tr\u003e\n\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\n\n## Overview\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"https://tva1.sinaimg.cn/large/0082zybply1gbyo8d4ltoj31ag0n3tby.jpg\" alt=\"Sample\"  width=\"600px\" height=\"auto\"\u003e\n\t\u003cp align=\"center\"\u003e\n\t\t\u003cem\u003eArchitecture Diagram\u003c/em\u003e\n\t\u003c/p\u003e\n\u003c/p\u003e\n\nPaddlePALM is a well-designed high-level NLP framework. You can efficiently achieve **supervised learning, unsupervised/self-supervised learning, multi-task learning and transfer learning** with minor codes based on PaddlePALM. There are three layers in PaddlePALM architecture, i.e., component layer, trainer layer and high-level trainer layer from bottom to top. \n\nIn component layer, PaddlePALM supplies 6 **decoupled** components to achieve a NLP task. Each component contains rich `pre-defined` classes and a `Base` class. Pre-defined classes are aiming at typical NLP tasks, and the base class is to help users develop a new Class (based on pre-defined ones or from the base). \n\nThe trainer layer is to establish a computation graph with selected components and do training and predicting. The training strategy, model saving and loading, evaluation and predicting procedures are described in this layer. Noted a trainer can only process one task. \n\nThe high-level trainer layer is for complicated learning and inference strategy, e.g., multi-task learning. You can add auxilary tasks to train robust NLP models (improve test set and out-of-domain performance of a model), or jointly training multiple related tasks to gain more performance for each task.\n\n| module | illustration | \n| - | - |\n| **paddlepalm** | an open source NLP pretraining and multitask learning framework, built on paddlepaddle. |\n| **paddlepalm.reader** | a collection of elastic task-specific dataset readers. |\n| **paddlepalm.backbone** | a collection of classic NLP representation models, e.g., BERT, ERNIE, RoBERTa. |\n| **paddlepalm.head** | a collection of task-specific output layers. |\n| **paddlepalm.lr_sched** | a collection of learning rate schedualers. |\n| **paddlepalm.optimizer** | a collection of optimizers. |\n| **paddlepalm.downloader** | a download module for pretrained models with configure and vocab files. |\n| **paddlepalm.Trainer** | the core unit to start a single task training/predicting session. A trainer is to build computation graph, manage training and evaluation process, achieve model/checkpoint saving and pretrain_model/checkpoint loading.|\n| **paddlepalm.MultiHeadTrainer** | the core unit to start a multi-task training/predicting session. A MultiHeadTrainer is built based on several Trainers. Beyond the inheritance of Trainer, it additionally achieves model backbone reuse across tasks, trainer sampling for multi-task learning, and multi-head inference for effective evaluation and prediction. |\n\n\n## Installation\n\nPaddlePALM support both python2 and python3, linux and windows, CPU and GPU. The preferred way to install PaddlePALM is via `pip`. Just run following commands in your shell.\n\n```bash\npip install paddlepalm\n```\n\n### Installing via source\n\n```shell\ngit clone https://github.com/PaddlePaddle/PALM.git\ncd PALM \u0026\u0026 python setup.py install\n```\n\n### Library Dependencies\n- Python \u003e= 2.7\n- cuda \u003e= 9.0\n- cudnn \u003e= 7.0\n- PaddlePaddle \u003e= 1.7.0 (Please refer to [this](http://www.paddlepaddle.org/#quick-start) to install)\n\n\n### Downloading pretrain models\nWe incorporate many pretrained models to initialize model backbone parameters. Training big NLP model, e.g., 12-layer transformers, with pretrained models is practically much more effective than that with randomly initialized parameters. To see all the available pretrained models and download, run following code in python interpreter (input command `python` in shell):\n\n```python\n\u003e\u003e\u003e from paddlepalm import downloader\n\u003e\u003e\u003e downloader.ls('pretrain')\nAvailable pretrain items:\n  =\u003e RoBERTa-zh-base\n  =\u003e RoBERTa-zh-large\n  =\u003e ERNIE-v2-en-base\n  =\u003e ERNIE-v2-en-large\n  =\u003e XLNet-cased-base\n  =\u003e XLNet-cased-large\n  =\u003e ERNIE-v1-zh-base\n  =\u003e ERNIE-v1-zh-base-max-len-512\n  =\u003e BERT-en-uncased-large-whole-word-masking\n  =\u003e BERT-en-cased-large-whole-word-masking\n  =\u003e BERT-en-uncased-base\n  =\u003e BERT-en-uncased-large\n  =\u003e BERT-en-cased-base\n  =\u003e BERT-en-cased-large\n  =\u003e BERT-multilingual-uncased-base\n  =\u003e BERT-multilingual-cased-base\n  =\u003e BERT-zh-base\n\n\u003e\u003e\u003e downloader.download('pretrain', 'BERT-en-uncased-base', './pretrain_models')\n...\n```\n\n\n## Usage\n\n#### Quick Start\n\n8 steps to start a typical NLP training task.\n\n1. use `paddlepalm.reader` to create a *reader* for dataset loading and input features generation, then call `reader.load_data` method to load your training data.\n2. use `paddlepalm.backbone` to create a model *backbone* to extract text features (e.g., contextual word embedding, sentence embedding).\n3. register your *reader* with your *backbone* through `reader.register_with` method. After this step, your reader is able to yield input features used by backbone.\n4. use `paddlepalm.head` to create a task output *head*. This head can provide task loss for training and predicting results for model inference.\n5. create a task *trainer* with `paddlepalm.Trainer`, then build forward graph with backbone and task head (created in step 2 and 4) through `trainer.build_forward`.\n6. use `paddlepalm.optimizer` (and `paddlepalm.lr_sched` if is necessary) to create a *optimizer*, then build backward through `trainer.build_backward`.\n7. fit prepared reader and data (achieved in step 1) to trainer with `trainer.fit_reader` method.\n8. load pretrain model with `trainer.load_pretrain`, or load checkpoint with `trainer.load_ckpt` or nothing to do for training from scratch, then do training with `trainer.train`.\n\nFor more implementation details, see following demos: \n\n- [Sentiment Classification](https://github.com/PaddlePaddle/PALM/tree/master/examples/classification)\n- [Question Pairs matching](https://github.com/PaddlePaddle/PALM/tree/master/examples/matching)\n- [Named Entity Recognition](https://github.com/PaddlePaddle/PALM/tree/master/examples/tagging)\n- [SQuAD-like Machine Reading Comprehension](https://github.com/PaddlePaddle/PALM/tree/master/examples/mrc).\n\n\n#### Multi-task Learning\nTo run with multi-task learning mode:\n\n1. repeatedly create components (i.e., reader, backbone and head) for each task followed with step 1~5 above. \n2. create empty trainers (each trainer is corresponded to one task) and pass them to create a `MultiHeadTrainer`. \n3. build multi-task forward graph with `multi_head_trainer.build_forward` method.\n4. use `paddlepalm.optimizer` (and `paddlepalm.lr_sched` if is necessary) to create a *optimizer*, then build backward through `multi_head_trainer.build_backward`.\n5. fit all prepared readers and data to multi_head_trainer with `multi_head_trainer.fit_readers` method.\n6. load pretrain model with `multi_head_trainer.load_pretrain`, or load checkpoint with `multi_head_trainer.load_ckpt` or nothing to do for training from scratch, then do training with `multi_head_trainer.train`.\n\nThe save/load and predict operations of a multi_head_trainer is the same as a trainer.\n\nFor more implementation details with `multi_head_trainer`, see\n\n- [ATIS: joint training of dialogue intent recognition and slot filling](https://github.com/PaddlePaddle/PALM/tree/master/examples/multi-task)\n\n#### Save models\n\nTo save models/checkpoints and logs during training, just call `trainer.set_saver` method. More implementation details see [this](https://github.com/PaddlePaddle/PALM/tree/master/examples).\n\n#### Evaluation/Inference\nTo do predict/evaluation after a training stage, just create another three reader, backbone and head instance with `phase='predict'` (repeat step 1~4 above). Then do predicting with `predict` method in trainer (no need to create another trainer). More implementation details see [this](https://github.com/PaddlePaddle/PALM/tree/master/examples/predict).\n\nIf you want to do evaluation during training process, use `trainer.train_one_step()` instead of `trainer.train()`. The `trainer.train_one_step(batch)` achieves to train only one step, thus you can insert evaluation code into any point of training process. The argument `batch` can be fetched from `trainer.get_one_batch`.\n\nPaddlePALM also supports multi-head inference, please reference `examples/multi-task/joint_predict.py`.\n\n#### Play with Multiple GPUs\nIf there exists multiple GPUs in your environment, you can control the number and index of these GPUs through the environment variable [CUDA_VISIBLE_DEVICES](https://devblogs.nvidia.com/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices/). For example, if 4 GPUs in your enviroment, indexed with 0,1,2,3, you can run with GPU2 only with following commands\n\n```shell\nCUDA_VISIBLE_DEVICES=2 python run.py\n```\n\nMultiple GPUs should be seperated with `,`. For example, running with GPU2 and GPU3, following commands is refered:\n\n```shell\nCUDA_VISIBLE_DEVICES=2,3 python run.py\n```\n\nOn multi-gpu mode, PaddlePALM will automatically split each batch onto the available cards. For example, if the `batch_size` is set 64, and there are 4 cards visible for PaddlePALM, then the batch_size in each card is actually 64/4=16. Therefore, when running with multiple cards, **you need to ensure that the set batch_size can be divided by the number of cards.**\n\n## License\n\nThis tutorial is contributed by [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) and licensed under the [Apache-2.0 license](https://github.com/PaddlePaddle/models/blob/develop/LICENSE).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPaddlePaddle%2FPALM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPaddlePaddle%2FPALM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPaddlePaddle%2FPALM/lists"}