{"id":13441150,"url":"https://github.com/RUCAIBox/LLMBox","last_synced_at":"2025-03-20T11:35:35.193Z","repository":{"id":226839880,"uuid":"709815787","full_name":"RUCAIBox/LLMBox","owner":"RUCAIBox","description":"A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation.","archived":false,"fork":false,"pushed_at":"2024-08-29T12:01:16.000Z","size":23178,"stargazers_count":548,"open_issues_count":1,"forks_count":71,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-08-29T13:29:01.072Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RUCAIBox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-25T13:05:19.000Z","updated_at":"2024-08-29T12:01:20.000Z","dependencies_parsed_at":"2024-03-17T04:05:17.043Z","dependency_job_id":"30042252-5a6f-495b-bfa1-c02023516f48","html_url":"https://github.com/RUCAIBox/LLMBox","commit_stats":null,"previous_names":["rucaibox/llmbox"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCAIBox%2FLLMBox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCAIBox%2FLLMBox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCAIBox%2FLLMBox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCAIBox%2FLLMBox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RUCAIBox","download_url":"https://codeload.github.com/RUCAIBox/LLMBox/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221752332,"owners_count":16874963,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:01:30.481Z","updated_at":"2025-03-20T11:35:35.161Z","avatar_url":"https://github.com/RUCAIBox.png","language":"Python","funding_links":[],"categories":["Python","A01_文本生成_文本对话","微调 Fine-Tuning","Industry Strength Natural Language Processing"],"sub_categories":["大语言对话模型及数据"],"readme":"**LLMBox** | [Training](https://github.com/RUCAIBox/LLMBox/tree/main/training) | [Utilization](https://github.com/RUCAIBox/LLMBox/tree/main/utilization)\n\n# LLMBox\n\nLLMBox is a comprehensive library for implementing LLMs, including **a unified training pipeline** and **comprehensive model evaluation**. LLMBox is designed to be a one-stop solution for training and utilizing LLMs. Through a practical library design, we achieve a high-level of **flexibility** and **efficiency** in both training and utilization stages.\n\n\u003cimg style=\"display: block; margin: 25 auto;\" src=\"docs/assets/llmbox.png\" alt=\"\" /\u003e\n\n\n## Key Features\n\nTraining\n\n- **Diverse training strategies:** We support multiple training strategies, including Supervised Fine-tuning (`SFT`), Pre-training (`PT`), `PPO` and `DPO`.\n- **Comprehensive SFT datasets:** We support 9 SFT datasets as the inputs for training.\n- **Tokenizer Vocabulary Merging:** We support the tokenizer merging function to expand the vocabulary.\n- **Data Construction Strategies:** We currently support merging multiple datasets for training. `Self-Instruct` and `Evol-Instruct` are also available to process the dataset.\n- **Parameter Efficient Fine-Tuning:** `LoRA` and `QLoRA` are supported in SFT or PT.\n- **Efficient Training:** We support [`Flash Attention`](https://github.com/Dao-AILab/flash-attention) and `Deepspeed` for efficient training.\n\nUtilization\n\n- **Blazingly Fast:** By managing the KV Cache of prefixes or using [`vLLM`](https://github.com/vllm-project/vllm), we can speed up local inference by up to 6x 🚀.\n- **Comprehensive Evaluation:** 59+ commonly used [datasets](https://github.com/RUCAIBox/LLMBox/blob/main/docs/utilization/supported-datasets.md) and benchmarks in evaluating LLMs.\n- **Evaluation Methods:** 📏 Accurately reproduce results from original papers of OpenAI, LLaMA, Mistral, and other models.\n- **In-Context Learning:** We support various ICL strategies, including [`KATE`](https://aclanthology.org/2022.deelio-1.10/), [`GlobalE`](https://aclanthology.org/2022.acl-long.556/), and [`APE`](https://arxiv.org/abs/2211.01910).\n- **Chain-of-Thought:** For some datasets, we support three types of CoT evaluation: `base`, [`least-to-most`](https://arxiv.org/abs/2205.10625), and [`pal`](https://arxiv.org/abs/2211.10435).\n- **Quantization:** BitsAndBytes and GPTQ quantization are supported.\n- **Easy To Use**: Detailed results are provided for users to debug or integrate new models/datasets/cot.\n\n\n## Documentations\n\nSee [documentations](https://github.com/RUCAIBox/LLMBox/tree/main/docs) for more details.\n\n## Quick Start\n\n### Install\n\n```bash\ngit clone https://github.com/RUCAIBox/LLMBox.git \u0026\u0026 cd LLMBox\npip install -r requirements.txt\n```\n\nIf you are only evaluating the OpenAI (or OpenAI compatible like DeepSeek, Perplexity) models, you can install the minimal requirements `requirements-openai.txt`.\n\nFor installation problem, see [trouble shooting](https://github.com/RUCAIBox/LLMBox/tree/main/docs/trouble_shooting/vllm_no_module_name_packaging.md).\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eUpdate LLMBox\u003c/b\u003e\u003c/summary\u003e\n\nCurrently, you can simply pull the latest repository from GitHub to update LLMBox.\n\n```bash\ngit pull\n```\n\nIf you are facing a merge conflict, please try to drop, stash, or commit your local changes first.\n\n```bash\ngit checkout local_changes \u0026\u0026 git add -p \u0026\u0026 git commit -m \"local changes\"\ngit checkout main\ngit pull\n```\n\nThe above commands show how to commit your local changes to a new branch, and then update the LLMBox.\n\n\u003c/details\u003e\n\n### Quick Start with Training\n\nYou can start with training a SFT model based on LLaMA-2 (7B) with deepspeed3:\n\n```bash\ncd training\nbash download.sh\nbash bash/run_ds3.sh\n```\n\n### Quick Start with Utilization\n\nTo utilize your model, or evaluate an existing model, you can run the following command:\n\n```bash\npython inference.py -m gpt-3.5-turbo -d copa  # --num_shot 0 --model_type chat\n```\n\nThis is default to run the OpenAI GPT 3.5 turbo model on the CoPA dataset in a zero-shot manner.\n\n\n## Training\n\nLLMBox Training supports various training strategies and dataset construction strategies, along with some efficiency-improving modules. You can train your model with the following command:\n\n```bash\npython train.py \\\n    --model_name_or_path meta-llama/Llama-2-7b-hf \\\n    --data_path data/ \\\n    --dataset alpaca_data_1k.json \\\n    --output_dir $OUTPUT_DIR \\\n    --num_train_epochs 2 \\\n    --per_device_train_batch_size 8 \\\n    --gradient_accumulation_steps 2 \\\n    --save_strategy \"epoch\" \\\n    --save_steps 2 \\\n    --save_total_limit 2 \\\n    --learning_rate 1e-5 \\\n    --lr_scheduler_type \"constant\"\n```\n\nAlternatively, you can use the following preset bash scripts to train your model:\n\n### Merging Tokenizer\n\nIf you want to pre-train your models on corpora with languages or tokens not well-supported in original language mdoels(e.g., LLaMA), we provide the tokenizer merging function to expand the vocabulary based on the corpora by using [sentencepiece](https://github.com/google/sentencepiece). You can check [merge_tokenizer.py](training/merge_tokenizer.py) for detailed information. Please follow the guide in [Pre-train](https://github.com/RUCAIBox/LLMBox/tree/main/training#2-continual-pre-training-with-your-own-corpora).\n\n```bash\nbash bash/run_7b_pt.sh\n```\n\n### Merging Datasets\n\nIf you want to train your models with a mix of multiple datasets, you can pass a list of dataset files or names to LLMBox. LLMBox will transfer each file or name into a PTDataset or SFTDataset, and merge them together to construct a combined dataset. You can also set the merging ratio of each dataset by passing a list of floats to LLMBox. Please follow the guide in [Merge Dataset](https://github.com/RUCAIBox/LLMBox/tree/main/training#3-merging-different-datasets-with-designated-ratios-for-training).\n\n```bash\nbash bash/run_7b_hybrid.sh\n```\n\n### Self-Instruct and Evol-Instruct\n\nSince manually creating instruction data of high qualities to train the model is very time-consuming and labor-intensive, Self-Instruct and Evol-Instruct are proposed to create large amounts of instruction data with varying levels of complexity using LLM instead of humans. LLMBox support both Self-Instruct and Evol-Instruct to augment or enhance the input data files. Please follow the guide in [Self-Insturct and Evol-Instruct](https://github.com/RUCAIBox/LLMBox/tree/main/training#8-self-instruct-and-evol-instruct-for-generation-instructions)\n\n```bash\npython self_instruct/self_instruct.py --seed_tasks_path=seed_tasks.jsonl\n```\n\nFor more details, view the [training](https://github.com/RUCAIBox/LLMBox/tree/main/training) documentation.\n\n## Utilization\n\nWe provide a broad support on Huggingface models (e.g. `LLaMA-3`, `Mistral`, or the model you are building on), OpenAI, Anthropic, QWen and other OpenAI-compatible models for further utilization. Full list of model backends: [here](https://github.com/RUCAIBox/LLMBox/tree/main/utilization#supported-models).\n\nCurrently a total of 59+ commonly used datasets are supported, including: `HellaSwag`, `MMLU`, `GSM8K`, `GPQA`, `AGIEval`, `CEval`, and `CMMLU`. Full list of datasets: [here](https://github.com/RUCAIBox/LLMBox/blob/main/docs/utilization/supported-datasets.md).\n\n```bash\nCUDA_VISIBLE_DEVICES=0 python inference.py \\\n  -m llama-2-7b-hf \\\n  -d mmlu agieval:[English] \\\n  --model_type chat \\\n  --num_shot 5 \\\n  --ranking_type ppl_no_option\n```\n\n- 🔥 Recently supported datasets: `imbue_code`, `imbue_public`, and `imbue_private`.\n\n- 🔥 See [benchmarking LLaMA3](https://github.com/RUCAIBox/LLMBox/blob/main/docs/utilization/benchmarking_llama3.md) for more examples.\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd colspan=4 align=\"center\"\u003e\u003cb\u003ePerformance\u003c/b\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd rowspan=2\u003e\u003cb\u003eModel\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ccode\u003eget_ppl\u003c/code\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ccode\u003eget_prob\u003c/code\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ccode\u003egeneration\u003c/code\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003cb\u003eHellaswag (0-shot)\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cb\u003eMMLU (5-shot)\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cb\u003eGSM (8-shot)\u003c/b\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003cb\u003eGPT-3.5 Turbo\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e79.98\u003c/td\u003e\n        \u003ctd\u003e69.25\u003c/td\u003e\n        \u003ctd\u003e75.13\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003cb\u003eLLaMA-2 (7B)\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e76\u003c/td\u003e\n        \u003ctd\u003e45.95\u003c/td\u003e\n        \u003ctd\u003e14.63\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n### Efficient Evaluation\n\nWe by default enable prefix caching for efficient evaluation. vLLM is also supported.\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd colspan=6 align=\"center\"\u003e\u003cb\u003eTime\u003c/b\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd rowspan=2\u003e\u003cb\u003eModel\u003c/b\u003e\u003c/td\u003e\n        \u003ctd rowspan=2\u003e\u003cb\u003eEfficient Method\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ccode\u003eget_ppl\u003c/code\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ccode\u003eget_prob\u003c/code\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ccode\u003egeneration\u003c/code\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003cb\u003eHellaswag (0-shot)\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cb\u003eMMLU (5-shot)\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cb\u003eGSM (8-shot)\u003c/b\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd rowspan=3\u003e\u003cb\u003eLLaMA-2 (7B)\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cb\u003eVanilla\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e0:05:32\u003c/td\u003e\n        \u003ctd\u003e0:18:30\u003c/td\u003e\n        \u003ctd\u003e2:10:27\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003cb\u003evLLM\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e0:06:37\u003c/td\u003e\n        \u003ctd\u003e0:14:55\u003c/td\u003e\n        \u003ctd\u003e0:03:36\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003cb\u003ePrefix Caching\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e0:05:48\u003c/td\u003e\n        \u003ctd\u003e0:05:51\u003c/td\u003e\n        \u003ctd\u003e0:17:13\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\nYou can also use the following command to use vllm:\n\n\n```bash\npython inference.py -m ../Llama-2-7b-hf -d mmlu:abstract_algebra,anatomy --vllm True  # --prefix_caching False --flash_attention False\n```\n\nTo evaluate with quantization, you can use the following command:\n\n```python\npython inference.py -m model -d dataset --load_in_4bits  # --load_in_8_bits or --gptq\n```\n\n### Evaluation Method\n\nVarious types of evaluation methods are supported:\n\n\u003c/br\u003e\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003cb\u003eDataset\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cb\u003eEvaluation Method\u003c/b\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cb\u003eInstruction\u003c/b\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003cp\u003e\u003cb\u003eGeneration\u003c/b\u003e\u003c/p\u003e\n        \u003cp\u003e\u003cpre\u003e\u003ccode\u003e{\n  \"question\":\n    \"when was ...\",\n  \"answer\": [\n    '14 December 1972',\n    'December 1972'\n  ]\n}\u003c/code\u003e\u003c/pre\u003e\u003c/p\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cp\u003e\u003ccode\u003egeneration\u003c/code\u003e\u003c/p\u003e\u003cp\u003eGenerate based on the source text\u003c/p\u003e\u003cp\u003eExample: \u003ca href='https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/eval_details.md#arc-challenge'\u003eARC-Challenge\u003c/a\u003e (extract choice characters like A, B, C, D from model generation), \u003ca href='https://github.com/meta-llama/llama3/blob/bb55334adcedfa9f5da66d2e1ed64e6f3dbd82ed/eval_details.md#gsm8k'\u003eGSM8K\u003c/a\u003e, \u003ca href='https://github.com/meta-llama/llama3/blob/bb55334adcedfa9f5da66d2e1ed64e6f3dbd82ed/eval_details.md#humaneval'\u003eHumanEval\u003c/a\u003e\u003c/p\u003e\u003cp\u003eNotes: \u003ccode\u003evLLM\u003c/code\u003e generally generates faster than \u003ccode\u003etransformers\u003c/code\u003e but has different default parameters. Users may need to adjust sampling parameters like \u003ccode\u003etemperature\u003c/code\u003e and \u003ccode\u003elength_penalty\u003c/code\u003e for optimal results.\u003c/p\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cp style=\"text-align: center;\"\u003e\u003ccode\u003egeneration\u003c/code\u003e\u003c/p\u003e\u003cp\u003e\u003cpre\u003e\u003ccode\u003eQ: When was ...?\nA: ________\u003c/code\u003e\u003c/pre\u003e\u003c/p\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd rowspan=3\u003e\u003cp\u003e\u003cb\u003eMultipleChoice\u003c/b\u003e\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003e{\n  \"question\":\n    \"What is the ...?\",\n  \"choices\": [\n    \"The first\",\n    \"The second\",\n    ...\n  ],\n  \"answer\": 3\n}\u003c/code\u003e\u003c/pre\u003e\u003c/td\u003e\n        \u003ctd rowspan=2\u003e\u003cp\u003e\u003ccode\u003eget_ppl\u003c/code\u003e\u003c/p\u003e\u003cp\u003eCalculate perplexity of the option text based on the source text (i.e. compute log-likelihood over the suffix)\u003c/p\u003e\u003cp\u003eExample: \u003ca href='https://github.com/meta-llama/llama3/blob/bb55334adcedfa9f5da66d2e1ed64e6f3dbd82ed/eval_details.md#winogrande'\u003eWinoGrande\u003c/a\u003e, \u003ca href='https://github.com/meta-llama/llama3/blob/bb55334adcedfa9f5da66d2e1ed64e6f3dbd82ed/eval_details.md#boolq'\u003eBoolQ\u003c/a\u003e\u003c/p\u003e\u003cp\u003eNotes: Some datasets, such as ARC, OpenbookQA, and RACE use \u003ca href='https://arxiv.org/pdf/2005.14165'\u003enormalized accuracy\u003c/a\u003e when evaluated with the \u003ccode\u003eget_ppl\u003c/code\u003e method.\u003c/p\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cp style=\"text-align: center;\"\u003e\u003ccode\u003eppl_no_option\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003e\u003cpre\u003e\u003ccode\u003eQ: What is ...?\nA: The first\n   └--ppl--┘\u003c/code\u003e\u003c/pre\u003e\u003c/p\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003cp style=\"text-align: center;\"\u003e\u003ccode\u003eppl\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003e\u003cpre\u003e\u003ccode style=\"border-style: solid;\"\u003eQ: What is ...?\nA. The first\nB. The second\nC. ...\nA: A. The first\n   └----ppl---┘\u003c/code\u003e\u003c/pre\u003e\u003c/p\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003cp\u003e\u003ccode\u003eget_prob\u003c/code\u003e\u003c/p\u003e\u003cp\u003eGet the probability of each option label (i.e. over choice characters)\u003c/p\u003e\u003cp\u003eExample: \u003ca href='https://github.com/meta-llama/llama3/blob/bb55334adcedfa9f5da66d2e1ed64e6f3dbd82ed/eval_details.md#mmlu'\u003eMMLU\u003c/a\u003e, \u003ca href='https://github.com/meta-llama/llama3/blob/bb55334adcedfa9f5da66d2e1ed64e6f3dbd82ed/eval_details.md#arc-challenge'\u003eARC-Challange\u003c/a\u003e\u003c/p\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cp style=\"text-align: center;\"\u003e\u003ccode\u003eprob\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003e\u003cpre\u003e\u003ccode\u003eQ: What is ...?\nA. The first\nB. The second\nC. ...\nA: _\n   └→ [A B C D]\u003c/code\u003e\u003c/pre\u003e\u003c/p\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\nYou can find more evaluation details for each dataset at [supported datasets](https://github.com/RUCAIBox/LLMBox/blob/main/docs/utilization/supported-datasets.md).\n\nYou can use `--instruction` to pass a jinja template to override the default instruction.\n\nBy default, we use the `get_ppl` method with `ppl_no_option` ranking type for `MultipleChoiceDataset` and the `generation` method for `GenerationDataset`. You can also use the following command to use the `get_prob` method or `ppl` variant of `get_ppl` for MultipleChoiceDataset:\n\n```python\npython inference.py -m model -d dataset --ranking_type prob  # or ppl\n```\n\nWe also support In-Context Learning and Chain-of-Thought evaluation for some datasets:\n\n```python\npython inference.py -m model -d dataset --kate  # --globale or --ape\npython inference.py -m model -d dataset --cot least_to_most  # --base or --pal\n```\n\nFor a more detailed instruction on model utilization, view the [utilization](https://github.com/RUCAIBox/LLMBox/tree/main/utilization) documentation.\n\nFor a full list of evaluation results, see our paper [LLMBox: A Comprehensive Library for Large Language Models](https://aclanthology.org/2024.acl-demos.37.pdf).\n\n## Contributing\n\nPlease let us know if you encounter a bug or have any suggestions by [filing an issue](https://github.com/RUCAIBox/LLMBox/issues).\n\nWe welcome all contributions from bug fixes to new features and extensions.\n\nWe expect all contributions discussed in the issue tracker and going through PRs.\n\nFor more details, view the [CONTRIBUTING](https://github.com/RUCAIBox/LLMBox/tree/main/docs/CONTRIBUTING.md) documentation.\n\n---\n\nWe thank the following contributors for their contributions to LLMBox:\n\n- [@xansar](https://github.com/xansar) for fixing multiple complex issues like batch sampler and self-consistency.\n\n\n## The Team\n\nLLMBox is developed and maintained by [AI Box](http://aibox.ruc.edu.cn/). See more details in [change log](https://github.com/RUCAIBox/LLMBox/tree/main/utilization#change-log)\n\n## License\n\nLLMBox uses [MIT License](./LICENSE).\n\n## Reference\n\nIf you find LLMBox useful for your research or development, please cite the following papers:\n\n```\n@inproceedings{tang2024llmbox,\n  title={LLMBox: A Comprehensive Library for Large Language Models},\n  author={Tang, Tianyi and Yiwen, Hu and Li, Bingqian and Luo, Wenyang and Qin, ZiJing and Sun, Haoxiang and Wang, Jiapeng and Xu, Shiyi and Cheng, Xiaoxue and Guo, Geyang and others},\n  booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},\n  pages={388--399},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRUCAIBox%2FLLMBox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRUCAIBox%2FLLMBox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRUCAIBox%2FLLMBox/lists"}