{"id":13393547,"url":"https://github.com/AetherCortex/Llama-X","last_synced_at":"2025-03-13T19:31:40.632Z","repository":{"id":149314979,"uuid":"621305466","full_name":"AetherCortex/Llama-X","owner":"AetherCortex","description":"Open Academic Research on Improving LLaMA to SOTA LLM","archived":false,"fork":false,"pushed_at":"2023-08-30T00:25:38.000Z","size":7776,"stargazers_count":1605,"open_issues_count":24,"forks_count":102,"subscribers_count":42,"default_branch":"main","last_synced_at":"2024-10-26T18:30:18.014Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AetherCortex.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-30T11:57:12.000Z","updated_at":"2024-10-23T17:06:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"d9303cee-e78d-4544-8be8-b01e65e1e6b5","html_url":"https://github.com/AetherCortex/Llama-X","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AetherCortex%2FLlama-X","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AetherCortex%2FLlama-X/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AetherCortex%2FLlama-X/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AetherCortex%2FLlama-X/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AetherCortex","download_url":"https://codeload.github.com/AetherCortex/Llama-X/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243469172,"owners_count":20295700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T17:00:55.752Z","updated_at":"2025-03-13T19:31:40.613Z","avatar_url":"https://github.com/AetherCortex.png","language":"Python","readme":"\r\n\u003cp align=\"center\" width=\"100%\"\u003e\r\n\u003ca \u003e\u003cimg src=\"src/imgs/pandallm.png\" alt=\"Llama-X\" style=\"width: 60%; min-width: 300px; display: block; margin: auto;\"\u003e\u003c/a\u003e\r\n\u003c/p\u003e\r\n\r\n\r\n[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg)](https://github.com/tatsu-lab/stanford_alpaca/blob/main/LICENSE)\r\n[![Data License](https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-red.svg)](https://github.com/tatsu-lab/stanford_alpaca/blob/main/DATA_LICENSE)\r\n\r\n## Llama-X: Open Academic Research on Improving LLaMA to SOTA LLM\r\n\r\n\r\nThis is the repo for the Llama-X, which aims to:\r\n\r\n- Progressively improve the performance of LLaMA to SOTA LLM with open-source community.\r\n- Conduct Llama-X as an open academic research which is long-term, systematic and rigorous.\r\n- Save the repetitive work of community and we work together to create more and faster increment.\r\n\r\nThe project will follow these principles:\r\n\r\n- We will publish all the `code`, `model`, `data`, and `experiments` details.\r\n- We will `continuously` improve the model version by version and open the `newest` method.\r\n- We will summary the method of each main version as `academic papers`.\r\n- We announce a complete [research plan](#research-areas). The contributors are wellcome to cooperate with \r\neach other to progressively improve Llama-X through \r\niteration of the [target versions](#model).\r\n- The check-in of the new model must achieve significant improvement with current version on [automatic evaluation](#evaluation).\r\n\r\n\u0026#x1F4E3; Please join \u003ca href=\"https://discord.gg/2etwhe6GvU\"\u003e\u003cimg alt=\"Join us on Discord\" src=\"https://img.shields.io/discord/823813159592001537?color=5865F2\u0026logo=discord\u0026logoColor=white\"\u003e\u003c/a\u003e if you are interested in Llama-X.\r\n\r\n## Contents\r\n1. [News](#news)\r\n\r\n2. [Ten main research areas](#research-areas)\r\n\r\n3. [Llama-X Model Version](#model)\r\n\r\n4. [Llama-X Evaluation](#evaluation)\r\n\r\n5. [Llama-X Paper List](#paper)\r\n\r\n6. [Usage](#usage)\r\n\r\n7. [How to contribute](#contribute)\r\n\r\n\r\n\u003ch2 id=\"news\"\u003eNews\u003c/h2\u003e\r\n\r\nWe have completed the training of our first version of model (Llama-X 3.0.1 7B). Please experience our model in the [demo page](https://cedfabe00f0fcbde.gradio.app), and the data, code and model weights of different scales will be updated in this repo later.\r\n\r\n\u003ch2 id=\"research-areas\"\u003eTen main research areas\u003c/h2\u003e\r\n\r\n[1]. Research on `Instruction Tuning`\r\n- [ ] instruction-following tuning\r\n\r\n[2]. Research on `RLHF \u0026 RLAIF`\r\n- [ ] fundamental RLHF \r\n- [ ] AI learning from AI\r\n\r\n[3]. Research on `Data Quality`\r\n- [ ] high quality data for pre-training, fine-tuning, user feedbacks, multi-modality, etc\r\n\r\n[4]. Research on `Long Context Transformer`\r\n- [ ] enable efficient transformers for long sequence (\u003e30k)\r\n\r\n[5]. Research on `Multi-modal (text + image) Modeling`\r\n- [ ] text + image in; text out\r\n\r\n[6]. Research on `Multilingual`\r\n- [ ] comparable multilingual performance with English\r\n\r\n[7]. Research on `Efficient infrastructure and optimization`\r\n- [ ] improve training and inference speed\r\n- [ ] build deep learning stack which scales predictably\r\n\r\n[8]. Research on `Evaluation`\r\n- [ ] comprehensive evaluation of model capabilities\r\n\r\n[9]. Research on `Interpretability`\r\n- [ ] interpret the source of each capability of LLM\r\n\r\n[10]. Research on `LLM on Actions`\r\n- [ ] combine LLM with search, recommendation and other plugins\r\n\r\n\r\n\u003ch2 id=\"model\"\u003eLlama-X Model Version\u003c/h2\u003e\r\n\r\n\r\n\r\n| Llama-X       | Baseline         | Performance                        |\r\n|---------------|------------------|------------------------------------|\r\n| 3.0.0 (LLaMA) | GPT-3            | Outperform |\r\n| 3.1.0         | text-davinci-001 | Comparable                         |\r\n| 3.2.0         | text-davinci-002 | Comparable                         |\r\n| 3.3.0         | text-davinci-003 | Comparable                         |\r\n| 3.5.0         | gpt-35-turbo     | Comparable                         |\r\n| 3.6.0         | GPT-4            | 80% Avg.Gap                        |\r\n| 3.7.0         | GPT-4            | 60% Avg.Gap                        |\r\n| 3.8.0         | GPT-4            | 40% Avg.Gap                        |\r\n| 3.9.0         | GPT-4            | 20% Avg.Gap                        |\r\n| 4.0.0         | GPT-4            | Comparable                         |\r\n\r\n\r\nWe are focusing on the above research areas [1] \u0026 [3] now, and would public our first version of model (Llama-X 3.0.1) and paper.\r\n\r\n\r\n\r\n\u003ch2 id=\"evaluation\"\u003eLlama-X Evaluation\u003c/h2\u003e\r\n\r\nEach new version of Llama-X model should significantly outperform (+\u003e1%) the current version model on the automatic evaluation \r\nof all the following Type-A benchmarks. And the additional evaluation for Type-B benchmarks should be added in the 3.6.0+ versions:\r\n\r\n| Type | Benchmarks          |                       \r\n|------|---------------------|\r\n| A    | MMLU                | \r\n| A    | HumanEval           |\r\n| A    | GSM-8K              | \r\n| A    | NaturalQuestions    |\r\n| A    | TruthfulQA          | \r\n| B    | Leetcode            | \r\n| B    | GRE                 | \r\n| B    | AP                  | \r\n| B    | MMLU-Multilingual   |\r\n| B    | Visual Inputs (TBD) |\r\n\r\n\r\nResults:\r\n\r\n| Model                        | MMLU   | TruthfulQA | GSM-8K | NaturalQuestions |\r\n|------------------------------|--------|------------|--------|------------------|\r\n|InstructGPT davinci v2 (175B)^ | 0.57   | 0.62       |  0.35  | 0.389            |\r\n|Llama-X 3.0.1 (7B)            | 0.4412 | 0.2032     |  0.1887| 0.2422           |\r\n|Llama-i (7B)                  | 0.5121 | 0.2142     |  0.2259| 0.3499           |\r\n\r\n^ The results of `InstructGPT davinci v2 (175B)` are copied from [Stanford CRFM Benchmark](https://crfm.stanford.edu/).\r\n\r\n\u003ch2 id=\"paper\"\u003eLlama-X Paper List\u003c/h2\u003e\r\n\r\n1. [LLaMA: Open and Efficient Foundation Language Models.](https://arxiv.org/abs/2302.13971v1)\r\n\r\n\r\n\u003ch2 id=\"usage\"\u003eUsage\u003c/h2\u003e\r\n\r\n- Setup. Install the conda environment:\r\n```bash\r\nconda create -n llamax python=3.10\r\nconda activate llamax\r\ngit clone https://github.com/AetherCortex/Llama-X.git\r\ncd Llama-X/src\r\nconda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.3 -c pytorch\r\npip install transformers==4.31.0\r\ncd ../..\r\npip install -r requirements.txt\r\n```\r\n\r\n- Training data example (e.g., [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json)):\r\n```bash\r\nLlama-X/src/data/alpaca_data.json\r\n```\r\n\r\n- Convert LLaMA checkpoint to HuggingFace format:\r\n```bash\r\ncd Llama-X/src\r\npython transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py \\\r\n    --input_dir /path/to/llama-7B/ \\\r\n    --model_size 7B \\\r\n    --output_dir /path/to/llama-7B/hf\r\n```\r\n\r\n- Train LLaMA-7B on DeepSpeed Zero-3:\r\n```bash\r\ndeepspeed train.py \\\r\n    --model_name_or_path /path/to/llama-7B/hf \\\r\n    --data_path /path/to/example_data.json \\\r\n    --output_dir /path/to/llama-7B/hf/ft \\\r\n    --num_train_epochs 3 \\\r\n    --model_max_length 512 \\\r\n    --per_device_train_batch_size 64 \\\r\n    --per_device_eval_batch_size 1 \\\r\n    --gradient_accumulation_steps 1 \\\r\n    --evaluation_strategy \"no\" \\\r\n    --save_strategy \"steps\" \\\r\n    --save_steps 100 \\\r\n    --save_total_limit 2 \\\r\n    --learning_rate 2e-5 \\\r\n    --warmup_steps 2 \\\r\n    --logging_steps 2 \\\r\n    --lr_scheduler_type \"cosine\" \\\r\n    --report_to \"tensorboard\" \\\r\n    --gradient_checkpointing True \\\r\n    --deepspeed configs/deepspeed_config.json \\\r\n    --fp16 True\r\n```\r\n- Train LLaMA-7B on DeepSpeed Zero-3 with Multi-nodes\r\n```bash\r\ndeepspeed --num_gpus num_of_gpus_in_each_node \\\r\n    --num_nodes num_of_nodes \\\r\n    --master_addr ip_address_of_main_node \\\r\n    --master_port 34545 \\\r\n    --hostfile configs/hostfile \\\r\n    train.py \\\r\n    --model_name_or_path /path/to/llama-7B/hf \\\r\n    --data_path /path/to/example_data.json \\\r\n    --output_dir /path/to/llama-7B/hf/ft \\\r\n    --num_train_epochs 3 \\\r\n    --model_max_length 512 \\\r\n    --per_device_train_batch_size 64 \\\r\n    --per_device_eval_batch_size 4 \\\r\n    --gradient_accumulation_steps 1 \\\r\n    --evaluation_strategy \"no\" \\\r\n    --save_strategy \"steps\" \\\r\n    --save_steps 100 \\\r\n    --save_total_limit 2 \\\r\n    --learning_rate 2e-5 \\\r\n    --warmup_steps 2 \\\r\n    --logging_steps 2 \\\r\n    --lr_scheduler_type \"cosine\" \\\r\n    --report_to \"tensorboard\" \\\r\n    --gradient_checkpointing True \\\r\n    --deepspeed configs/deepspeed_config.json \\\r\n    --fp16 True\r\n```\r\n\r\n- The current code of Llama-X support:\r\n    - Fully Finetune: Optimize full LLaMA checkpoint, instead of `Low-Rank Adaptation (LoRA)`.\r\n    - High Efficiency: Training 7B model with `50k examples/epoch` \u0026 `batch_size=64` within `1 hour` on `8 x V100 GPUs`.\r\n\r\n| LLaMA  | Batch Size | V100s  | Time (h)    |\r\n|--------|------------|--------|-------------|\r\n| 7 B    | 64         | 8      | 1.00        |\r\n| 13 B   | 32         | 8      | 2.00        |\r\n\r\n\r\n- Inference\r\n```bash\r\n# web demo inference\r\npython generate.py\r\n\r\n# batch inference\r\nTo Do\r\n```\r\n\r\n\r\n\u003ch2 id=\"contribute\"\u003eHow to contribute\u003c/h2\u003e\r\n\r\nDevelopers can become Contributors by contributing helpful code, data, paper and computing resource, etc.\r\n\r\n1. Code: Including algorithm implementation, training optimization, inference optimization, and model deployment.\r\n\r\n2. Data: Every [research area](#research-areas) and [version iteration](#model) requires high-quality data, including instruction-answer, pre-training, multi-modal, multilingual, and user feedbacks data, etc.\r\n\r\n3. Paper: We will maintain a [Llama-X Paper List](#paper), and use Llama-X as the base model for optimized, fully tested, and significantly improved academic papers. You can check in to the Llama X Paper List.\r\n\r\n4. Computing resource: We hope to help accelerate model iteration speed by coordinating redundant computing power from some developers or non-profit sponsorship from universities/enterprises.\r\n\r\n\u003ch2 id=\"communication\"\u003eHow to communicate with us\u003c/h2\u003e\r\n\r\n1. Github Issues\r\n\r\n2. Email: llama-x@mail.com\r\n\r\n3. Discord: \u003ca href=\"https://discord.gg/2etwhe6GvU\"\u003e\u003cimg alt=\"Join us on Discord\" src=\"https://img.shields.io/discord/823813159592001537?color=5865F2\u0026logo=discord\u0026logoColor=white\"\u003e\u003c/a\u003e\r\n\r\n\r\n## Thanks For\r\n\r\nThis project has been inspired by multiple open source projects:\r\n\r\n[Meta AI LLaMA](https://arxiv.org/abs/2302.13971v1)\r\n\r\n[Huggingface Transformers Llama](https://github.com/huggingface/transformers/tree/main/src/transformers/models/llama)\r\n\r\n[Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) and [Alpaca-LoRA](https://github.com/tloen/alpaca-lora)\r\n\r\n\r\n## Disclaimer\r\n\r\nThe use of resources(e.g., code, data and model weights) related to this project is limited to academic research and is prohibited for commercial purposes. The content generated by any model of Llama-X is subject to factors such as randomness and uncontrollability, and this project cannot guarantee its accuracy. This project does not assume any legal responsibility for the content of the model output, nor does it assume any responsibility for any losses that may arise from the use of related resources and output results.\r\n\r\n\r\n\r\n\r\n","funding_links":[],"categories":["Fine-tuning","Open Source LLM","LLM-List","Python","A01_文本生成_文本对话","multi lang models"],"sub_categories":["Open-LLM","大语言对话模型及数据"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAetherCortex%2FLlama-X","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAetherCortex%2FLlama-X","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAetherCortex%2FLlama-X/lists"}