{"id":20779469,"url":"https://github.com/freedomintelligence/fastllm","last_synced_at":"2025-04-30T19:44:01.504Z","repository":{"id":195680110,"uuid":"691468247","full_name":"FreedomIntelligence/FastLLM","owner":"FreedomIntelligence","description":"Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler]; ","archived":false,"fork":false,"pushed_at":"2024-01-04T11:54:05.000Z","size":831,"stargazers_count":36,"open_issues_count":0,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T19:22:35.111Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-14T08:35:50.000Z","updated_at":"2024-12-22T14:37:44.000Z","dependencies_parsed_at":"2023-09-19T07:54:49.592Z","dependency_job_id":null,"html_url":"https://github.com/FreedomIntelligence/FastLLM","commit_stats":null,"previous_names":["wangxidong06/fastest_sft","wangxidong06/fast_sft","wangxidong06/fast_llm","wangxidong06/fastllm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FFastLLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FFastLLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FFastLLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FFastLLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/FastLLM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251772587,"owners_count":21641465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T13:28:04.080Z","updated_at":"2025-04-30T19:44:01.472Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fast-LLM\n![Fast_SFT](assets/title.png)\n\n\u003cp align=\"center\"\u003e\n   📃 \u003ca href=\"https://eegb6fzscd.feishu.cn/wiki/XTMBwrMBgii6nnkqNsLcZHxznLE?from=from_copylink\" target=\"_blank\"\u003eDoc\u003c/a\u003e \n\u003c/p\u003e\n\n- Fast LLM Training CodeBase [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler]\n- With dynamic strategy choosing\n\n## How to use\n1. Get Model size, GPU memory usage, training time and strategy.\n2. Prepare Model and Data.\n3. Use recommended parameters in 1. for training.\n4. Calculate TFLOPs.\n5. Transfer model to Huggingface format. \n\n### 1. Calculate reference indicators\nModify the CONSTANT in pre_train_math.py\n```\npython 1.pre_train_math.py\n```\nThe output of training LLAMA-70B with 6Node*4GPU(80G) and 0.7B tokens:\n```\n-----------Model_Size and GPU_Mem-----------\n+--------------+------------------------+----------------------+\n| Model size/B | ratio(NHIDDEN/NLAYERS) | Usable_mem_per_GPU/G |\n+--------------+------------------------+----------------------+\n|    64.72     |          102           |          79          |\n+--------------+------------------------+----------------------+\n-----------With Mixed Precision(bp16)-----------\n-----Memory_reference_indicator(Batch_size=8)-----\n+-------------------------+----------+------------------+-------------------+\n| Module                  |   Size/B |   Eval_memory/GB |   Train_momery/GB |\n+=========================+==========+==================+===================+\n| emb                     |     0.3  |             0.59 |              4.73 |\n+-------------------------+----------+------------------+-------------------+\n| one_layer               |     0.81 |             1.61 |             12.89 |\n+-------------------------+----------+------------------+-------------------+\n| input                   |     0.27 |             0.54 |              0.54 |\n+-------------------------+----------+------------------+-------------------+\n| activation(batchsize=1) |     9.55 |            19.11 |             19.11 |\n+-------------------------+----------+------------------+-------------------+\n| ALL                     |    92.01 |           184.03 |           1090.17 |\n+-------------------------+----------+------------------+-------------------+\n-----Strategy_reference_indicator(Batch_size=8)-----\n+------------+--------------------------+---------------------------+\n| Strategy   |   Eval_memory_per_gpu/GB |   Train_momery_per_gpu/GB |\n+============+==========================+===========================+\n| Zero1      |                   129.45 |                    345.84 |\n+------------+--------------------------+---------------------------+\n| Zero2      |                   129.45 |                    221.78 |\n+------------+--------------------------+---------------------------+\n| Zero3      |                     5.39 |                     97.73 |\n+------------+--------------------------+---------------------------+\n---------------------Strategy_Recommand---------------------\nYou can't use pure Zero1 or Zero2 strategy.\nRecommand_Strategy:\n+-----------------+------+------+------+---------------------------+-----------------+\n| Zero            |   DP |   TP |   PP |   Train_momery_per_gpu/GB |   Trianing_days |\n+=================+======+======+======+===========================+=================+\n| Zero1+TP+PP     |    1 |    4 |    6 |                     56.79 |            1.25 |\n+-----------------+------+------+------+---------------------------+-----------------+\n| Zero3+(offload) |   24 |    1 |    1 |                     97.73 |            1.25 |\n+-----------------+------+------+------+---------------------------+-----------------+\nPlease find the best batch_size by adjusting BATCH_SIZE\n```\n\n### 2. Prepare Model and Data\nUnder Construction\n#### 2.1 Prepare Model\n- LLAMA:\n  1. Convert LLAMA from Meta format checkpoints to HF format\n    ```\n    python /src/tools/convert_checkpoint/convert_llama_weights_to_hf.py --input_dir $LLAMA_FORMAT_DIR --output_dir $HF_FORMAT_DIR --model_size 7B\n    # --model_size include 7B, 13B, and 70B (for pretrained-only models), and 7Bf, 13Bf, and 70Bf (for chat-finetuned models).\n    ```\n\n  2. Convert HF checkpoints to Megatron format\n    ```\n    python /src/tools/checkpoint/util.py \\\n          --model-type GPT \\\n          --loader llama2_hf \\\n          --saver megatron \\\n          --target-tensor-parallel-size ${TP} \\\n          --load-dir ${HF_FORMAT_DIR} \\\n          --save-dir ${MEGATRON_FORMAT_DIR} \\\n          --tokenizer-model ${TOKENIZER_MODEL}\n    ```\n- Others:\n```\npython tools/convert_checkpoint/deepspeed_to_megatron.py --input_folder INPUT_FOLDER --output_folder OUTPUT_FOLDER --target_tp TARGET_TP --target_pp TARGET_PP \n```\n#### 2.2 Prepare Data\nData_item is in jsonl\n```\n{\"text\": \"The quick brown fox\"}\n{\"text\": \"jumps over the lazy dog\"}\n```\nThe name of the text field of the json can be changed by using the --json-key flag in preprocess_data.py. \"text\" by default\n\n```\npython tools/preprocess_data.py \\\n       --input data.json \\\n       --output-prefix llama2 \\\n       --vocab-file VOCAB_FILE \\\n       --dataset-impl mmap \\\n       --tokenizer-type GPT2BPETokenizer \\\n       --merge-file gpt2-merges.txt \\\n       --append-eod\n```\n\n\n### 3. Train Model\nModify the CONSTANT in 3.pretrain_xxxxxx.sh\n```\nbash 3.pretrain_xxxxxx.sh\n```\n\n### 4. Calculate TFLOPs.\nModify the CONSTANT in aft_train_math.py\n```\npython 4.aft_train_math.py\n```\n\n### 5. Transfer model to HF Transformers. \n\n```bash\npython /src/tools/convert_checkpoint/deepspeed_to_transformers.py  \\\n--input_folder /path/to/checkpoint \\\n--output_folder /path/to/transformers/checkpoint\n```\n\n## To do list\n1. Support Baichuan2.\n2. Support Instruction tuning.\n3. Benchmark TFLOPS with other Repo on different settings\n\n## Acknowledgement\n- Megatron-DeepSpeed: https://github.com/microsoft/Megatron-DeepSpeed\n- DeepSpeed: https://github.com/microsoft/DeepSpeed\n- Megatron-LM: https://github.com/NVIDIA/Megatron-LM\n\n\n\n## Citation\n```\n@misc{fastllm,\n  title={Fast LLM Training CodeBase},\n  author={Xidong Wang},\n  year = {2023},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/wangxidong06/Fast_LLM}},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Ffastllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Ffastllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Ffastllm/lists"}