{"id":18673761,"url":"https://github.com/opencsgs/llm-finetune","last_synced_at":"2025-04-12T01:31:57.487Z","repository":{"id":233447882,"uuid":"786697199","full_name":"OpenCSGs/llm-finetune","owner":"OpenCSGs","description":"The framework of training large language models，support lora, full parameters fine tune etc, define yaml to start training/fine tune of your defined models, data and methods. Easy define and easy start. ","archived":false,"fork":false,"pushed_at":"2024-09-19T07:09:00.000Z","size":1359,"stargazers_count":25,"open_issues_count":4,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-25T21:38:42.741Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenCSGs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-15T06:12:37.000Z","updated_at":"2025-02-04T05:19:28.000Z","dependencies_parsed_at":"2024-04-16T08:06:43.736Z","dependency_job_id":"27a48c5e-bb92-4986-9340-9a62f288b1bd","html_url":"https://github.com/OpenCSGs/llm-finetune","commit_stats":null,"previous_names":["opencsgs/llm-finetune"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenCSGs%2Fllm-finetune","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenCSGs%2Fllm-finetune/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenCSGs%2Fllm-finetune/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenCSGs%2Fllm-finetune/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenCSGs","download_url":"https://codeload.github.com/OpenCSGs/llm-finetune/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248504289,"owners_count":21115141,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T09:16:30.816Z","updated_at":"2025-04-12T01:31:57.002Z","avatar_url":"https://github.com/OpenCSGs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# LLM - Finetune\n\nThe framework of training large language models，support lora, full parameters fine tune etc, define yaml to start training/fine tune of your defined models, data and methods. Easy define and easy start. A large-scale model training framework that supports tasks such as LoRA and full-parameter fine-tuning. Easily initiate your large model training and fine-tuning work by defining a YAML file specifying the base model, dataset, and training parameters. Feedback and stars⭐️ are welcome!\n\n\u003cimg src=\"./docs/llm-finetune.png\" alt=\"image\" width=600 height=\"auto\"\u003e\n\nTwo steps to run your LLM finetune:\n\n## 1. Easy Install\n\n### Installation\n\nUse SHELL `bash` for command\n\n```bash\ngit clone https://github.com/OpenCSGs/llm-finetune.git\ncd llm-finetune\npip install .  # Install from CN: 'pip install . -i https://pypi.tuna.tsinghua.edu.cn/simple'\n```\n\n## 2. Easy Run\n### Finetune model by command\n\n```\nllmfinetune run-ft --ft=./models/ft--sequenceclassification--bert-base-uncased-lora.yaml\n```\n\nYour Finetune task is starting now. \n\n- You can add more yaml for your own to define your tasks.\n\n*******\n## (Optional) Addtional launcher: Launch by accelerate or deepspeed\nEnsure accelerate and deepspeed are installed, then follow below steps.\n\n### Launch by accelerate\n\nModify parameters of `accelerate launch` for distributed train.\n\n#### Finetune on CPU\n\n```\n# Use CPU\naccelerate launch --cpu --num_machines=1 --num_processes=1 --num_cpu_threads_per_process=1 --mixed_precision=no --dynamo_backend=no llm_finetune.py run-ft --ft=/Users/hub/code/jihulab/opencsg/llm-inference/models/ft--sequenceclassification--bert-base-cased.yaml\n```\n\n#### Finetune on GPU on single host\n\nControl GPU Visibility with `CUDA_VISIBLE_DEVICES`. \n\n```\n# Use GPU:0\nCUDA_VISIBLE_DEVICES=0 accelerate launch llmfinetune run-ft --ft=./models/ft--sequenceclassification--bert-base-uncased-lora.yaml\n\n# Use GPU:1 \nCUDA_VISIBLE_DEVICES=1 accelerate launch llmfinetune run-ft --ft=./models/ft--sequenceclassification--bert-base-uncased-lora.yaml\n\n# Use GPU:0\naccelerate launch --num_machines=1 --num_processes=1 --gpu_ids=0 llmfinetune ...\n```\n\n#### Finetune on multi-GPUs on single host\n\n```\n# Use all GPUs with mixed precision disabled\naccelerate launch --multi_gpu llmfinetune ...\n\n# Use all GPUs with mxied precision\naccelerate launch --multi_gpu --mixed_precision=fp16 llmfinetune ...\n\n# Use GPU:0 and GPU:1\nCUDA_VISIBLE_DEVICES=0,1 accelerate launch --multi_gpu --gpu_ids=0,1 llmfinetune ...\n\n# Launch with 2 GPUs\naccelerate launch --multi_gpu --num_processes 2 llmfinetune ...\n```\n\n```\n# Use default_config.yaml\ncompute_environment: LOCAL_MACHINE\ndeepspeed_config: {}\ndistributed_type: MULTI_GPU\ndowncast_bf16: 'no'\nmachine_rank: 'NO'\nfsdp_config: {}\ngpu_ids: all                  # all GPUs id\nmachine_rank: 0\nmain_training_function: main\nmegatron_lm_config: {}\nmixed_precision: fp16         # mixed precsion\nnum_machines: 1               # a single machine\nnum_processes: 4              # 4 GPUs\nrdzv_backend: static\nsame_network: true\nuse_cpu: false\n\n\naccelerate launch --config_file default_config.yaml llmfinetune ...\n```\n\n#### Finetune on multi-GPUs on multi-hosts\n\nAll hosts need access without password each other\n\n```\n# default_config.yaml\ncompute_environment: LOCAL_MACHINE\ndeepspeed_config:\n  deepspeed_multinode_launcher: standard\n  gradient_accumulation_steps: 1\n  gradient_clipping: 1.0\n  offload_optimizer_device: none\n  offload_param_device: none\n  zero3_init_flag: true\n  zero3_save_16bit_model: true\n  zero_stage: 3\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\ndynamo_config: {}\nfsdp_config: {}\nmain_training_function: main\nmegatron_lm_config: {}\nmixed_precision: fp16\nnum_machines: 2               # 2 nodes\nnum_processes: 16             # 16 GPUs of all nodes \ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n\n# Run on all hosts by specify `RANK`, `MASTER_ADDR`, `MASTER_PORT`\naccelerate launch --config_file default_config.yaml \\\n    --machine_rank ${RANK} \\\n    --main_process_ip ${MASTER_ADDR} \\\n    --main_process_port ${MASTER_PORT} \\ \n    ...\n\n# --machine_rank: 0 for the main/master node, for other nodes is 1,2,3 etc.\n```\n\n#### Finetune by Deepspeed for multi-GPUs on multi-hosts\n\nAll hosts need access without password each other\n\n```\n# myhostfile\nnode1 slots=1\nnode2 slots=1\n\n# deepspeed.json\n{\n    \"train_batch_size\": \"auto\",\n    \"train_micro_batch_size_per_gpu\": \"auto\",\n    \"gradient_accumulation_steps\": \"auto\",\n    \"gradient_clipping\": \"auto\",\n    \"zero_allow_untested_optimizer\": true,\n    \"fp16\": {\n      \"enabled\": \"auto\",\n      \"loss_scale\": 0,\n      \"initial_scale_power\": 16,\n      \"loss_scale_window\": 1000,\n      \"hysteresis\": 2,\n      \"min_loss_scale\": 1\n    },\n    \"zero_optimization\": {\n      \"stage\": 2,\n      \"allgather_partitions\": true,\n      \"allgather_bucket_size\": 5e8,\n      \"reduce_scatter\": true,\n      \"reduce_bucket_size\": 5e8,\n      \"overlap_comm\": false,\n      \"contiguous_gradients\": true\n    }\n}\n\ndeepspeed --num_nodes=2 --hostfile=myhostfile --deepspeed deepspeed.json ...\n\n# --num_nodes: num of host\n# --hostfile:  host file include all hosts IP and num of GPUs\n# --deepspeed: deepspeed config file\n\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencsgs%2Fllm-finetune","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopencsgs%2Fllm-finetune","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencsgs%2Fllm-finetune/lists"}