{"id":33079605,"url":"https://github.com/CSHaitao/ChatGLM_mutli_gpu_tuning","last_synced_at":"2025-11-23T23:01:36.060Z","repository":{"id":163053973,"uuid":"637700085","full_name":"CSHaitao/ChatGLM_mutli_gpu_tuning","owner":"CSHaitao","description":"deepspeed+trainer简单高效实现多卡微调大模型","archived":false,"fork":false,"pushed_at":"2023-05-27T07:07:02.000Z","size":139,"stargazers_count":102,"open_issues_count":10,"forks_count":10,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-06-17T05:38:16.581Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CSHaitao.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-05-08T08:21:31.000Z","updated_at":"2024-06-03T12:35:26.000Z","dependencies_parsed_at":"2024-01-14T14:29:26.157Z","dependency_job_id":"fd97884d-e071-44ae-9656-0b42f030cf32","html_url":"https://github.com/CSHaitao/ChatGLM_mutli_gpu_tuning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CSHaitao/ChatGLM_mutli_gpu_tuning","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSHaitao%2FChatGLM_mutli_gpu_tuning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSHaitao%2FChatGLM_mutli_gpu_tuning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSHaitao%2FChatGLM_mutli_gpu_tuning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSHaitao%2FChatGLM_mutli_gpu_tuning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CSHaitao","download_url":"https://codeload.github.com/CSHaitao/ChatGLM_mutli_gpu_tuning/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSHaitao%2FChatGLM_mutli_gpu_tuning/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286035620,"owners_count":27274546,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-23T02:00:06.149Z","response_time":135,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-14T12:00:24.458Z","updated_at":"2025-11-23T23:01:36.032Z","avatar_url":"https://github.com/CSHaitao.png","language":"Python","funding_links":[],"categories":["文本生成"],"sub_categories":["ChatGLM-6B以及扩展"],"readme":"\u003c!--\n * @Author: lihaitao\n * @Date: 2023-05-08 17:52:50\n * @LastEditors: Do not edit\n * @LastEditTime: 2023-05-09 14:34:49\n * @FilePath: /lht/ChatGLM_LoRA/README.md\n--\u003e\n\n# ChatGLM_multi_gpu_zero_Tuning\n\n实现multi_gpu + zero 微调ChatGLM, 目前已实现LoRA, Ptuning-v2, Freeze三种微调方式。\n相关debug记录可以见[知乎](https://zhuanlan.zhihu.com/p/627741267)\n\n使用该代码训练的法律大模型LexiLaw可见[LexiLaw](https://github.com/CSHaitao/LexiLaw)\n\n## Data Structure\n\ndata.py中使用InstrutionDataset和InstrutionCollator处理数据。\n\n每条数据包含instruction,inpuy,answer,示例如下:\n\n\u003e {\"instruction\": \"\", \"input\": \"工作时把右小指挤段又接上的，XXXX左右的工资，在说我们感觉太不公平了，刚手术做了XX个小时\", \"answer\": \"这种情况，是属于工伤的。按照法律规定，需要先行认定工伤，认定工伤后，再进行劳动能力鉴定，然后再具体索赔。涉及赔偿项目比较多，一般包括医疗费、护理费、伙食费、停工留薪期内的工资，构成伤残的，还可以主张一次性伤残补助金、一次性伤残就业补助金、一次性工伤医疗补助金等，每一项赔偿的标准不一样，需要根据实际情况来具体计算。由于工伤案件，涉及程序较多、法律关系复杂，建议委托专业律师代理处理。我们经常代理工伤赔偿案件，如果愿意，我们可以协助处理。详情可以电联，或者面谈。\"}\n\nInstrutionDataset中的prefix只有ptuning时使用, InstrutionCollator中max_len和max_input_len用来控制输入长度\n\n## Train\n\n### LoRA\n\n1.修改lora.sh中模型和数据路径\n\n2.运行sh lora.sh\n```bash\n    CUDA_VISIBLE_DEVICES=${TOT_CUDA} deepspeed --master_port=$PORT --num_gpus=3 lora.py \\\n        --train_path ./instrution_data.json \\\n        --max_len 768 \\\n        --max_input_len 512 \\\n        --model_name_or_path ./chatGLM-6B \\\n        --tokenizer_name ./chatGLM-6B \\\n        --lora_rank 8 \\\n        --per_device_train_batch_size 16 \\\n        --gradient_accumulation_steps 4 \\\n        --num_train_epochs 10 \\\n        --save_steps 900 \\\n        --learning_rate 1e-5 \\\n        --fp16 \\\n        --remove_unused_columns false \\\n        --logging_steps 50 \\\n        --output_dir /output \\\n        --deepspeed /ds_config.json \\\n```\n单卡运行可以改为 num_gpus == 1\n\nLoRA的参数如下,可根据实际情况调整:\n\n```bash\n    peft_config = LoraConfig(\n        task_type=\"CAUSAL_LM\",\n        lora_alpha=32,  \n        target_modules=[\"query_key_value\"],\n        inference_mode=False,\n        r=training_args.lora_rank,\n        lora_dropout=0.1,\n        bias=\"none\",\n        fan_in_fan_out = False\n    )\n```\n\n### P-tuning-v2\n\n根据ChatGLM-6B官方P—tuning代码修改。\n\n1.修改ptuning.sh中模型和数据路径\n\n2.运行sh ptuning.sh\n```bash\n    CUDA_VISIBLE_DEVICES=${TOT_CUDA} deepspeed --master_port=$PORT --num_gpus=2 finetune_ptuning.py \\\n        --train_path ./instrution_data.json \\\n        --max_len 768 \\\n        --max_input_len 512 \\\n        --model_name_or_path /chatGLM-6B \\\n        --tokenizer_name/chatGLM-6B \\\n        --per_device_train_batch_size 8 \\\n        --gradient_accumulation_steps 4 \\\n        --num_train_epochs 10 \\\n        --save_steps 2000 \\\n        --learning_rate 1e-5 \\\n        --fp16 \\\n        --logging_steps 50 \\\n        --prefix_projection True \\\n        --pre_seq_len $PRE_SEQ_LEN \\\n        --output_dir /output \\\n        --deepspeed ds_config.json \\\n```\n\n其中$PRE_SEQ_LEN是soft prompt的长度, 可以根据实际情况修改。\n\n### Freeze\n\n1.修改freeze.sh中模型和数据路径\n\n2.运行sh freeze.sh\n\n```bash\n    CUDA_VISIBLE_DEVICES=${TOT_CUDA} deepspeed --master_port=$PORT --num_gpus=3 finetune_freeze.py \\\n        --train_path  \\\n        --max_len 768 \\\n        --max_input_len 512 \\\n        --model_name_or_path /chatGLM-6B \\\n        --tokenizer_name /chatGLM-6B \\\n        --lora_rank 8 \\\n        --per_device_train_batch_size 16 \\\n        --gradient_accumulation_steps 4 \\\n        --num_train_epochs 10 \\\n        --save_steps 900 \\\n        --learning_rate 1e-5 \\\n        --fp16 \\\n        --remove_unused_columns false \\\n        --logging_steps 50 \\\n        --output_dir output_freeze \\\n        --deepspeed ds_config.json \\\n```\n\n可通过以下代码修改可训练的层数:\n```bash\n    for name, param in model.named_parameters():\n        if not any(nd in name for nd in [\"layers.27\", \"layers.26\", \"layers.25\", \"layers.24\", \"layers.23\"]):\n            param.requires_grad = False\n```\n\n## Requirements\n\n```\npython=3.9\ntransformers==4.28.1\ntqdm==4.64.1\ndatasets==2.8.0\npytorch==1.12.1\ndeepspeed==0.9.1\npeft==0.3.0 \n```\n一定要使用peft==0.3.0\n\n## Todo\n\n- [ ] 增加模型并行和多卡inference\n\n\n## Contact\n\nIf you find our work useful, please do not save your star!\n\nIf you have any questions, please email liht22@mails.tsinghua.edu.cn\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCSHaitao%2FChatGLM_mutli_gpu_tuning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCSHaitao%2FChatGLM_mutli_gpu_tuning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCSHaitao%2FChatGLM_mutli_gpu_tuning/lists"}