{"id":20775112,"url":"https://github.com/cscfi/llm-fine-tuning-examples","last_synced_at":"2025-09-28T03:30:53.075Z","repository":{"id":255326753,"uuid":"848106268","full_name":"CSCfi/llm-fine-tuning-examples","owner":"CSCfi","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-15T10:14:20.000Z","size":48,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-15T12:09:37.230Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CSCfi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-27T06:25:48.000Z","updated_at":"2025-01-15T10:14:22.000Z","dependencies_parsed_at":"2024-08-29T10:08:41.634Z","dependency_job_id":"c5a4df54-ae20-41e7-b6bd-8e183c92c54b","html_url":"https://github.com/CSCfi/llm-fine-tuning-examples","commit_stats":null,"previous_names":["mvsjober/fine-tuning-examples","cscfi/llm-fine-tuning-examples"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fllm-fine-tuning-examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fllm-fine-tuning-examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fllm-fine-tuning-examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fllm-fine-tuning-examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CSCfi","download_url":"https://codeload.github.com/CSCfi/llm-fine-tuning-examples/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234480703,"owners_count":18840191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T12:34:33.913Z","updated_at":"2025-09-28T03:30:52.785Z","avatar_url":"https://github.com/CSCfi.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fine-tuning LLMs on supercomputers\n\nExample scripts showing how to fine-tune LLMs on CSC's supercomputers.\n\nThe script `finetuning.py` runs fine-tuning with the IMDb movie\nreviews dataset on a given Hugging Face model, by default it uses\n[EleutherAI/gpt-neo-1.3B](https://huggingface.co/EleutherAI/gpt-neo-1.3B)\nwhich fits comfortably into the GPU memory of a V100. You can select\nanother model with the `--model` argument.\n\nThe launch scripts are:\n\n- `run-finetuning-puhti-gpu1.sh` - fine-tuning on Puhti with 1 GPU\n- `run-finetuning-puhti-gpu4.sh` - fine-tuning on Puhti with one full node (4 GPUs)\n- `run-finetuning-puhti-gpu8.sh` - fine-tuning on Puhti with two full nodes (8 GPUs in total)\n- `run-finetuning-puhti-gpu4-accelerate.sh` - fine-tuning on Puhti with one full node using [Accelerate](https://huggingface.co/docs/transformers/accelerate)\n- `run-finetuning-puhti-gpu8-accelerate.sh` - fine-tuning on Puhti with two full nodes using Accelerate\n\nThere are also versions for Mahti, similarly named: just replace\n`puhti` with `mahti`. The scripts for LUMI are named slightly\ndifferent due to the larger number of GPUs (or actually GCDs, due to\nthe dual chip cards).\n\n- `run-finetuning-lumi-gpu1.sh` - fine-tuning on LUMI with 1 GPU\n- `run-finetuning-lumi-gpu8.sh` - fine-tuning on LUMI with one full node (8 GCDs)\n- `run-finetuning-lumi-gpu16.sh` - fine-tuning on LUMI with two full nodes (16 GCDs in total)\n- `run-finetuning-lumi-gpu8-accelerate.sh` - fine-tuning on LUMI with one full node using [Accelerate](https://huggingface.co/docs/transformers/accelerate)\n- `run-finetuning-lumi-gpu16-accelerate.sh` - fine-tuning on LUMI with two full nodes using Accelerate\n\n**Note:** the scripts are for the most part made to be run in the\n`gputest` or `dev-g` partition with a 15 minute time-limit. You\nnaturally need to change to the proper partition for longer jobs for\nyour real runs. Also change the `--account` parameter to your own\nproject code.\n\nYou can use [PEFT (Parameter-Efficient\nFine-Tuning)](https://huggingface.co/docs/peft/index) which adaptively\ntrains a smaller number of parameters, thus decreasing the GPU memory\nrequirements for training a lot. PEFT can be enabled with the `--peft`\nargument.\n\nThe [Accelerate](https://huggingface.co/docs/transformers/accelerate)\nlibrary supports more advanced modes of distributed training such as\n[FSDP](https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/)\nwhich enables using models which are too large for a single GPU's\nmemory.\n\nFinally, by installing the `bitsandbytes` library, you can also try\n4-bit quantization with the `--4bit` argument, decreasing even further\nthe memory requirements.\n\n## Run examples\n\nRun on 1 GPU with specified model and using PEFT:\n\n```bash\nsbatch run-finetuning-puhti-gpu1.sh --model=EleutherAI/gpt-neo-1.3B --peft\n```\n\nRun on 4 GPUs (note that batch_size has to be a multiple of the number of GPUs):\n```bash\nsbatch run-finetuning-puhti-gpu4.sh --model=EleutherAI/gpt-neo-1.3B --b 4\n```\n\nRun on 8 GPUs (over two nodes) with Accelerate and FSDP (note: with\nthe accelerate launch script we need to specify which config file to\nuse):\n\n```bash\nsbatch run-finetuning-puhti-gpu8-accelerate.sh accelerate_config_fsdp.yaml \\\n       --model=microsoft/Phi-3.5-mini-instruct --b 8\n```\n\n**Note:** for a new model it might work best if your first run is with\na single GPU to get the model downloaded to the cache. Downloading\nwith multiple processes doesn't yet work well in the current script.\n\nFine-tune Llama-3.1-8B on Mahti with just 2 GPUs using Accelerate,\nFSDP and PEFT:\n\n```bash\nsbatch run-finetuning-mahti-gpu2-accelerate.sh accelerate_config_fsdp.yaml \\\n       --model=meta-llama/Meta-Llama-3.1-8B --b 4 --peft\n```\n\nNote that the `Meta-LLama-3.1-8B` model is a \"Gated model\" on Hugging\nFace, it requires that you log in and ask for access to the\nmodel. Once you have recieved access you can generate an [Access Token\nin Hugging Face](https://huggingface.co/settings/tokens). Just click\n\"Create new token\" and select Token type: \"Read\".\n\nOn the supercomputer you can then install the access token like this:\n\n```bash\nexport HF_HOME=/scratch/YOUR_PROJECT/${USER}/hf-cache\nmodule load pytorch/2.4\nhuggingface-cli login\n```\n\nIn the above command you need to replace `YOUR_PROJECT` with the\nproject you use for your runs. The important thing is just that you\nuse the same Hugging Face cache path as in the scripts. The\n`huggingface-cli login` command will ask for the access token you\ncreated. Just reply `n` to the question about git credentials (unless\nyou know you use that feature).\n\nFine-tune Llama-3.1-8B with 4bit quantization and PEFT on Puhti:\n\n```bash\nsbatch run-finetuning-puhti-gpu1.sh --model=meta-llama/Meta-Llama-3.1-8B \\\n       --b 4 --peft --4bit\n```\n\nAnother example for fine-tuning the Poro model on LUMI with just 2\nGPUs (thanks to PEFT/LoRA and 4bit quantization):\n\n```bash\nsbatch run-finetuning-lumi-gpu2.sh --model=LumiOpen/Poro-34B --b 8 --4bit --peft\n```\n\n\n## Inference\n\nThere's also a example of inference (generating text with the model)\nin `inference-demo.py` with corresponding launch script\n`run-inference-puhti.sh`.\n\nFor example to run inference with a checkpoint of a model you have\nfine-tuned previously, you would run something like:\n\n```bash\nsbatch run-inference-puhti.sh --model=/PATH/TO/CHECKPOINT \\\n       --prompt=\"The movie was great because\"\n```\n\nNaturally, you need to replace `/PATH/TO/CHECKPOINT` with the real\npath to the checkpoint you wish to use. The path where checkpoints are\nstored will usually be printed at the end of the job, but you need to\ncheck yourself what is the specific checkpoint you wish to use. With\nthe above training scripts it will usually be something like\n`/scratch/YOUR_PROJECT/${USER}/hf-data/MODEL_NAME/checkpoint-NNN/`\nwhere `NNN` is the number of training steps when the checkpoint was\ncreated.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fllm-fine-tuning-examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcscfi%2Fllm-fine-tuning-examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fllm-fine-tuning-examples/lists"}