{"id":20229280,"url":"https://github.com/rahulunair/tiny_llm_finetuner","last_synced_at":"2025-05-07T10:32:26.760Z","repository":{"id":169520735,"uuid":"645472935","full_name":"rahulunair/tiny_llm_finetuner","owner":"rahulunair","description":"LLM finetuning on Intel XPUs - LoRA on intel discrete GPUs","archived":false,"fork":false,"pushed_at":"2023-10-31T03:57:01.000Z","size":23,"stargazers_count":10,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2023-10-31T04:30:35.179Z","etag":null,"topics":["gpu","intel","intel-arc","xpu"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rahulunair.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-05-25T18:25:15.000Z","updated_at":"2023-10-17T17:00:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"66c4369a-b847-47be-9600-3a5cc8908595","html_url":"https://github.com/rahulunair/tiny_llm_finetuner","commit_stats":null,"previous_names":["rahulunair/tiny_llm_finetuning"],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahulunair%2Ftiny_llm_finetuner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahulunair%2Ftiny_llm_finetuner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahulunair%2Ftiny_llm_finetuner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahulunair%2Ftiny_llm_finetuner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rahulunair","download_url":"https://codeload.github.com/rahulunair/tiny_llm_finetuner/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224586269,"owners_count":17335947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpu","intel","intel-arc","xpu"],"created_at":"2024-11-14T07:35:07.361Z","updated_at":"2024-11-14T07:35:08.172Z","avatar_url":"https://github.com/rahulunair.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Tiny llm Finetuner for Intel dGPUs\n\n\n\u003cimg src=\"https://github.com/rahulunair/tiny_llm_finetuning/assets/786476/2d967cb0-2a18-429b-8303-1257afe15ffc\" width=\"255\" height=\"255\"\u003e\n\n\n\n#### Finetuning openLLAMA on Intel discrete GPUS\n\nA finetuner\u003csup id=\"a1\"\u003e[1](#f1)\u003c/sup\u003e \u003csup id=\"a2\"\u003e[2](#f2)\u003c/sup\u003e for LLMs on Intel XPU devices, with which you could finetune the openLLaMA-3b model to sound like your favorite book.\n\n![image](https://github.com/rahulunair/tiny_llm_finetuning/assets/786476/f060f4f4-f85e-42e5-82c7-fb95fad932fd)\n\n\n#### Setup and activate conda env\n\n```bash\nconda env create -f env.yml\nconda activate pyt_llm_xpu\n```\n\n**Warning**: OncePyTorch and intel extension for PyTorch is already setup, then install peft without dependencies as peft requires PyTorch 2.0(not supported yet on Intel XPU devices.)\n\n#### Generate data\n\nFetch a book from guttenberg (default: pride and prejudice) and generate the dataset.\n\n```python\npython fetch_data.py\n```\n\n#### Finetune\n\n```bash\npython finetune.py --input_data ./book_data.json --batch_size=64 --micro_batch_size=16 --num_steps=300\n```\n\n#### Inference\n\nFor inference, you can either provide a input prompt, or the model will take a default prompt\n\n##### Without user provided prompt\n\n```bash\npython inference.py --infer\n```\n\n##### Using your own prompt for inference\n\n```bash\npython inference.py --infer --prompt \"my prompt\"\n```\n\n##### Benchmark Inference\n\n```bash\npython inference.py --bench\n```\n\u003cb id=\"f1\"\u003e1:\u003c/b\u003e adapted from: [source](https://github.com/modal-labs/doppel-bot/blob/main/src/finetune.py) [↩](#a1)  \n\u003cb id=\"f2\"\u003e2:\u003c/b\u003e adapted from: [source](https://github.com/tloen/alpaca-lora/blob/65fb8225c09af81feb5edb1abb12560f02930703/finetune.py) [↩](#a2)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahulunair%2Ftiny_llm_finetuner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frahulunair%2Ftiny_llm_finetuner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahulunair%2Ftiny_llm_finetuner/lists"}