{"id":13655382,"url":"https://github.com/zetavg/LLaMA-LoRA-Tuner","last_synced_at":"2025-04-23T17:30:34.829Z","repository":{"id":152286315,"uuid":"622943402","full_name":"zetavg/LLaMA-LoRA-Tuner","owner":"zetavg","description":"UI tool for fine-tuning and testing your own LoRA models base on LLaMA, GPT-J and more. One-click run on Google Colab. + A Gradio ChatGPT-like Chat UI to demonstrate your language models.","archived":false,"fork":false,"pushed_at":"2023-05-29T23:27:51.000Z","size":1032,"stargazers_count":442,"open_issues_count":23,"forks_count":88,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-10-30T01:01:51.455Z","etag":null,"topics":["ai","alpaca","alpaca-lora","google-colab","gpt","gpt-j","language-model","llama","lora","machine-learning","peft"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zetavg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-04-03T11:34:52.000Z","updated_at":"2024-10-29T04:50:23.000Z","dependencies_parsed_at":"2023-09-18T17:17:17.246Z","dependency_job_id":null,"html_url":"https://github.com/zetavg/LLaMA-LoRA-Tuner","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zetavg%2FLLaMA-LoRA-Tuner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zetavg%2FLLaMA-LoRA-Tuner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zetavg%2FLLaMA-LoRA-Tuner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zetavg%2FLLaMA-LoRA-Tuner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zetavg","download_url":"https://codeload.github.com/zetavg/LLaMA-LoRA-Tuner/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223758397,"owners_count":17197713,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","alpaca","alpaca-lora","google-colab","gpt","gpt-j","language-model","llama","lora","machine-learning","peft"],"created_at":"2024-08-02T04:00:18.199Z","updated_at":"2024-11-10T08:30:15.969Z","avatar_url":"https://github.com/zetavg.png","language":"Python","funding_links":[],"categories":["Fine Tuning ([home](#awesome-llm))","A01_文本生成_文本对话","Python"],"sub_categories":["大语言对话模型及数据"],"readme":"# 🦙🎛️ LLaMA-LoRA Tuner\n\n\u003ca href=\"https://colab.research.google.com/github/zetavg/LLaMA-LoRA-Tuner/blob/main/LLaMA_LoRA.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e\n\nMaking evaluating and fine-tuning LLaMA models with low-rank adaptation (LoRA) easy.\n\n\u003e **Update**:\n\u003e \n\u003e On the `dev` branch, there's a new Chat UI and a new *Demo Mode* config as a simple and easy way to demonstrate new models. \n\u003e \n\u003e However, the new version does not have the fine-tuning feature yet and is not backward compatible as it uses a new way to define how models are loaded, and also a new format of prompt templates (from [LangChain](https://github.com/hwchase17/langchain)).\n\u003e\n\u003e For more info, see: https://github.com/zetavg/LLaMA-LoRA-Tuner/discussions/28.\n\u003e \n\u003e https://github.com/zetavg/LLaMA-LoRA-Tuner/assets/3784687/ae81a5ed-fe8b-4b17-bea1-455837c2e909\n\n## Features\n\n**[See a demo on Hugging Face](https://huggingface.co/spaces/zetavg/LLaMA-LoRA-UI-Demo)** **Only serves UI demonstration. To try training or text generation, [run on Colab](#run-on-google-colab).*\n\n* **[1-click up and running in Google Colab](#run-on-google-colab)** with a standard GPU runtime.\n  * Loads and stores data in Google Drive.\n* Evaluate various LLaMA LoRA models stored in your folder or from Hugging Face.\u003cbr /\u003e\u003ca href=\"https://youtu.be/IoEMgouZ5xU\"\u003e\u003cimg width=\"640px\" src=\"https://user-images.githubusercontent.com/3784687/231023326-f28c84e2-df74-4179-b0ac-c25c4e8ca001.gif\" /\u003e\u003c/a\u003e\n* Switch between base models such as `decapoda-research/llama-7b-hf`, `nomic-ai/gpt4all-j`, `databricks/dolly-v2-7b`, `EleutherAI/gpt-j-6b`, or `EleutherAI/pythia-6.9b`.\n* Fine-tune LLaMA models with different prompt templates and training dataset format.\u003cbr /\u003e\u003ca href=\"https://youtu.be/IoEMgouZ5xU?t=60\"\u003e\u003cimg width=\"640px\" src=\"https://user-images.githubusercontent.com/3784687/231026640-b5cf5c79-9fe9-430b-8d4e-7346eb9567ad.gif\" /\u003e\u003c/a\u003e\n  * Load JSON and JSONL datasets from your folder, or even paste plain text directly into the UI.\n  * Supports Stanford Alpaca [seed_tasks](https://github.com/tatsu-lab/stanford_alpaca/blob/main/seed_tasks.jsonl), [alpaca_data](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) and [OpenAI \"prompt\"-\"completion\"](https://platform.openai.com/docs/guides/fine-tuning/data-formatting) format.\n  * Use prompt templates to keep your dataset DRY.\n\n\n## How to Start\n\nThere are various ways to run this app:\n\n* **[Run on Google Colab](#run-on-google-colab)**: The simplest way to get started, all you need is a Google account. Standard (free) GPU runtime is sufficient to run generation and training with micro batch size of 8. However, the text generation and training is much slower than on other cloud services, and Colab might terminate the execution in inactivity while running long tasks.\n* **[Run on a cloud service via SkyPilot](#run-on-a-cloud-service-via-skypilot)**: If you have a cloud service (Lambda Labs, GCP, AWS, or Azure) account, you can use SkyPilot to run the app on a cloud service. A cloud bucket can be mounted to preserve your data.\n* **[Run locally](#run-locally)**: Depends on the hardware you have.\n\n### Run On Google Colab\n\n*See [video](https://youtu.be/lByYOMdy9h4) for step-by-step instructions.*\n\nOpen [this Colab Notebook](https://colab.research.google.com/github/zetavg/LLaMA-LoRA-Tuner/blob/main/LLaMA_LoRA.ipynb) and select **Runtime \u003e Run All** (`⌘/Ctrl+F9`).\n\nYou will be prompted to authorize Google Drive access, as Google Drive will be used to store your data. See the \"Config\"/\"Google Drive\" section for settings and more info.\n\nAfter approximately 5 minutes of running, you will see the public URL in the output of the \"Launch\"/\"Start Gradio UI 🚀\" section (like `Running on public URL: https://xxxx.gradio.live`). Open the URL in your browser to use the app.\n\n### Run on a cloud service via SkyPilot\n\nAfter following the [installation guide of SkyPilot](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html), create a `.yaml` to define a task for running the app:\n\n```yaml\n# llm-tuner.yaml\n\nresources:\n  accelerators: A10:1  # 1x NVIDIA A10 GPU, about US$ 0.6 / hr on Lambda Cloud. Run `sky show-gpus` for supported GPU types, and `sky show-gpus [GPU_NAME]` for the detailed information of a GPU type.\n  cloud: lambda  # Optional; if left out, SkyPilot will automatically pick the cheapest cloud.\n\nfile_mounts:\n  # Mount a presisted cloud storage that will be used as the data directory.\n  # (to store train datasets trained models)\n  # See https://skypilot.readthedocs.io/en/latest/reference/storage.html for details.\n  /data:\n    name: llm-tuner-data  # Make sure this name is unique or you own this bucket. If it does not exists, SkyPilot will try to create a bucket with this name.\n    store: s3  # Could be either of [s3, gcs]\n    mode: MOUNT\n\n# Clone the LLaMA-LoRA Tuner repo and install its dependencies.\nsetup: |\n  conda create -q python=3.8 -n llm-tuner -y\n  conda activate llm-tuner\n\n  # Clone the LLaMA-LoRA Tuner repo and install its dependencies\n  [ ! -d llm_tuner ] \u0026\u0026 git clone https://github.com/zetavg/LLaMA-LoRA-Tuner.git llm_tuner\n  echo 'Installing dependencies...'\n  pip install -r llm_tuner/requirements.lock.txt\n\n  # Optional: install wandb to enable logging to Weights \u0026 Biases\n  pip install wandb\n\n  # Optional: patch bitsandbytes to workaround error \"libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats\"\n  BITSANDBYTES_LOCATION=\"$(pip show bitsandbytes | grep 'Location' | awk '{print $2}')/bitsandbytes\"\n  [ -f \"$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so\" ] \u0026\u0026 [ ! -f \"$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so.bak\" ] \u0026\u0026 [ -f \"$BITSANDBYTES_LOCATION/libbitsandbytes_cuda121.so\" ] \u0026\u0026 echo 'Patching bitsandbytes for GPU support...' \u0026\u0026 mv \"$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so\" \"$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so.bak\" \u0026\u0026 cp \"$BITSANDBYTES_LOCATION/libbitsandbytes_cuda121.so\" \"$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so\"\n  conda install -q cudatoolkit -y\n\n  echo 'Dependencies installed.'\n\n  # Optional: Install and setup Cloudflare Tunnel to expose the app to the internet with a custom domain name\n  [ -f /data/secrets/cloudflared_tunnel_token.txt ] \u0026\u0026 echo \"Installing Cloudflare\" \u0026\u0026 curl -L --output cloudflared.deb https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb \u0026\u0026 sudo dpkg -i cloudflared.deb \u0026\u0026 sudo cloudflared service uninstall || : \u0026\u0026 sudo cloudflared service install \"$(cat /data/secrets/cloudflared_tunnel_token.txt | tr -d '\\n')\"\n\n  # Optional: pre-download models\n  echo \"Pre-downloading base models so that you won't have to wait for long once the app is ready...\"\n  python llm_tuner/download_base_model.py --base_model_names='decapoda-research/llama-7b-hf,nomic-ai/gpt4all-j'\n\n# Start the app. `hf_access_token`, `wandb_api_key` and `wandb_project` are optional.\nrun: |\n  conda activate llm-tuner\n  python llm_tuner/app.py \\\n    --data_dir='/data' \\\n    --hf_access_token=\"$([ -f /data/secrets/hf_access_token.txt ] \u0026\u0026 cat /data/secrets/hf_access_token.txt | tr -d '\\n')\" \\\n    --wandb_api_key=\"$([ -f /data/secrets/wandb_api_key.txt ] \u0026\u0026 cat /data/secrets/wandb_api_key.txt | tr -d '\\n')\" \\\n    --wandb_project='llm-tuner' \\\n    --timezone='Atlantic/Reykjavik' \\\n    --base_model='decapoda-research/llama-7b-hf' \\\n    --base_model_choices='decapoda-research/llama-7b-hf,nomic-ai/gpt4all-j,databricks/dolly-v2-7b' \\\n    --share\n```\n\nThen launch a cluster to run the task:\n\n```\nsky launch -c llm-tuner llm-tuner.yaml\n```\n\n`-c ...` is an optional flag to specify a cluster name. If not specified, SkyPilot will automatically generate one.\n\nYou will see the public URL of the app in the terminal. Open the URL in your browser to use the app.\n\nNote that exiting `sky launch` will only exit log streaming and will not stop the task. You can use `sky queue --skip-finished` to see the status of running or pending tasks, `sky logs \u003ccluster_name\u003e \u003cjob_id\u003e` connect back to log streaming, and `sky cancel \u003ccluster_name\u003e \u003cjob_id\u003e` to stop a task.\n\nWhen you are done, run `sky stop \u003ccluster_name\u003e` to stop the cluster. To terminate a cluster instead, run `sky down \u003ccluster_name\u003e`.\n\n**Remember to stop or shutdown the cluster when you are done to avoid incurring unexpected charges.** Run `sky cost-report` to see the cost of your clusters.\n\n\u003cdetails\u003e\n  \u003csummary\u003eLog into the cloud machine or mount the filesystem of the cloud machine on your local computer\u003c/summary\u003e\n\n  To log into the cloud machine, run `ssh \u003ccluster_name\u003e`, such as `ssh llm-tuner`.\n\n  If you have `sshfs` installed on your local machine, you can mount the filesystem of the cloud machine on your local computer by running a command like the following:\n\n  ```bash\n  mkdir -p /tmp/llm_tuner_server \u0026\u0026 umount /tmp/llm_tuner_server || : \u0026\u0026 sshfs llm-tuner:/ /tmp/llm_tuner_server\n  ```\n\u003c/details\u003e\n\n### Run locally\n\n\u003cdetails\u003e\n  \u003csummary\u003ePrepare environment with conda\u003c/summary\u003e\n\n  ```bash\n  conda create -y python=3.8 -n llm-tuner\n  conda activate llm-tuner\n  ```\n\u003c/details\u003e\n\n```bash\npip install -r requirements.lock.txt\npython app.py --data_dir='./data' --base_model='decapoda-research/llama-7b-hf' --timezone='Atlantic/Reykjavik' --share\n```\n\nYou will see the local and public URLs of the app in the terminal. Open the URL in your browser to use the app.\n\nFor more options, see `python app.py --help`.\n\n\u003cdetails\u003e\n  \u003csummary\u003eUI development mode\u003c/summary\u003e\n\n  To test the UI without loading the language model, use the `--ui_dev_mode` flag:\n\n  ```bash\n  python app.py --data_dir='./data' --base_model='decapoda-research/llama-7b-hf' --share --ui_dev_mode\n  ```\n\n  \u003e To use [Gradio Auto-Reloading](https://gradio.app/developing-faster-with-reload-mode/#python-ide-reload), a `config.yaml` file is required since command line arguments are not supported. There's a sample file to start with: `cp config.yaml.sample config.yaml`. Then, just run `gradio app.py`.\n\u003c/details\u003e\n\n\n## Usage\n\nSee [video on YouTube](https://youtu.be/IoEMgouZ5xU).\n\n\n## Acknowledgements\n\n* https://github.com/tloen/alpaca-lora\n* https://github.com/lxe/simple-llama-finetuner\n* ...\n\nTBC\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzetavg%2FLLaMA-LoRA-Tuner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzetavg%2FLLaMA-LoRA-Tuner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzetavg%2FLLaMA-LoRA-Tuner/lists"}