{"id":19482614,"url":"https://github.com/predibase/lora_bakeoff","last_synced_at":"2025-08-09T01:42:39.886Z","repository":{"id":229045944,"uuid":"774030332","full_name":"predibase/lora_bakeoff","owner":"predibase","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-26T15:36:08.000Z","size":4712,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-28T15:57:28.357Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/predibase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-18T20:26:17.000Z","updated_at":"2024-04-26T15:36:12.000Z","dependencies_parsed_at":"2024-03-26T15:57:34.265Z","dependency_job_id":null,"html_url":"https://github.com/predibase/lora_bakeoff","commit_stats":null,"previous_names":["predibase/lora_bakeoff"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/predibase%2Flora_bakeoff","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/predibase%2Flora_bakeoff/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/predibase%2Flora_bakeoff/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/predibase%2Flora_bakeoff/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/predibase","download_url":"https://codeload.github.com/predibase/lora_bakeoff/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224007933,"owners_count":17240294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T20:11:30.247Z","updated_at":"2024-11-10T20:11:31.505Z","avatar_url":"https://github.com/predibase.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lora Land: 310 Fine-tuned LLMs that Rival GPT-4\n\n- Full technical report. (TODO: Add arxiv link).\n- [LoRA Land web application](https://predibase.com/lora-land).\n- [LoRA Land blog post](https://predibase.com/blog/lora-land-fine-tuned-open-source-llms-that-outperform-gpt-4).\n\n![img](img/header.png)\n\nLow Rank Adaptation (LoRA) has emerged as one of the most widely adopted methods\nfor Parameter Efficient Fine-Tuning (PEFT) of Large Language Models (LLMs). LoRA\nreduces the number of trainable parameters and memory usage while achieving comparable\nperformance to full fine-tuning. We aim to assess the viability of training and serving LLMs\nfine-tuned with LoRA in real-world applications. First, we measure the quality of LLMs\nfine-tuned with quantized low rank adapters across 10 base models and 31 tasks for a total of\n310 models. We find that 4-bit LoRA fine-tuned models outperform base models by 34 points\nand GPT-4 by 10 points on average. Second, we investigate the most effective base models\nfor fine-tuning and assess the correlative and predictive capacities of dataset complexity\nheuristics in forecasting the outcomes of fine-tuning. Finally, we evaluate the latency and\nconcurrency capabilities of [LoRAX](https://github.com/predibase/lorax), an open-source LLM serving framework that facilitates\nthe deployment of multiple LoRA fine-tuned models on a single GPU using shared base\nmodel weights and dynamic adapter loading. We use LoRAX to develop [LoRA Land](https://predibase.com/lora-land), a\nweb application that hosts 25 LoRA fine-tuned Mistral-7B LLMs on a single NVIDIA A100\nGPU with 80GB memory. This demonstration highlights the quality and cost-effectiveness\nof employing multiple specialized LLMs over a single, general-purpose LLM.\n\n## Tasks\n\nThe preprocessing code, prompt templates, and splits for all of our experiments can be found in the `datasets/` directory.\n\n![img](img/tasks.png)\n\n## Prompt design\n\n![img](img/prompt_design.png)\n\n![img](img/prompt_examples.png)\n\n## Base models\n\n![img](img/base_models.png)\n\n## Training configuration template\n\n![img](img/training_config.png)\n\nBased on [Ludwig](https://ludwig.ai/latest/).\n\n## Results\n\n![img](img/performance_lift.png)\n\n![img](img/full_table.png)\n\n## Meta-correlations betweendataset complexity and model quality\n\n![img](img/meta_correlations.png)\n\n## Evaluation Harness\n\nTo use this eval harness, please set your `PREDIBASE_API_TOKEN` in `.env` before running the harness.\n\nIn the `tasks` directory, there is a subdirectory named `glue_sst2`, and inside of that subdirectory, there is a file named `metadata.yaml`. This is an example of the task-based organizational structure that the harness relies on. To add another task to the directory, it must follow the same convention: `tasks/{task_name}/metadata.yaml`.\n\nThe `metadata.yaml` file includes metadata like:\n- `data_path`: the path to the relevant dataset\n- `prompt_template`: the prompt template\n- `target_col`: the target column\n- `metric_name`: the metric function to use\n- `split_column`: the name of the column that defines splits in the dataset (optional)\n- `split_column_value`:which split to use (optional)\n\nIn the `eval/glue_sst2` directory, the `run.sh` script serves as an example script to follow for other evaluations. There are three components of the script:\n- `prep_pbase_requests.py` -- Generates the JSON payloads to the REST API\n- `pbase_request_parallel_processor.py` -- Calls the REST API to get the responses from an adapter\n- `parse_responses.py` -- Calculates a metric score over the responses from the adapter (metrics listed in `metric_fns.py`)\n\nIn particular, the `task` flag refers to the name of the subdirectory containing the relevant `metadata.yaml` file.\n\nTo run the `run.sh` script, you will need to:\n1. Train your own adapter on the provided dataset and switch out the `adapter_id` parameter in the script with the model repo name and model version of your fine-tuned adapter.\n2. Change the `deployment_base_model` parameter to the name of the deployment you would like to use\n3. Change the `tenant_id` parameter to your tenant ID.\n\nThese three parameters can be found in the Predibase UI under \"Models\", \"Prompt\", and \"Settings\", respectively.\n\n## LoRAX Benchmarking\n\nSee `lorax_load_test.js`.\n\nSample command:\n\n```\nk6 run --env CONCURRENT_REQUESTS=2 --env NUM_INPUT_WORDS_LOWER_BOUND=90 --env NUM_INPUT_WORDS_UPPER_BOUND=110 --env MAX_NEW_TOKENS_LOWER_BOUND=90 --env MAX_NEW_TOKENS_UPPER_BOUND=110 --env SERVING_GATEWAY=serving.app.predibase.com --env TENANT=fd6c79 --env DEPLOYMENT_NAME=llama-2-7b-chat --env AUTH_TOKEN=pb_jcN0OPMdWt-yrgIg0aBnTA load_test.js\n```\n\n## Web application\n\n[LoRA Land](https://predibase.com/lora-land)\n\n![img](img/lora_land.png)\n\n\n## Citation\n\n```\n@misc{loraland2024,\n    title = {LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report},\n    url = {https://predibase.com/blog/lora-land-fine-tuned-open-source-llms-that-outperform-gpt-4},\n    author = {Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Piero Molino, Travis Addair, Devvret Rishi},\n    month = {April},\n    year = {2024}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpredibase%2Flora_bakeoff","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpredibase%2Flora_bakeoff","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpredibase%2Flora_bakeoff/lists"}