{"id":14034886,"url":"https://github.com/LiuXiaoxuanPKU/OSD","last_synced_at":"2025-07-27T03:31:02.044Z","repository":{"id":243575841,"uuid":"673959226","full_name":"LiuXiaoxuanPKU/OSD","owner":"LiuXiaoxuanPKU","description":null,"archived":false,"fork":false,"pushed_at":"2024-06-11T02:11:29.000Z","size":96000,"stargazers_count":11,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-06-12T04:53:31.491Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LiuXiaoxuanPKU.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-02T20:17:44.000Z","updated_at":"2024-06-11T19:26:27.000Z","dependencies_parsed_at":"2024-06-09T23:37:33.019Z","dependency_job_id":"e02c94ab-8d31-4cfb-90f9-0e109edf2df6","html_url":"https://github.com/LiuXiaoxuanPKU/OSD","commit_stats":null,"previous_names":["liuxiaoxuanpku/osd"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiuXiaoxuanPKU%2FOSD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiuXiaoxuanPKU%2FOSD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiuXiaoxuanPKU%2FOSD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiuXiaoxuanPKU%2FOSD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LiuXiaoxuanPKU","download_url":"https://codeload.github.com/LiuXiaoxuanPKU/OSD/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":214982088,"owners_count":15811653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-12T03:01:10.611Z","updated_at":"2024-08-12T03:02:33.385Z","avatar_url":"https://github.com/LiuXiaoxuanPKU.png","language":"Python","funding_links":[],"categories":["Adaptive Computation for Black-box models","⚡ Speculative-Decoding Distillation"],"sub_categories":["🔁 Iterative Self-Bootstrapping"],"readme":"# [ICML2024] Online Speculative Decoding (OSD)\n\nThis is the official project repository for [Online Speculative Decoding](https://arxiv.org/pdf/2310.07177.pdf) by Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, Hao Zhang.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"assets/arch.png\" alt=\"Architecture\" width=\"800\"\u003e\u003c/p\u003e\n\n## TL;DR\nOnline speculative decoding (OSD) exploits spare FLOPs in LLM serving systems to refine the draft model through distillation. OSD improves the draft model's speculation accuracy and responsiveness to query distribution shifts, translating into at most 3.06x latency reduction, without sacrificing any user experience. \n\n## Abstract\nSpeculative decoding is a pivotal technique to accelerate the inference of large language models (LLMs) by employing a smaller draft model to predict the target model’s outputs. However, its efficacy can be limited due to the low predictive accuracy of the draft model, particularly when faced with diverse text inputs and a significant capability gap between the draft and target models. We introduce online speculative decoding to address this challenge. The main idea is to continually update (multiple) draft model(s) on observed user query data using the abundant excess computational power in an LLM serving cluster. Given that LLM inference is memory-bounded, the surplus computational power in a typical LLM serving cluster can be repurposed for online retraining of draft models, thereby making the training cost-neutral. Since the query distribution of an LLM service is relatively simple, retraining on query distribution enables the draft model to more accurately predict the target model’s outputs, particularly on data originating from query distributions. As the draft model evolves online, it aligns with the query distribution in real time, mitigating distribution shifts. We develop a prototype of online speculative decoding based on online knowledge distillation and evaluate it using both synthetic and real query data on several popular LLMs. The results show a substantial increase in the token acceptance rate by 0.1 to 0.65, which translates into 1.22× to 3.06× latency reduction.\n\n## Install\n1. Environment setup:\n```\nconda create -n osd python=3.8\nconda activate osd\n```\n2. Clone this repository and build from source:\n```\ngit clone https://github.com/LiuXiaoxuanPKU/OSD\ncd OSD\n```\n3. Install dependency:\n```\npip install -r requirements.txt\n```\n\n## Usage\n### Prepare data\n```\ncd data\nmkdir raw_data\npython clean_{dataset}.py\n```\ndataset can take the value of `spider`, `finance`, `code_search`, `gsm8k`.\n\n### Download teacher \u0026 draft model\n```\n# download teacher model to your path\n# download draft model, example: go to ./data/\ngit clone https://huggingface.co/JackFram/llama-160m\n```\n\n### LLaMA\n0. Prepare dataset (if you want to run offline experiments):\n```\n# under ./data/\npython generate_answer.py --filename {your_cleaned_file_path} --model {teacher_model_path}\n```\n1. Run online distillation:\n```\nbash bash_scripts/{dataset_name}/online.sh {your_savepath} {sample_source} {distillation_method}\n```\nAn example will be:\n```\nbash bash_scripts/spider/online.sh ./output/ teacher forward\n```\nFor example, running the command above on the `spider` dataset first 100 data records, you should see similar alphas (token acceptance rates)/loss on `wandb` as below:\n\u003cp align=\"center\"\u003e\u003cimg src=\"assets/spider.png\" alt=\"Spider Online\" width=\"800\"\u003e\u003c/p\u003e\n\n2. Run offline distillation:\n```\nbash bash_scripts/{dataset_name}/offline.sh {your_savepath} {sample_source} {distillation_method}\n```\nAn example will be:\n```\nbash bash_scripts/spider/offline.sh ./output/ teacher forward\n```\nNotice the current script will run evaluation across the whole eval dataset after each epoch, which might be very slow.\nIf you just want to distill the model without evaluation, feel free to change the evaluation strategy in the `offline.sh` script to `no`:\n```\n--evaluation_strategy \"no\"\n```\n\n\n\n### T5\n1. Customized offline distillation:\n```\nbash bash_scripts/t5/offline.sh {your_savepath} {dataset_name} {sample_source} {distillation_method}\n```\n2. Customized online distillation:\n```\nbash bash_scripts/t5/online.sh {your_savepath} {dataset_name} {sample_source} {distillation_method}\n```\n\n### Command options\n```\n--student_model_path: path to the student (small) model\n--teacher_model_path: path to the teacher (big) model\n--mode: distillation mode. Select one from {online, offline} \\\n--sample_source: sampling methods. Select one from {teacher, student, mix_token, mix_request} \\\n--kl_method: distillation methods. Select one from {forward, reverse, jsd} \\\n```\n\n\n### Datasets\nThis repo currently supports distillation and evaluation on the following datasets:\n\nModels | GSM8K | Spider | Finance-Alpaca | CSN Python | PIQA | Starcode | Arena | CNN Dailymail | Xsum |\n:---: | :---: | :---: | :---: | :---: | :---: | :---: |:---: | :---: | :---: |\n LLaMA | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |  |  |\nT5 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |  |  |  | :heavy_check_mark: | :heavy_check_mark: |\n\n### Evaluations\n\n1. We conduct our latency experiments with [llamacpp](https://github.com/ggerganov/llama.cpp) on a single A100-80G. To reproduce the results, please first clone the repository:\n```\n# go back to the project root\ngit clone https://github.com/ggerganov/llama.cpp.git\n```\n\n2. Copy your cleaned data files from `data/raw_data` to `llama.cpp/data/raw_data`.\n\n3. Set up [cpp json processing](https://github.com/nlohmann/json). Copy `nlohmann/json.hpp` to your cpp include path.\n\n4. Apply the following patches:\n```\ncp -r llamacpp_patches llama.cpp/\ncp -rv llamacpp_patches/speculative.cpp llama.cpp/examples/speculative/speculative.cpp\n\n# replace line 65 in speculative.cpp with the path to your cleaned test file.\n```\n\n5. Follow llamacpp's README to build the project with cuBLAS support.\n\n6. Prepare models:\n```\nbash llama.cpp/llamacpp_patches/download_models_160m.sh\nbash llama.cpp/llamacpp_patches/download_models.sh\nmv llama.cpp/llamacpp_patches/quantize_160m.sh llama.cpp/build \u0026\u0026 bash llama.cpp/build/quantize_160m.sh\nmv llama.cpp/llamacpp_patches/quantize.sh llama.cpp/build \u0026\u0026 bash llama.cpp/build/quantize.sh\n```\n\n7. Run conventional llamacpp speculative decoding inference to stream the dataset and collect runtime data for all queries. \n\n### Model Weights\n\n#### Distilled draft models \n\n| Size | Dataset |  Huggingface Repo                             |\n| ---- | -------- | --------------------------------------------- | \n| 160m   | Finance-Alpaca |  [https://huggingface.co/eqhylxx/finance-llama160m](https://huggingface.co/eqhylxx/finance-llama160m)   |\n| 160m  | GSM8K | [https://huggingface.co/eqhylxx/gsm8k-llama160m](https://huggingface.co/eqhylxx/gsm8k-llama160m) |\n| 160m  | Spider | [https://huggingface.co/eqhylxx/spider-llama160m](https://huggingface.co/eqhylxx/spider-llama160m) |\n| 160m  | Code-Search-Net Python | [https://huggingface.co/eqhylxx/code-llama160m](https://huggingface.co/eqhylxx/code-llama160m) |\n| 160m  | ShareGPT | https://huggingface.co/eqhylxx/vicuna-160m |\n\n## Citation\nThis is the official project repository for the following paper. If you find this repository helpful, Please cite:\n```\n@misc{liu2024online,\n      title={Online Speculative Decoding}, \n      author={Xiaoxuan Liu and Lanxiang Hu and Peter Bailis and Alvin Cheung and Zhijie Deng and Ion Stoica and Hao Zhang},\n      year={2024},\n      eprint={2310.07177},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLiuXiaoxuanPKU%2FOSD","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLiuXiaoxuanPKU%2FOSD","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLiuXiaoxuanPKU%2FOSD/lists"}