{"id":14964767,"url":"https://github.com/autonomicperfectionist/pipeinfer","last_synced_at":"2025-10-05T10:30:40.209Z","repository":{"id":233359233,"uuid":"784247294","full_name":"AutonomicPerfectionist/PipeInfer","owner":"AutonomicPerfectionist","description":"PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation","archived":false,"fork":false,"pushed_at":"2024-11-16T21:15:49.000Z","size":18299,"stargazers_count":24,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-24T03:32:28.833Z","etag":null,"topics":["inference","llamacpp","llm","speculative-decoding"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AutonomicPerfectionist.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-09T13:30:10.000Z","updated_at":"2025-01-19T12:41:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"711d8bc1-6cf3-4511-a9c7-49e4ea0ba532","html_url":"https://github.com/AutonomicPerfectionist/PipeInfer","commit_stats":{"total_commits":1641,"total_committers":411,"mean_commits":"3.9927007299270074","dds":0.7489335770871419,"last_synced_commit":"123a824d9b53e56efa2a7645894d30a9a1135ed2"},"previous_names":["autonomicperfectionist/pipeinfer"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutonomicPerfectionist%2FPipeInfer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutonomicPerfectionist%2FPipeInfer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutonomicPerfectionist%2FPipeInfer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutonomicPerfectionist%2FPipeInfer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AutonomicPerfectionist","download_url":"https://codeload.github.com/AutonomicPerfectionist/PipeInfer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235383702,"owners_count":18981198,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["inference","llamacpp","llm","speculative-decoding"],"created_at":"2024-09-24T13:33:44.982Z","updated_at":"2025-10-05T10:30:38.666Z","avatar_url":"https://github.com/AutonomicPerfectionist.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation\n\n[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)\n\nBased on llama.cpp, inference of [LLaMA](https://arxiv.org/abs/2302.13971) model in pure C/C++.\n----\n\n\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#description\"\u003eDescription\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#get-the-code\"\u003eGet the Code\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#build\"\u003eBuild\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#run\"\u003eRun\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#tuning-pipeinfer\"\u003eTuning PipeInfer\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n## Description\n\nPipeInfer is a modification of [llama.cpp](https://github.com/ggerganov/llama.cpp) that is designed\nfor acceleration of inference across multi-node clusters using pipelined asynchronous speculation.\n\n- Supports Llama/Llama2, Falcon, Baichuan, Starcoder, Bloom, and other architectures.\n- Based on llama.cpp commit [9656026](https://github.com/ggerganov/llama.cpp/commit/9656026b53236ed7328458269c4c798dd50ac8d1)\n- See progress on porting to the latest version of llama.cpp [here](https://github.com/AutonomicPerfectionist/llama.cpp/tree/mpi-heterogenous)\n\n**Supported platforms:**\n\n- [X] Mac OS\n- [X] Linux\n- [X] Docker\n\n**Supported models:**\n\n- [X] LLaMA 🦙\n- [x] LLaMA 2 🦙🦙\n- [X] Falcon\n- [X] [Alpaca](https://github.com/ggerganov/llama.cpp#instruction-mode-with-alpaca)\n- [X] [GPT4All](https://github.com/ggerganov/llama.cpp#using-gpt4all)\n- [X] [Chinese LLaMA / Alpaca](https://github.com/ymcui/Chinese-LLaMA-Alpaca) and [Chinese LLaMA-2 / Alpaca-2](https://github.com/ymcui/Chinese-LLaMA-Alpaca-2)\n- [X] [Vigogne (French)](https://github.com/bofenghuang/vigogne)\n- [X] [Vicuna](https://github.com/ggerganov/llama.cpp/discussions/643#discussioncomment-5533894)\n- [X] [Koala](https://bair.berkeley.edu/blog/2023/04/03/koala/)\n- [X] [OpenBuddy 🐶 (Multilingual)](https://github.com/OpenBuddy/OpenBuddy)\n- [X] [Pygmalion/Metharme](#using-pygmalion-7b--metharme-7b)\n- [X] [WizardLM](https://github.com/nlpxucan/WizardLM)\n- [X] [Baichuan 1 \u0026 2](https://huggingface.co/models?search=baichuan-inc/Baichuan) + [derivations](https://huggingface.co/hiyouga/baichuan-7b-sft)\n- [X] [Aquila 1 \u0026 2](https://huggingface.co/models?search=BAAI/Aquila)\n- [X] [Starcoder models](https://github.com/ggerganov/llama.cpp/pull/3187)\n- [X] [Mistral AI v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)\n- [X] [Refact](https://huggingface.co/smallcloudai/Refact-1_6B-fim)\n- [X] [Persimmon 8B](https://github.com/ggerganov/llama.cpp/pull/3410)\n- [X] [MPT](https://github.com/ggerganov/llama.cpp/pull/3417)\n- [X] [Bloom](https://github.com/ggerganov/llama.cpp/pull/3553)\n- [X] [StableLM-3b-4e1t](https://github.com/ggerganov/llama.cpp/pull/3586)\n\n\n---\n## Usage\n\nPipeInfer is implemented in the `speculative` example binary. To run PipeInfer follow these steps.\n\n### Get the Code\n\n```bash\ngit clone https://github.com/AutonomicPerfectionist/PipeInfer\ncd PipeInfer\n```\n\n### Install Dependencies\n\nMake sure you have Make or CMake installed, as well as an MPI implementation and compiler.\nOn Debian-based systems these can be installed with the following:\n\n```bash\n$ sudo apt install build-essential cmake openmpi-bin libopenmpi-dev\n```\n\n### Build\n\nTo build PipeInfer you have two different options.\n\n- Using `make`:\n  - On Linux or MacOS:\n\n      ```bash\n      make speculative CC=mpicc CXX=mpicxx LLAMA_MPI=ON -j\n      ```\n\n- Using `CMake`:\n\n    ```bash\n    mkdir build\n    cd build\n    cmake .. -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DLLAMA_MPI=1\n    cmake --build . --target speculative --config Release\n    ```\n\n\n### Download Models\nPipeInfer requires two models to be downloaded: the large target model\nand a smaller speculative model. Both models must have compatible vocabulary,\nminor differences are allowed but may cause performance degradation. The models\nused in our paper can be downloaded from the following links:\n\n- [Dolphin 70B](https://huggingface.co/TheBloke/Dolphin-2.2-70B-GGUF)\n- [TinyLlama OpenOrca 1.1B](https://huggingface.co/TheBloke/TinyLlama-1.1B-1T-OpenOrca-GGUF)\n- [Orca2](https://huggingface.co/TheBloke/Orca-2-7B-GGUF)\n- [Goliath 120B](https://huggingface.co/TheBloke/goliath-120b-GGUF)\n- [XWinLM 7B](https://huggingface.co/TheBloke/Xwin-LM-7B-V0.2-GGUF)\n- [XWinLM 13B](https://huggingface.co/TheBloke/Xwin-LM-13B-v0.2-GGUF)\n- [Falcon 180B](https://huggingface.co/TheBloke/Falcon-180B-GGUF)\n- [Falcon 7B](https://huggingface.co/maddes8cht/tiiuae-falcon-7b-gguf)\n- [Falcon 40B](https://huggingface.co/maddes8cht/tiiuae-falcon-40b-gguf)\n\n\n\n\n### Run\nTo run PipeInfer, make sure you have MPI configured on all nodes, and that the head node can SSH into all other nodes.\nIt is recommended to place the model files on shared storage to avoid replicating them to each node. We have had success\nusing a dedicated NFS file server. By default `llama.cpp` loads the models by `mmap`'ing them, and only\nthe tensors needed by the layers assigned to a node will be faulted in. For multi-socket systems, it is recommended to clear the Linux file cache\nwhenever the layer allocations are changed, as otherwise the tensors may be kept on the wrong NUMA node.\n\nThis can be done with the following command:\n\n```bash\n$ echo 3 | sudo tee /proc/sys/vm/drop_caches\n```\n\nAlternatively, using the `--no-mmap` switch when running PipeInfer/llama.cpp will read the file contents into RAM directly, bypassing the file cache.\nNote that doing so forces the entire model to be loaded into RAM, potentially exceeding the memory capacity of the node.\n\nTo execute PipeInfer on a cluster, ensure that each node has access to the `speculative` binary and the model file. Like any other MPI program,\nit can be launched with `mpirun` or through job managers like Slurm. PipeInfer also requires the layer allocations to be passed in on the command line,\nthrough the `--mpi-layer-split` argument. For each node in the cluster, one must pass in a floating-point value denoting the percentage\nof layers the node will handle. When using multiple models, as in the case of PipeInfer, one must also specify how to split the\nnodes into two communicators. This split is denoted by a slash `/`. All percentages in the first half are tied to the target model, and percentages after\nthe slash are tied to the speculative model. An example: if there are 5 nodes total, and we want 4 nodes to handle the target model and one node to handle\nthe speculative model, with even splits among the target nodes, we would pass the following:\n\n```bash\n--mpi-layer-split 0.25,0.25,0.25,0.25/1.0\n```\n\nIf we wanted to dedicate two nodes to the speculative model instead and only 3 to the target model, we would move the slash so that there are three values\non the left half and two values on the right. The values themselves must also be adjusted. PipeInfer will automatically allocate any leftover layers to the\nhead node (node 0), but it is best to allocate them explicitly.\n\nAn example command to launch PipeInfer across a cluster of 4 nodes is shown below:\n\n```bash\nmpirun -hostfile /mnt/cluster/hostsfile --bind-to none \\\n    /var/llama.cpp/bin/speculative \\\n    -md /mnt/cluster/models/tinyllama-1.1b-1t-openorca.Q4_K_M.gguf \\\n    -m /mnt/cluster/models/dolphin-2.1-70b.Q3_K_M.gguf  \\\n    -e \\\n    -f /mnt/cluster/llama.cpp/prompts/dolphin.txt \\\n    -n 128 \\\n    --mpi-layer-split 0.1,0.4,0.5/1.0 \\\n    --ignore-eos \\\n    --temp -1.0 \\\n    --repeat-last-n 0 \\\n    --draft 4 \\\n    -tb 12,32,40,32 \\\n    -t 6,32,20,32 \\\n    -c 1024 \\\n    -pa 0.001 \\\n    -ps 0.8 \\\n    --numa \\\n    -pr 0.4 \\\n    -pd 0.01 \\\n    -np 3\n```\n\nThe confidence cutoff recovery and decay factors are the `-pr` and `-pd` parameters, respectively. The `-pa` parameter is the base acceptance confidence cutoff, and the `-ps` parameter\nis the split confidence threshold. The `-np` parameter determines the maximum number of sequences within a sequence partition. The `--draft` parameter determines the maximum number of tokens\nin a speculative microbatch. The `-t` and `-tb` parameters set the number of threads to use for single-token and batched inference for each node. Finally, we pass the `--bind-to none` parameter\nto `mpirun` to enable each rank to use more than one thread. All other parameters are unmodified from upstream `llama.cpp`; use the `--help` parameter to list all available\nparameters and what they do.\n\n## Tuning PipeInfer\nPipeInfer offers many tunable parameters that can drastically affect the performance. The most important of these are the `-pa`, `-ps`, `-np`, and `--draft` parameters. These four parameters determine the\naverage size of a speculative microbatch. An important fact to consider is that the acceptance confidence cutoff (`-pa`) corresponds to how confident the speculative model is in its own\noutput, *not* how confident it is that it matches the target's output.\n\nSome models, such as Orca 2, have high confidence in their own outputs but low alignment with their target. In such a case,\nit is best to set the acceptance confidence cutoff high, such as `0.5`, and the draft length (`--draft`) low, such as `3`.\n\nConversely, some models exhibit high alignment with their target. In these cases, increasing the draft length slightly can produce\nbetter performance, but it is almost always best to keep the draft length below `8` to maintain low latency and high granularity with regard to cancellation.\nInstead, the acceptance confidence cutoff can be lowered significantly, down to `0.1` in our testing with Goliath and XWin 7B. Such a low cutoff\nenables PipeInfer to generate as many speculative runs as possible. Keeping the draft length low as well (`3` for Goliath and XWin 7B) enables the system to\ngenerate many small microbatches that can be evaluated with low latency while maintaining a high enough acceptance rate that the pipeline is not thrashed\nwith flushing operations.\n\nThe split confidence threshold and the maximum number of sequences per partition can be tuned to produce a wide and shallow speculative tree; however, we have observed\nthat narrow and deep trees produce the best overall performance. Keeping the number of sequences per partition low, such as `3` or `4`, allows early inference cancellation\nfiner control over which sequences to cancel. In the future, it may be possible to extend cancellation to work on individual sequences, but at the moment, we have observed\nwide trees to decrease performance universally.\n\nThe two adaptive parameters, `-pr` and `-pd`, can be set to allow the system to automatically adjust the speculative tree size and speculation frequency according to the current conditions.\nThe recovery factor (`-pr`) increases the acceptance confidence cutoff every time a speculative tree is generated, reset upon acceptance of a token in the verification phase.\nThis has the effect of making it harder to generate speculative trees the longer it's been since a token has been validated. The recovery factor is most useful when the target pipeline is very\ndeep, and the speculative model diverges fairly quickly. Without this factor, such a system would generate speculative runs even when it is likely the speculative tree has long since diverged from the target model,\nwasting CPU cycles and requiring a cancellation later on. We have not observed this factor to increase performance; its only usefulness is in decreasing wasted cycles and transfers.\n\nThe decay factor is the inverse, decreasing the acceptance confidence cutoff the longer it's been since any speculative tree was generated, reset upon successful token acceptance\nin the verification phase. This factor enables the system to opportunistically generate speculations when it would otherwise be idle, even if they aren't likely to be accepted.\nIt is best used when the base acceptance cutoff is fairly high, and one observes significant idle time. We have found this factor must be very carefully managed, setting it too high results in\nlarge speculative trees being pushed through the pipeline as soon as there is any idle time, causing an imbalance in evaluation time for each run, therefore causing further idle periods. This can\nbe mitigated by keeping the draft length low.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomicperfectionist%2Fpipeinfer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautonomicperfectionist%2Fpipeinfer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomicperfectionist%2Fpipeinfer/lists"}