{"id":23749281,"url":"https://github.com/turboderp-org/exllamav2","last_synced_at":"2025-05-13T19:16:03.219Z","repository":{"id":194217973,"uuid":"684980325","full_name":"turboderp-org/exllamav2","owner":"turboderp-org","description":"A fast inference library for running LLMs locally on modern consumer-class GPUs","archived":false,"fork":false,"pushed_at":"2025-05-07T17:59:13.000Z","size":22204,"stargazers_count":4161,"open_issues_count":140,"forks_count":312,"subscribers_count":34,"default_branch":"master","last_synced_at":"2025-05-07T18:24:46.725Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/turboderp-org.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"ko_fi":"turboderp"}},"created_at":"2023-08-30T08:54:22.000Z","updated_at":"2025-05-07T17:59:16.000Z","dependencies_parsed_at":"2023-09-25T20:23:32.994Z","dependency_job_id":"43af0eb9-5755-44ef-a2d7-61f3d45de1c9","html_url":"https://github.com/turboderp-org/exllamav2","commit_stats":null,"previous_names":["turboderp/exllamav2","turboderp-org/exllamav2"],"tags_count":40,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/turboderp-org%2Fexllamav2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/turboderp-org%2Fexllamav2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/turboderp-org%2Fexllamav2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/turboderp-org%2Fexllamav2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/turboderp-org","download_url":"https://codeload.github.com/turboderp-org/exllamav2/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254010822,"owners_count":21999002,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-31T15:08:53.875Z","updated_at":"2025-05-13T19:15:58.165Z","avatar_url":"https://github.com/turboderp-org.png","language":"Python","funding_links":["https://ko-fi.com/turboderp"],"categories":["Python","📦 Compression","Browse The Shelves","Quantization, Distillation, and Compression"],"sub_categories":["Quantization","Local LLM developer tools","Quantization libraries"],"readme":"# ExLlamaV2\n\nExLlamaV2 is an inference library for running local LLMs on modern consumer GPUs.\n\nThe official and recommended backend server for ExLlamaV2 is [TabbyAPI](https://github.com/theroyallab/tabbyAPI/),\nwhich provides an OpenAI-compatible API for local or remote inference, with extended features like HF model\ndownloading, embedding model support and support for HF Jinja2 chat templates.\n\nSee the [wiki](https://github.com/theroyallab/tabbyAPI/wiki/1.-Getting-Started) for help getting started.\n\n\n## New in v0.1.0+:\n\n- ExLlamaV2 now supports paged attention via [Flash Attention](https://github.com/Dao-AILab/flash-attention) 2.5.7+\n- New generator with dynamic batching, smart prompt caching, K/V cache deduplication and simplified API\n\n![alt_text](doc/dynamic_gen.gif)\n\n## Dynamic generator\n\nThe dynamic generator supports all inference, sampling and speculative decoding features of the previous two \ngenerators, consolidated into one API (with the exception of FP8 cache, though the Q4 cache mode is supported and\nperforms better anyway, see [here](doc/qcache_eval.md).)\n\nThe generator is explained in detail [here](doc/dynamic.md).\n\n- Single generation:\n  ```python\n  output = generator.generate(prompt = \"Hello, my name is\", max_new_tokens = 200)\n  ```\n- Batched generation:\n    ```python\n    outputs = generator.generate(\n        prompt = [\n            \"Hello, my name is\",\n            \"Once upon a time,\",\n            \"Large language models are\",\n        ], \n        max_new_tokens = 200\n    )\n    ```\n- Streamed generation with `asyncio`:\n    ```python\n    job = ExLlamaV2DynamicJobAsync(\n        generator,\n        input_ids = tokenizer.encode(\"You can lead a horse to water\"),\n        banned_strings = [\"make it drink\"],\n        gen_settings = ExLlamaV2Sampler.Settings.greedy(),\n        max_new_tokens = 200\n    )  \n    async for result in job:\n        text = result.get(\"text\", \"\")\n        print(text, end = \"\")       \n    ``` \nSee the full, updated examples [here](https://github.com/turboderp/exllamav2/tree/master/examples).\n\n\n## Performance\n\nSome quick tests to compare performance with ExLlama V1. There may be more performance optimizations in the future,\nand speeds will vary across GPUs, with slow CPUs still being a potential bottleneck:\n\n| Model      | Mode         | Size  | grpsz | act | 3090Ti  | 4090        |\n|------------|--------------|-------|-------|-----|---------|-------------|\n| Llama      | GPTQ         | 7B    | 128   | no  | 181 t/s | **205** t/s |\n| Llama      | GPTQ         | 13B   | 128   | no  | 110 t/s | **114** t/s |\n| Llama      | GPTQ         | 33B   | 128   | yes | 44 t/s  | **48** t/s  |\n| OpenLlama  | GPTQ         | 3B    | 128   | yes | 259 t/s | **296** t/s |\n| CodeLlama  | EXL2 4.0 bpw | 34B   | -     | -   | 44 t/s  | **50** t/s  |\n| Llama2     | EXL2 3.0 bpw | 7B    | -     | -   | 217 t/s | **257** t/s |\n| Llama2     | EXL2 4.0 bpw | 7B    | -     | -   | 185 t/s | **211** t/s |\n| Llama2     | EXL2 5.0 bpw | 7B    | -     | -   | 164 t/s | **179** t/s |\n| Llama2     | EXL2 2.5 bpw | 70B   | -     | -   | 33 t/s  | **38** t/s  |\n| TinyLlama  | EXL2 3.0 bpw | 1.1B  | -     | -   | 656 t/s | **770** t/s |\n| TinyLlama  | EXL2 4.0 bpw | 1.1B  | -     | -   | 602 t/s | **700** t/s |\n\n\n## How to\n\nTo install from the repo you'll need the CUDA Toolkit and either gcc on Linux or (Build Tools for) Visual Studio\non Windows). Also make sure you have an appropriate version of [PyTorch](https://pytorch.org/get-started/locally/), then run:\n\n```sh\ngit clone https://github.com/turboderp/exllamav2\ncd exllamav2\npip install -r requirements.txt\npip install .\n\npython test_inference.py -m \u003cpath_to_model\u003e -p \"Once upon a time,\"\n# Append the '--gpu_split auto' flag for multi-GPU inference\n```\n\nA simple console chatbot is included. Run it with:\n\n```sh\npython examples/chat.py -m \u003cpath_to_model\u003e -mode llama -gs auto\n```\n\n\nThe `-mode` argument chooses the prompt format to use. `raw` will produce a simple chatlog-style chat that works with base \nmodels and various other finetunes. Run with `-modes` for a list of all available prompt formats. You can also provide\na custom system prompt with `-sp`. \n\n\n## Integration and APIs\n\n- [TabbyAPI](https://github.com/theroyallab/tabbyAPI/) is a FastAPI-based server that provides an OpenAI-style web API\ncompatible with [SillyTavern](https://sillytavernai.com/) and other frontends.  \n\n- [ExUI](https://github.com/turboderp/exui) is a simple, standalone single-user web UI that serves an ExLlamaV2 instance\ndirectly with chat and notebook modes.\n\n- [text-generation-webui](https://github.com/oobabooga/text-generation-webui) supports ExLlamaV2 through the **exllamav2**\nand **exllamav2_HF** loaders.\n\n- [lollms-webui](https://github.com/ParisNeo/lollms-webui) supports ExLlamaV2 through the exllamav2 binding.\n\n## Installation\n\n### Method 1: Install from source\n\nTo install the current dev version, clone the repo and run the setup script:\n\n```sh\ngit clone https://github.com/turboderp/exllamav2\ncd exllamav2\npip install -r requirements.txt\npip install .\n```\n\nBy default this will also compile and install the Torch C++ extension (`exllamav2_ext`) that the library relies on. \nYou can skip this step by setting the `EXLLAMA_NOCOMPILE` environment variable:\n\n```sh\nEXLLAMA_NOCOMPILE= pip install .\n```\n\nThis will install the \"JIT version\" of the package, i.e. it will install the Python components without building the\nC++ extension in the process. Instead, the extension will be built the first time the library is used, then cached in \n`~/.cache/torch_extensions` for subsequent use.\n\n### Method 2: Install from release (with prebuilt extension)\n\nReleases are available [here](https://github.com/turboderp/exllamav2/releases), with prebuilt wheels that contain the extension binaries. Make sure to grab\nthe right version, matching your platform, Python version (`cp`) and CUDA version. Crucially, you must also match\nthe prebuilt wheel with your PyTorch version, since the Torch C++ extension ABI breaks with every new version of \nPyTorch.\n\nEither download an appropriate wheel or install directly from the appropriate URL:\n\n```sh\npip install https://github.com/turboderp/exllamav2/releases/download/v0.0.12/exllamav2-0.0.12+cu121-cp311-cp311-linux_x86_64.whl\n```\n\nThe `py3-none-any.whl` version is the JIT version which will build the extension on first launch. The `.tar.gz` file\ncan also be installed this way, and it will build the extension while installing.\n\n### Method 3: Install from PyPI\n\nA PyPI package is available as well. This is the same as the JIT version (see above). It can be installed with:\n\n```sh\npip install exllamav2\n```\n\n\n## EXL2 quantization\n\nExLlamaV2 supports the same 4-bit GPTQ models as V1, but also a new \"EXL2\" format. EXL2 is based on the same\noptimization method as GPTQ and supports 2, 3, 4, 5, 6 and 8-bit quantization. The format allows for mixing quantization\nlevels within a model to achieve any average bitrate between 2 and 8 bits per weight.\n\nMoreover, it's possible to apply multiple quantization levels to each linear layer, producing something akin to sparse \nquantization wherein more important weights (columns) are quantized with more bits. The same remapping trick that lets\nExLlama work efficiently with act-order models allows this mixing of formats to happen with little to no impact on\nperformance.\n\nParameter selection is done automatically by quantizing each matrix multiple times, measuring the quantization \nerror (with respect to the chosen calibration data) for each of a number of possible settings, per layer. Finally, a\ncombination is chosen that minimizes the maximum quantization error over the entire model while meeting a target\naverage bitrate.\n\nIn my tests, this scheme allows Llama2 70B to run on a single 24 GB GPU with a 2048-token context, producing coherent \nand mostly stable output with 2.55 bits per weight. 13B models run at 2.65 bits within 8 GB of VRAM, although currently\nnone of them uses GQA which effectively limits the context size to 2048. In either case it's unlikely that the model\nwill fit alongside a desktop environment. For now.\n\n[![chat_screenshot](doc/llama2_70b_chat_thumb.png)](doc/llama2_70b_chat.png)\n[![chat_screenshot](doc/codellama_13b_instruct_thumb.png)](doc/codellama_13b_instruct.png)\n\n### Conversion\n\nA script is provided to quantize models. Converting large models can be somewhat slow, so be warned. The conversion\nscript and its options are explained in [detail here](doc/convert.md)\n\n### Evaluation\n\nA number of evaluaion scripts are provided. See [here](doc/eval.md) for details.\n\n### Community\n\nA test community is provided at https://discord.gg/NSFwVuCjRq \nQuanting service free of charge is provided at #bot test. The computation is generiously provided by the Bloke powered by Lambda labs. \n\n### HuggingFace repos\n\n- I've uploaded a few EXL2-quantized models to Hugging Face to play around with, [here](https://huggingface.co/turboderp).\n\n- [LoneStriker](https://huggingface.co/LoneStriker) provides a large number of EXL2 models on Hugging Face. \n\n- [bartowski](https://huggingface.co/bartowski) has some more EXL2 models on HF.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fturboderp-org%2Fexllamav2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fturboderp-org%2Fexllamav2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fturboderp-org%2Fexllamav2/lists"}