{"id":51401445,"url":"https://github.com/dphnai/aphrodite-engine","last_synced_at":"2026-07-04T07:00:54.296Z","repository":{"id":176405046,"uuid":"657414607","full_name":"dphnAI/aphrodite-engine","owner":"dphnAI","description":"Large-scale LLM inference engine","archived":false,"fork":false,"pushed_at":"2026-07-02T20:16:35.000Z","size":236355,"stargazers_count":1778,"open_issues_count":127,"forks_count":200,"subscribers_count":27,"default_branch":"main","last_synced_at":"2026-07-02T20:27:11.458Z","etag":null,"topics":["api-rest","cuda","inference-engine","inferentia","intel","lora","machine-learning","rocm","speculative-decoding","tpu"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dphnAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"AlpinDale"}},"created_at":"2023-06-23T02:34:29.000Z","updated_at":"2026-07-01T21:15:00.000Z","dependencies_parsed_at":"2023-10-15T00:45:53.339Z","dependency_job_id":"b4ae3ce3-d1e6-425e-a6f3-a503679f57d9","html_url":"https://github.com/dphnAI/aphrodite-engine","commit_stats":{"total_commits":1016,"total_committers":37,"mean_commits":27.45945945945946,"dds":"0.10629921259842523","last_synced_commit":"62120722456521967f2b5563a90e8119d996d6fa"},"previous_names":["pygmalionai/aphrodite-engine","aphrodite-engine/aphrodite-engine","dphnai/aphrodite-engine"],"tags_count":110,"template":false,"template_full_name":null,"purl":"pkg:github/dphnAI/aphrodite-engine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dphnAI%2Faphrodite-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dphnAI%2Faphrodite-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dphnAI%2Faphrodite-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dphnAI%2Faphrodite-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dphnAI","download_url":"https://codeload.github.com/dphnAI/aphrodite-engine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dphnAI%2Faphrodite-engine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35112708,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-04T02:00:05.987Z","response_time":113,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-rest","cuda","inference-engine","inferentia","intel","lora","machine-learning","rocm","speculative-decoding","tpu"],"created_at":"2026-07-04T07:00:45.441Z","updated_at":"2026-07-04T07:00:54.286Z","avatar_url":"https://github.com/dphnAI.png","language":"C++","funding_links":["https://github.com/sponsors/AlpinDale"],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\nBreathing Life into Language\n\u003c/h1\u003e\n\n![aphrodite](https://raw.githubusercontent.com/PygmalionAI/aphrodite-engine/main/assets/aphrodite.png)\n\nAphrodite is an inference engine that optimizes the serving of HuggingFace-compatible models at scale. Built on vLLM's Paged Attention technology, it delivers high-performance model inference for multiple concurrent users. Aphrodite serves as the backend engine powering [PygmalionAI](https://pygmalion.chat)'s chat platforms and API infrastructure.\n\nAphrodite builds upon and integrates the exceptional work from [various projects](#acknowledgements), primarily [vLLM](https://vllm.ai).\n\n## Features\n\n- Continuous Batching\n- Efficient K/V management with [PagedAttention](https://vllm.ai) from vLLM\n- Optimized CUDA kernels for improved inference\n- Quantization support via [AQLM](https://arxiv.org/abs/2401.06118), [AutoRound](https://arxiv.org/abs/2309.05516), [AWQ](https://arxiv.org/abs/2306.00978), [BitNet](https://arxiv.org/abs/2310.11453), [Bitsandbytes](https://arxiv.org/abs/2208.07339), [ExLlamaV3](https://github.com/turboderp-org/exllamav3), [GGUF](https://github.com/ggml-org/llama.cpp), [GPTQ](https://arxiv.org/abs/2210.17323), [QuIP#](https://arxiv.org/abs/2402.04396), [SqueezeLLM](https://arxiv.org/abs/2306.07629), [Marlin](https://arxiv.org/abs/2408.11743), [[2]](https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html) [[3]](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/), [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer), [TorchAO](https://github.com/pytorch/ao), [VPTQ](https://arxiv.org/abs/2409.17066), [compressed_tensors](https://github.com/vllm-project/llm-compressor), [MXFP4](https://huggingface.co/blog/RakshitAralimatti/learn-ai-with-me), and more.\n- Distributed inference\n- Quantized KV cache using scaled and scale-less FP8, and TurboQuant\n- Support for modern samplers such as DRY, XTC, Mirostat, and more\n- Disaggregated inference\n- Speculative decoding, including EAGLE, DFlash, ngram, MTP, and more\n- Multimodal support\n- Multi-LoRA support\n\n## Quickstart\n\nInstall the engine:\n\n```sh\npip install -U aphrodite-engine\n```\n\nThen launch a model:\n\n```sh\naphrodite run Qwen/Qwen3.5-0.8B\n```\n\nThis will create a [OpenAI](https://platform.openai.com/docs/api-reference/)-compatible API server that can be accessed at port 2242 of the localhost. You can plug in the API into a UI that supports OpenAI, such as [SillyTavern](https://github.com/SillyTavern/SillyTavern).\n\n## Requirements\n\n- Operating System: Linux, Windows (WSL2)\n- Python: 3.10 to 3.13 (build from source for 3.14)\n\n#### Build Requirements\n\n- CUDA \u003e= 12\n\n### Notes\n\n1. By design, Aphrodite takes up 92% of your GPU's VRAM. If you're not serving an LLM at scale, you may want to limit the amount of memory it takes up. You can do this in the API example by launching the server with the `--gpu-memory-utilization 0.6` (0.6 means 60%).\n\n2. You can view the full list of commands by running `aphrodite run --help`.\n\n## Acknowledgements\n\nAphrodite Engine would have not been possible without the phenomenal work of other open-source projects. A (non-exhaustive) list:\n\n- [vLLM](https://github.com/vllm-project/vllm)\n- [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)\n- [xFormers](https://github.com/facebookresearch/xformers)\n- [Flash Attention](https://github.com/Dao-AILab/flash-attention)\n- [llama.cpp](https://github.com/ggerganov/llama.cpp)\n- [AutoAWQ](https://github.com/casper-hansen/AutoAWQ)\n- [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ)\n- [SqueezeLLM](https://github.com/SqueezeAILab/SqueezeLLM/)\n- [Exllamav2](https://github.com/turboderp/exllamav2)\n- [TabbyAPI](https://github.com/theroyallab/tabbyAPI)\n- [AQLM](https://github.com/Vahe1994/AQLM)\n- [KoboldAI](https://github.com/henk717/KoboldAI)\n- [Text Generation WebUI](https://github.com/oobabooga/text-generation-webui)\n- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)\n- [Ray](https://github.com/ray-project/ray)\n\n### Sponsors\n\nPast and present, in alphabetical order:\n\n- [Arc Compute](https://www.arccompute.io/)\n- [Lium](https://lium.io)\n- [Prime Intellect](https://www.primeintellect.ai/)\n- [PygmalionAI](https://pygmalion.chat)\n- [Ruliad AI](https://ruliad.ai)\n\n## Contributing\n\nEveryone is welcome to contribute. You can support the project by opening Pull Requests for new features, fixes, or general UX improvements.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdphnai%2Faphrodite-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdphnai%2Faphrodite-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdphnai%2Faphrodite-engine/lists"}