{"id":13409248,"url":"https://github.com/vllm-project/vllm","last_synced_at":"2026-10-20T20:25:33.109Z","repository":{"id":176349946,"uuid":"599547518","full_name":"vllm-project/vllm","owner":"vllm-project","description":"A high-throughput and memory-efficient inference and serving engine for LLMs","archived":false,"fork":false,"pushed_at":"2026-10-15T12:34:15.000Z","size":319602,"stargazers_count":93849,"open_issues_count":8899,"forks_count":23416,"subscribers_count":602,"default_branch":"main","last_synced_at":"2026-10-15T12:39:03.162Z","etag":null,"topics":["amd","blackwell","cuda","deepseek","deepseek-v3","gpt","gpt-oss","inference","kimi","llama","llm","llm-serving","model-serving","moe","openai","pytorch","qwen","qwen3","tpu","transformer"],"latest_commit_sha":null,"homepage":"https://vllm.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vllm-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":"docs/governance/collaboration.md","roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","claude":null,"gemini":null,"cursor":null,"copilot":null,"dco":"DCO","cla":null,"disclosure":null},"funding":{"github":["vllm-project"],"open_collective":"vllm"}},"created_at":"2023-02-09T11:23:20.000Z","updated_at":"2026-10-15T12:34:23.000Z","dependencies_parsed_at":"2026-10-08T06:43:02.710Z","dependency_job_id":null,"html_url":"https://github.com/vllm-project/vllm","commit_stats":{"total_commits":4106,"total_committers":701,"mean_commits":5.85734664764622,"dds":0.8816366293229421,"last_synced_commit":"7a3a83e3b87f50fe9c0985a5c5bcc1d4cf2e95cd"},"previous_names":["vllm-project/vllm"],"tags_count":206,"template":false,"template_full_name":null,"purl":"pkg:github/vllm-project/vllm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vllm-project%2Fvllm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vllm-project%2Fvllm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vllm-project%2Fvllm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vllm-project%2Fvllm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vllm-project","download_url":"https://codeload.github.com/vllm-project/vllm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vllm-project%2Fvllm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":344836268,"owners_count":38329930,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-10-03T21:59:58.778Z","status":"online","status_checked_at":"2026-10-15T02:00:06.843Z","response_time":105,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amd","blackwell","cuda","deepseek","deepseek-v3","gpt","gpt-oss","inference","kimi","llama","llm","llm-serving","model-serving","moe","openai","pytorch","qwen","qwen3","tpu","transformer"],"created_at":"2024-07-30T20:00:59.195Z","updated_at":"2026-10-17T03:55:22.239Z","avatar_url":"https://github.com/vllm-project.png","language":"Python","funding_links":["https://github.com/sponsors/vllm-project","https://opencollective.com/vllm"],"categories":["Models and Tools","Python","Other","🚀 Deployment \u0026 Serving","🔬 AI 研究ツール","Agent Infrastructure","Uncategorized","🎯 Tool Categories","Tools for deploying LLM","LLM部署与本地运行","INFERENCING FRAMEWORKS","LLM","Software","Serving","Inference \u0026 Serving","Models and Projects","Large Model Serving","\u003cimg src=\"./assets/cpu.svg\" width=\"16\" height=\"16\" style=\"vertical-align: middle;\"\u003e Backends","LLM Deployment","💻 Local Inference Tools","Deployment Stacks","Tools","3. Inference Engines \u0026 Serving","NLP","Inference \u0026 Deployment","🏗️ Reference Implementations \u0026 Case Studies","Related Repositories","HarmonyOS","Deployment and Serving","Tooling \u0026 Infrastructure","推理 Inference","openai compatible inference engines","⚡ LLM Inference \u0026 Local Serving Engines","Deep Learning","Summary","Frameworks","⚡ LLM Inference \u0026 Hosting","Awesome Open-Sourced LLMSys Projects","Hardware Acceleration and Deployment Strategies","Collections","🚀 MLOps","Local Inference","Generative KI","pytorch","🔓 Open Source Inference Engines","🏠 Local and Self-Hosted AI","Inference Engines","LLM Inference","Repos","公司列表","Model Serving Frameworks","🚀 Model Serving \u0026 Deployment","Inference Engine","Top 100 Open-Source LLM \u0026 Agent Projects","Inference engines","Model Serving","🛠️ AI 工具与框架","LLM Serving / Inference","Compute \u0026 infrastructure","🔢 Papers - Parametric Memory","Caching","🧠 Large Language Models (LLMs)","Production Tools","Inference Engines \u0026 Backends (22)","Language Models","Local LLM Runners","AI Serving and Inference Operations","⚙️ Frameworks \u0026 Inference Engines","10. Agent Deployment","Agentic AI ![](https://img.shields.io/badge/_-AGENTIC-22d3ee?style=flat-square\u0026logo=openai\u0026logoColor=white)","8. Inference Engines","Language Models for NLP","LLM Integration","Motores de Ejecución (Runtime)","🖥 Local Deployment Tools","MLOps \u0026 Deployment","⚙️ ინფერენსი და LLM ხელსაწყოები","vLLM \u0026 High-Throughput Inference","Machine Learning and AI","LLM Frameworks \u0026 Libraries","Local and Self-Hosted AI","📑 Table of Contents","LLMs Backend","GPU and AI Compute","AI \u0026 LLM","Local Inference and Serving","Supporting Infrastructure","Research \u0026 Data Analysis","💻 Open-Source Libraries","Open-Source Alternatives Cheat Sheet","⚙️ LLM Inference \u0026 Serving (15)","Inference","Open-Source Models and Local Deployment","*Ops for AI","Optimize","Model Inference","Serving \u0026 Inference","Model Serving \u0026 Inference","Inference and Local LLM Tools","Inference Runtimes \u0026 Backends","Model serving / gateway","🤖 AI, ML \u0026 Deep Learning","AI","⚙️ Systems and Multi-GPU Engineering","Tools for Deployment"],"sub_categories":["LLM Deployment","Scientific Computing","3. The Enterprise / High-Scale Stack (The 1%)","自動運転","Uncategorized","🤖 LLMOps \u0026 GenAI (2024-2025)","LLM 评估与数据","vLLM","Large Model Serving","Software Engineering \u0026 OS Operations","Ray + LLM","Production Inference Servers","Other","High-Performance Inference","T18 · Inference \u0026 Serving","Inference and hardware","Windows Manager","Deployment \u0026 Optimization","Embedding Models","Distributed Systems","Popular On-Device LLMs Framework","Useful Repositories","Tools","Local LLM Runners","Platform Guides","🔥 2025 新增热门领域 (AI, Cloud, Science)","LangManus","UI/Interface Tutorials","LLM 推理与部署","Local inference","🎥 Multimodal Memory (for Generation)","Inference infrastructure KV cache","🛠️ Self-Hosted Solutions","Inference \u0026 Serving","LLM / VLM runtimes","Self-Hosted","Local LLM runtimes","Server / Production","Efficient and Small Language Models","Model Providers","Inference Engines","Motores de Inferencia","Server Deployment \u0026 High-Performance Inference","Model Serving","LLM and Inference","Notable Valuations","🛠️ Hands-on Projects \u0026 Tools","Ray vs. Dask vs. Spark vs. Flink","LLM Apps \u0026 Interfaces","Serve at scale","Model Intelligence","[Langflow](https://github.com/langflow-ai/langflow)","Inference Engine","Response:","Inference","Model Serving \u0026 Inference","Serving Inference","Model Serving Frameworks","LLM inference","Lernphilosophie"],"readme":"\u003c!-- markdownlint-disable MD001 MD041 --\u003e\n\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/vllm-project/vllm/main/docs/assets/logos/vllm-logo-text-dark.png\"\u003e\n    \u003cimg alt=\"vLLM\" src=\"https://raw.githubusercontent.com/vllm-project/vllm/main/docs/assets/logos/vllm-logo-text-light.png\" width=55%\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\n\u003ch3 align=\"center\"\u003e\nEasy, fast, and cheap LLM serving for everyone\n\u003c/h3\u003e\n\n\u003cp align=\"center\"\u003e\n| \u003ca href=\"https://docs.vllm.ai\"\u003e\u003cb\u003eDocumentation\u003c/b\u003e\u003c/a\u003e | \u003ca href=\"https://blog.vllm.ai/\"\u003e\u003cb\u003eBlog\u003c/b\u003e\u003c/a\u003e | \u003ca href=\"https://arxiv.org/abs/2309.06180\"\u003e\u003cb\u003ePaper\u003c/b\u003e\u003c/a\u003e | \u003ca href=\"https://x.com/vllm_project\"\u003e\u003cb\u003eTwitter/X\u003c/b\u003e\u003c/a\u003e | \u003ca href=\"https://discuss.vllm.ai\"\u003e\u003cb\u003eUser Forum\u003c/b\u003e\u003c/a\u003e | \u003ca href=\"https://slack.vllm.ai\"\u003e\u003cb\u003eDeveloper Slack\u003c/b\u003e\u003c/a\u003e |\n\u003c/p\u003e\n\n🔥 We have built a vLLM website to help you get started with vLLM. Please visit [vllm.ai](https://vllm.ai) to learn more.\nFor events, please visit [vllm.ai/events](https://vllm.ai/events) to join us.\n\n---\n\n## About\n\nvLLM is a fast and easy-to-use library for LLM inference and serving.\n\nOriginally developed in the [Sky Computing Lab](https://sky.cs.berkeley.edu) at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.\n\nvLLM is fast with:\n\n- State-of-the-art serving throughput\n- Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)\n- Continuous batching of incoming requests, chunked prefill, prefix caching\n- Fast and flexible model execution with piecewise and full CUDA/HIP graphs\n- Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and [more](https://docs.vllm.ai/en/latest/features/quantization/index.html)\n- Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton\n- Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL\n- Speculative decoding including n-gram, suffix, EAGLE, DFlash\n- Automatic kernel generation and graph-level transformations using torch.compile\n- Disaggregated prefill, decode, and encode\n\nvLLM is flexible and easy to use with:\n\n- Seamless integration with popular Hugging Face models\n- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more\n- Tensor, pipeline, data, expert, and context parallelism for distributed inference\n- Streaming outputs\n- Generation of structured outputs using xgrammar or guidance\n- Tool calling and reasoning parsers\n- OpenAI-compatible API server, plus Anthropic Messages API and gRPC support\n- Efficient multi-LoRA support for dense and MoE layers\n- Support for NVIDIA GPUs, AMD GPUs, Intel GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more.\n\nvLLM seamlessly supports 200+ model architectures on Hugging Face, including:\n\n- Decoder-only LLMs (e.g., Llama, Qwen, Gemma)\n- Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS)\n- Hybrid attention and state-space models (e.g., Mamba, Qwen3.5)\n- Multi-modal models (e.g., LLaVA, Qwen-VL, Pixtral)\n- Embedding and retrieval models (e.g., E5-Mistral, GTE, ColBERT)\n- Reward and classification models (e.g., Qwen-Math)\n\nFind the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).\n\n## Getting Started\n\nInstall vLLM with [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`:\n\n```bash\nuv pip install vllm\nvllm download-kernels  # precompiled kernels for faster startup\n```\n\nOr [build from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source) for development.\n\nVisit our [documentation](https://docs.vllm.ai/en/latest/) to learn more.\n\n- [Installation](https://docs.vllm.ai/en/latest/getting_started/installation.html)\n- [Quickstart](https://docs.vllm.ai/en/latest/getting_started/quickstart.html)\n- [List of Supported Models](https://docs.vllm.ai/en/latest/models/supported_models.html)\n\n## Contributing\n\nWe welcome and value any contributions and collaborations.\nPlease check out [Contributing to vLLM](https://docs.vllm.ai/en/latest/contributing/index.html) for how to get involved.\n\n## Citation\n\nIf you use vLLM for your research, please cite our [paper](https://arxiv.org/abs/2309.06180):\n\n```bibtex\n@inproceedings{kwon2023efficient,\n  title={Efficient Memory Management for Large Language Model Serving with PagedAttention},\n  author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},\n  booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},\n  year={2023}\n}\n```\n\n## Contact Us\n\n\u003c!-- --8\u003c-- [start:contact-us] --\u003e\n- For technical questions and feature requests, please use GitHub [Issues](https://github.com/vllm-project/vllm/issues)\n- For discussing with fellow users, please use the [vLLM Forum](https://discuss.vllm.ai)\n- For coordinating contributions and development, please use [Slack](https://slack.vllm.ai)\n- For security disclosures, please use GitHub's [Security Advisories](https://github.com/vllm-project/vllm/security/advisories) feature\n- For collaborations and partnerships, please contact us at [collaboration@vllm.ai](mailto:collaboration@vllm.ai)\n\u003c!-- --8\u003c-- [end:contact-us] --\u003e\n\n## Media Kit\n\n- If you wish to use vLLM's logo, please refer to [our media kit repo](https://github.com/vllm-project/media-kit)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvllm-project%2Fvllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvllm-project%2Fvllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvllm-project%2Fvllm/lists"}