{"id":14380628,"url":"https://github.com/Lightning-AI/LitServe","last_synced_at":"2025-08-23T11:30:54.956Z","repository":{"id":222843486,"uuid":"730728120","full_name":"Lightning-AI/LitServe","owner":"Lightning-AI","description":"The easiest way to deploy agents, MCP servers, models, RAG, pipelines and more. No MLOps. No YAML.","archived":false,"fork":false,"pushed_at":"2025-08-20T22:56:17.000Z","size":1418,"stargazers_count":3511,"open_issues_count":21,"forks_count":234,"subscribers_count":30,"default_branch":"main","last_synced_at":"2025-08-21T00:28:35.061Z","etag":null,"topics":["ai","api","artificial-intelligence","deep-learning","developer-tools","fastapi","rest-api","serving","web"],"latest_commit_sha":null,"homepage":"https://lightning.ai/litserve","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Lightning-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-12-12T14:45:03.000Z","updated_at":"2025-08-20T17:19:36.000Z","dependencies_parsed_at":"2024-04-02T08:47:26.752Z","dependency_job_id":"2497a263-8dd5-4e2e-8ae6-d670d7fd0cdd","html_url":"https://github.com/Lightning-AI/LitServe","commit_stats":null,"previous_names":["lightning-ai/litserve"],"tags_count":39,"template":false,"template_full_name":null,"purl":"pkg:github/Lightning-AI/LitServe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lightning-AI%2FLitServe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lightning-AI%2FLitServe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lightning-AI%2FLitServe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lightning-AI%2FLitServe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Lightning-AI","download_url":"https://codeload.github.com/Lightning-AI/LitServe/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lightning-AI%2FLitServe/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271746830,"owners_count":24813588,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","api","artificial-intelligence","deep-learning","developer-tools","fastapi","rest-api","serving","web"],"created_at":"2024-08-28T14:01:20.059Z","updated_at":"2025-08-23T11:30:54.930Z","avatar_url":"https://github.com/Lightning-AI.png","language":"Python","readme":"\u003cdiv align='center'\u003e\n\n\u003ch2\u003e\n  The easiest way to deploy agents, MCP servers, RAG, pipelines, any model. \n  \u003cbr/\u003e\n  No MLOps. No YAML.\n\u003c/h2\u003e    \n\n\u003cimg alt=\"Lightning\" src=\"https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/ls_banner2.png\" width=\"800px\" style=\"max-width: 100%;\"\u003e\n\n\u0026nbsp; \n\u003c/div\u003e\n\nMost serving engines serve one model with rigid abstractions. LitServe lets you serve any model (vision, audio, text) and build full AI systems - agents, chatbots, MCP servers, RAG, pipelines - with full control, batching, multi-GPU, streaming, custom logic, multi-model support, and zero YAML. \n\nSelf host or deploy in one-click to [Lightning AI](https://lightning.ai/).\n\n\u0026nbsp;\n\n\u003cdiv align='center'\u003e\n  \n\u003cpre\u003e\n✅ Build full AI systems   ✅ 2× faster than FastAPI     ✅ Agents, RAG, pipelines, more\n✅ Custom logic + control  ✅ Any PyTorch model          ✅ Self-host or managed        \n✅ Multi-GPU autoscaling   ✅ Batching + streaming       ✅ BYO model or vLLM           \n✅ No MLOps glue code      ✅ Easy setup in Python       ✅ Serverless support          \n\n\u003c/pre\u003e\n\n\u003cdiv align='center'\u003e\n\n[![PyPI Downloads](https://static.pepy.tech/badge/litserve)](https://pepy.tech/projects/litserve)\n[![Discord](https://img.shields.io/discord/1077906959069626439?label=Get%20help%20on%20Discord)](https://discord.gg/WajDThKAur)\n![cpu-tests](https://github.com/Lightning-AI/litserve/actions/workflows/ci-testing.yml/badge.svg)\n[![codecov](https://codecov.io/gh/Lightning-AI/litserve/graph/badge.svg?token=SmzX8mnKlA)](https://codecov.io/gh/Lightning-AI/litserve)\n[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/Lightning-AI/litserve/blob/main/LICENSE)\n\n\u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003cdiv style=\"text-align: center;\"\u003e\n    \u003ca target=\"_blank\" href=\"#quick-start\" style=\"margin: 0 10px;\"\u003eQuick start\u003c/a\u003e •\n    \u003ca target=\"_blank\" href=\"#featured-examples\" style=\"margin: 0 10px;\"\u003eExamples\u003c/a\u003e •\n    \u003ca target=\"_blank\" href=\"#features\" style=\"margin: 0 10px;\"\u003eFeatures\u003c/a\u003e •\n    \u003ca target=\"_blank\" href=\"#performance\" style=\"margin: 0 10px;\"\u003ePerformance\u003c/a\u003e •\n    \u003ca target=\"_blank\" href=\"#host-anywhere\" style=\"margin: 0 10px;\"\u003eHosting\u003c/a\u003e •\n    \u003ca target=\"_blank\" href=\"https://lightning.ai/docs/litserve\" style=\"margin: 0 10px;\"\u003eDocs\u003c/a\u003e\n  \u003c/div\u003e\n\u003c/div\u003e\n\n\u0026nbsp;\n\n\u003cdiv align=\"center\"\u003e\n\u003ca target=\"_blank\" href=\"https://lightning.ai/docs/litserve/home/get-started\"\u003e\n  \u003cimg src=\"https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/get-started-badge.svg\" height=\"36px\" alt=\"Get started\"/\u003e\n\u003c/a\u003e\n\u003c/div\u003e\n\n\u0026nbsp; \n\n# Quick start\n\nInstall LitServe via pip ([more options](https://lightning.ai/docs/litserve/home/install)):\n\n```bash\npip install litserve\n```\n\n[Example 1](#inference-pipeline-example): Toy inference pipeline with multiple models.   \n[Example 2](#agent-example): Minimal agent to fetch the news (with OpenAI API).    \n([Advanced examples](#featured-examples)):    \n\n### Inference pipeline example   \n\n```python\nimport litserve as ls\n\n# define the api to include any number of models, dbs, etc...\nclass InferencePipeline(ls.LitAPI):\n    def setup(self, device):\n        self.model1 = lambda x: x**2\n        self.model2 = lambda x: x**3\n\n    def predict(self, request):\n        x = request[\"input\"]    \n        # perform calculations using both models\n        a = self.model1(x)\n        b = self.model2(x)\n        c = a + b\n        return {\"output\": c}\n\nif __name__ == \"__main__\":\n    # 12+ features like batching, streaming, etc...\n    server = ls.LitServer(InferencePipeline(max_batch_size=1), accelerator=\"auto\")\n    server.run(port=8000)\n```\n\nDeploy for free to [Lightning cloud](#hosting-options) (or self host anywhere):\n\n```bash\n# Deploy for free with autoscaling, monitoring, etc...\nlightning deploy server.py --cloud\n\n# Or run locally (self host anywhere)\nlightning deploy server.py\n# python server.py\n```\n\nTest the server: Simulate an http request (run this on any terminal):\n```bash\ncurl -X POST http://127.0.0.1:8000/predict -H \"Content-Type: application/json\" -d '{\"input\": 4.0}'\n```\n\n### Agent example\n\n```python\nimport re, requests, openai\nimport litserve as ls\n\nclass NewsAgent(ls.LitAPI):\n    def setup(self, device):\n        self.openai_client = openai.OpenAI(api_key=\"OPENAI_API_KEY\")\n\n    def predict(self, request):\n        website_url = request.get(\"website_url\", \"https://text.npr.org/\")\n        website_text = re.sub(r'\u003c[^\u003e]+\u003e', ' ', requests.get(website_url).text)\n\n        # ask the LLM to tell you about the news\n        llm_response = self.openai_client.chat.completions.create(\n           model=\"gpt-3.5-turbo\", \n           messages=[{\"role\": \"user\", \"content\": f\"Based on this, what is the latest: {website_text}\"}],\n        )\n        output = llm_response.choices[0].message.content.strip()\n        return {\"output\": output}\n\nif __name__ == \"__main__\":\n    server = ls.LitServer(NewsAgent())\n    server.run(port=8000)\n```\nTest it:\n```bash\ncurl -X POST http://127.0.0.1:8000/predict -H \"Content-Type: application/json\" -d '{\"website_url\": \"https://text.npr.org/\"}'\n```\n\n\u0026nbsp;\n\n# Key benefits   \n\nA few key benefits:\n\n- **Deploy any pipeline or model**: Agents, pipelines, RAG, chatbots, image models, video, speech, text, etc...\n- **No MLOps glue:** LitAPI lets you build full AI systems (multi-model, agent, RAG) in one place ([more](https://lightning.ai/docs/litserve/api-reference/litapi)).   \n- **Instant setup:** Connect models, DBs, and data in a few lines with `setup()` ([more](https://lightning.ai/docs/litserve/api-reference/litapi#setup)).    \n- **Optimized:** autoscaling, GPU support, and fast inference included ([more](https://lightning.ai/docs/litserve/api-reference/litserver)).    \n- **Deploy anywhere:** self-host or one-click deploy with Lightning ([more](https://lightning.ai/docs/litserve/features/deploy-on-cloud)).\n- **FastAPI for AI:** Built on FastAPI but optimized for AI - 2× faster with AI-specific multi-worker handling ([more]((#performance))).   \n- **Expert-friendly:** Use vLLM, or build your own with full control over batching, caching, and logic ([more](https://lightning.ai/lightning-ai/studios/deploy-a-private-llama-3-2-rag-api)).    \n\n\u003e ⚠️ Not a vLLM or Ollama alternative out of the box. LitServe gives you lower-level flexibility to build what they do (and more) if you need it.\n\n\u0026nbsp;\n\n# Featured examples    \nHere are examples of inference pipelines for common model types and use cases.      \n  \n\u003cpre\u003e\n\u003cstrong\u003eToy model:\u003c/strong\u003e      \u003ca target=\"_blank\" href=\"#define-a-server\"\u003eHello world\u003c/a\u003e\n\u003cstrong\u003eLLMs:\u003c/strong\u003e           \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-llama-3-2-vision-with-litserve\"\u003eLlama 3.2\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/openai-fault-tolerant-proxy-server\"\u003eLLM Proxy server\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-ai-agent-with-tool-use\"\u003eAgent with tool use\u003c/a\u003e\n\u003cstrong\u003eRAG:\u003c/strong\u003e            \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-a-private-llama-3-2-rag-api\"\u003evLLM RAG (Llama 3.2)\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-a-private-llama-3-1-rag-api\"\u003eRAG API (LlamaIndex)\u003c/a\u003e\n\u003cstrong\u003eNLP:\u003c/strong\u003e            \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-any-hugging-face-model-instantly\"\u003eHugging face\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-a-hugging-face-bert-model\"\u003eBERT\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-text-embedding-api-with-litserve\"\u003eText embedding API\u003c/a\u003e\n\u003cstrong\u003eMultimodal:\u003c/strong\u003e     \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-open-ai-clip-with-litserve\"\u003eOpenAI Clip\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-a-multi-modal-llm-with-minicpm\"\u003eMiniCPM\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-phi3-5-vision-api-with-litserve\"\u003ePhi-3.5 Vision Instruct\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/bhimrajyadav/studios/deploy-and-chat-with-qwen2-vl-using-litserve\"\u003eQwen2-VL\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-a-multi-modal-llm-with-pixtral\"\u003ePixtral\u003c/a\u003e\n\u003cstrong\u003eAudio:\u003c/strong\u003e          \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-open-ai-s-whisper-model\"\u003eWhisper\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-an-music-generation-api-with-meta-s-audio-craft\"\u003eAudioCraft\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-an-audio-generation-api\"\u003eStableAudio\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-a-noise-cancellation-api-with-deepfilternet\"\u003eNoise cancellation (DeepFilterNet)\u003c/a\u003e\n\u003cstrong\u003eVision:\u003c/strong\u003e         \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-a-private-api-for-stable-diffusion-2\"\u003eStable diffusion 2\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-an-image-generation-api-with-auraflow\"\u003eAuraFlow\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-an-image-generation-api-with-flux\"\u003eFlux\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-a-super-resolution-image-api-with-aura-sr\"\u003eImage Super Resolution (Aura SR)\u003c/a\u003e,\n                \u003ca target=\"_blank\" href=\"https://lightning.ai/bhimrajyadav/studios/deploy-background-removal-api-with-litserve\"\u003eBackground Removal\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-a-controlled-image-generation-api-controlnet\"\u003eControl Stable Diffusion (ControlNet)\u003c/a\u003e\n\u003cstrong\u003eSpeech:\u003c/strong\u003e         \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-a-voice-clone-api-coqui-xtts-v2-model\"\u003eText-speech (XTTS V2)\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/bhimrajyadav/studios/deploy-a-speech-generation-api-using-parler-tts-powered-by-litserve\"\u003eParler-TTS\u003c/a\u003e\n\u003cstrong\u003eClassical ML:\u003c/strong\u003e   \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-random-forest-with-litserve\"\u003eRandom forest\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-xgboost-with-litserve\"\u003eXGBoost\u003c/a\u003e\n\u003cstrong\u003eMiscellaneous:\u003c/strong\u003e  \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-an-media-conversion-api-with-ffmpeg\"\u003eMedia conversion API (ffmpeg)\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/deploy-both-pytorch-and-tensorflow-in-a-single-api\"\u003ePyTorch + TensorFlow in one API\u003c/a\u003e, \u003ca target=\"_blank\" href=\"https://lightning.ai/lightning-ai/studios/openai-fault-tolerant-proxy-server\"\u003eLLM proxy server\u003c/a\u003e\n\u003c/pre\u003e\n\u003c/pre\u003e\n\n[Browse 100+ community-built templates](https://lightning.ai/studios?section=serving)\n\n\u0026nbsp;\n\n# Host anywhere\n\nSelf-host with full control, or deploy with [Lightning AI](https://lightning.ai/) in seconds with autoscaling, security, and 99.995% uptime.  \n**Free tier included. No setup required. Run on your cloud**   \n\n```bash\nlightning deploy server.py --cloud\n```\n\nhttps://github.com/user-attachments/assets/ff83dab9-0c9f-4453-8dcb-fb9526726344\n\n\u0026nbsp;\n\n# Features\n\n\u003cdiv align='center'\u003e\n\n| [Feature](https://lightning.ai/docs/litserve/features)               | Self Managed                      | [Fully Managed on Lightning](https://lightning.ai/deploy)         |\n|----------------------------------------------------------------------|-----------------------------------|------------------------------------|\n| Docker-first deployment          | ✅ DIY                             | ✅ One-click deploy                |\n| Cost                             | ✅ Free (DIY)                      | ✅ Generous [free tier](https://lightning.ai/pricing) with pay as you go                |\n| Full control                     | ✅                                 | ✅                                 |\n| Use any engine (vLLM, etc.)      | ✅                                 | ✅ vLLM, Ollama, LitServe, etc.    |\n| Own VPC                          | ✅ (manual setup)                  | ✅ Connect your own VPC            |\n| [(2x)+ faster than plain FastAPI](#performance)                                               | ✅       | ✅                                 |\n| [Bring your own model](https://lightning.ai/docs/litserve/features/full-control)              | ✅       | ✅                                 |\n| [Build compound systems (1+ models)](https://lightning.ai/docs/litserve/home)                 | ✅       | ✅                                 |\n| [GPU autoscaling](https://lightning.ai/docs/litserve/features/gpu-inference)                  | ✅       | ✅                                 |\n| [Batching](https://lightning.ai/docs/litserve/features/batching)                              | ✅       | ✅                                 |\n| [Streaming](https://lightning.ai/docs/litserve/features/streaming)                            | ✅       | ✅                                 |\n| [Worker autoscaling](https://lightning.ai/docs/litserve/features/autoscaling)                 | ✅       | ✅                                 |\n| [Serve all models: (LLMs, vision, etc.)](https://lightning.ai/docs/litserve/examples)         | ✅       | ✅                                 |\n| [Supports PyTorch, JAX, TF, etc...](https://lightning.ai/docs/litserve/features/full-control) | ✅       | ✅                                 |\n| [OpenAPI compliant](https://www.openapis.org/)                                                | ✅       | ✅                                 |\n| [Open AI compatibility](https://lightning.ai/docs/litserve/features/open-ai-spec)             | ✅       | ✅                                 |\n| [MCP server support](https://lightning.ai/docs/litserve/features/mcp)                         | ✅       | ✅                                 |\n| [Asynchronous](https://lightning.ai/docs/litserve/features/async-concurrency)                 | ✅       | ✅                                 |\n| [Authentication](https://lightning.ai/docs/litserve/features/authentication)                  | ❌ DIY   | ✅ Token, password, custom         |\n| GPUs                             | ❌ DIY                             | ✅ 8+ GPU types, H100s from $1.75  |\n| Load balancing                   | ❌                                 | ✅ Built-in                        |\n| Scale to zero (serverless)       | ❌                                 | ✅ No machine runs when idle       |\n| Autoscale up on demand           | ❌                                 | ✅ Auto scale up/down              |\n| Multi-node inference             | ❌                                 | ✅ Distribute across nodes         |\n| Use AWS/GCP credits              | ❌                                 | ✅ Use existing cloud commits      |\n| Versioning                       | ❌                                 | ✅ Make and roll back releases     |\n| Enterprise-grade uptime (99.95%) | ❌                                 | ✅ SLA-backed                      |\n| SOC2 / HIPAA compliance          | ❌                                 | ✅ Certified \u0026 secure              |\n| Observability                    | ❌                                 | ✅ Built-in, connect 3rd party tools|\n| CI/CD ready                      | ❌                                 | ✅ Lightning SDK                   |\n| 24/7 enterprise support          | ❌                                 | ✅ Dedicated support               |\n| Cost controls \u0026 audit logs       | ❌                                 | ✅ Budgets, breakdowns, logs       |\n| Debug on GPUs                    | ❌                                 | ✅ Studio integration              |\n| [20+ features](https://lightning.ai/docs/litserve/features)                    | -                                 | -                                  |\n\n\u003c/div\u003e\n\n\u0026nbsp;\n\n# Performance  \nLitServe is designed for AI workloads. Specialized multi-worker handling delivers a minimum **2x speedup over FastAPI**.    \n\nAdditional features like batching and GPU autoscaling can drive performance well beyond 2x, scaling efficiently to handle more simultaneous requests than FastAPI and TorchServe.\n    \nReproduce the full benchmarks [here](https://lightning.ai/docs/litserve/home/benchmarks) (higher is better).  \n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg alt=\"LitServe\" src=\"https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/ls_charts_v6.png\" width=\"1000px\" style=\"max-width: 100%;\"\u003e\n\u003c/div\u003e \n\nThese results are for image and text classification ML tasks. The performance relationships hold for other ML tasks (embedding, LLM serving, audio, segmentation, object detection, summarization etc...).   \n    \n***💡 Note on LLM serving:*** For high-performance LLM serving (like Ollama/vLLM), integrate [vLLM with LitServe](https://lightning.ai/lightning-ai/studios/deploy-a-private-llama-3-2-rag-api), use [LitGPT](https://github.com/Lightning-AI/litgpt?tab=readme-ov-file#deploy-an-llm), or build your custom vLLM-like server with LitServe. Optimizations like kv-caching, which can be done with LitServe, are needed to maximize LLM performance.\n\n\u0026nbsp;\n\n\n# Community\nLitServe is a [community project accepting contributions](https://lightning.ai/docs/litserve/community) - Let's make the world's most advanced AI inference engine.\n\n💬 [Get help on Discord](https://discord.com/invite/XncpTy7DSt)    \n📋 [License: Apache 2.0](https://github.com/Lightning-AI/litserve/blob/main/LICENSE)    \n","funding_links":[],"categories":["Model Deployment","Python","General","Repos","artificial-intelligence","其他_机器学习与深度学习","2. **Production Tools**","Deployment and Serving"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLightning-AI%2FLitServe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLightning-AI%2FLitServe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLightning-AI%2FLitServe/lists"}