{"id":30556203,"url":"https://github.com/jd-opensource/xllm-service","last_synced_at":"2025-08-28T06:35:10.265Z","repository":{"id":310852490,"uuid":"1036706300","full_name":"jd-opensource/xllm-service","owner":"jd-opensource","description":null,"archived":false,"fork":false,"pushed_at":"2025-08-19T13:53:17.000Z","size":92,"stargazers_count":12,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-20T17:29:16.251Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jd-opensource.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-12T13:18:41.000Z","updated_at":"2025-08-20T16:31:16.000Z","dependencies_parsed_at":"2025-08-20T17:32:07.097Z","dependency_job_id":"fc677ea4-b326-41fb-96ba-2dda7a9fb468","html_url":"https://github.com/jd-opensource/xllm-service","commit_stats":null,"previous_names":["jd-opensource/xllm-service"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jd-opensource/xllm-service","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jd-opensource%2Fxllm-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jd-opensource%2Fxllm-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jd-opensource%2Fxllm-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jd-opensource%2Fxllm-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jd-opensource","download_url":"https://codeload.github.com/jd-opensource/xllm-service/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jd-opensource%2Fxllm-service/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272453851,"owners_count":24937469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-28T06:35:06.445Z","updated_at":"2025-08-28T06:35:10.256Z","avatar_url":"https://github.com/jd-opensource.png","language":"C++","readme":"\u003c!-- Copyright 2022 JD Co.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this project except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License. --\u003e\n\n[English](./README.md) | [中文](./README_zh.md)\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/assets/xllm_service_title.png\" alt=\"xLLM\" style=\"width:50%; height:auto;\"\u003e\n\u003c/p\u003e\n\n\n## 1. Project Overview\n**xLLM-service** is a service-layer framework developed based on the **xLLM** inference engine, providing efficient, fault-tolerant, and flexible LLM inference services for clustered deployment.\n\nxLLM-service targets to address key challenges in enterprise-level service scenarios:\n\n- How to ensure the SLA of online services and improve resource utilization of offline tasks in a hybrid online-offline deployment environment.\n\n- How to react to changing request loads in actual businesses, such as fluctuations in input/output lengths.\n\n- Resolving performance bottlenecks of multimodal model requests.\n\n- Ensuring high reliability of computing instances.\n\n---\n\n## 2. Key Features\nWith management of computing resource pools, intelligent scheduling and preemption of hybrid requests, and real-time monitoring of computing instances, xLLM-service achieves the following key features:\n\n- Unified scheduling of online and offline requests, with preemptive execution for online requests and best-effort execution for offline requests.\n\n- Adaptive dynamic allocation of PD ratios, supporting efficient switching of instance PD roles.\n\n- EPD three-stage disaggregation for multimodal requests, with intelligent resource allocation for different stages.\n\n- Fault-tolerant architecture, fast detection of instance error and automatic rescheduling for interrupted requests. \n\n---\n\n## 3. Core Architecture\n\n```\n├── xllm-service/\n|   : main source folder\n│   ├── chat_template/               # \n│   ├── common/                      # \n│   ├── examples/                    # \n│   ├── http_service/                # \n│   ├── rpc_service/                 # \n|   ├── tokenizers/                  #\n|   └── master.cpp                   # \n```\n\n---\n\n\n## 4. Quick Start\n#### Installation\n```\ngit clone git@coding.jd.com:xllm-ai/xllm_service.git\ncd xllm_service\ngit submodule init\ngit submodule update\n```\n#### Compilation\ncompile vcpkg, set env variable:\n```\nexport VCPKG_ROOT=/export/home/xxx/vcpkg-src\n```\ncompile xllm-service: \n```\nmkdir -p build \u0026\u0026 cd build\ncmake .. \u0026\u0026 make -j 8\n```\n\n\n--- \n\n## 5. Contributing\n\nThere are several ways you can contribute to xLLM:\n\n1. Reporting Issues (Bugs \u0026 Errors)\n2. Suggesting Enhancements\n3. Improving Documentation\n    + Fork the repository\n    + Add your view in document\n    + Send your pull request\n4. Writing Code\n    + Fork the repository\n    + Create a new branch\n    + Add your feature or improvement\n    + Send your pull request\n\nWe appreciate all kinds of contributions! 🎉🎉🎉\nIf you have problems about development, please check our document: * **[Document](./docs/docs/readme.md)**\n\n---\n\n## 6. Community \u0026 Support\n\nIf you encounter any issues along the way, you are welcomed to submit reproducible steps and log snippets in the project's Issues area, or contact the xLLM Core team directly via your internal Slack.\n\nWelcome to contact us:\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"\" alt=\"contact\" width=\"50%\" height=\"50%\"\u003e\n\u003c/div\u003e\n\n---\n## 7. About the Contributors\n\nThanks to all the following [developers](https://github.com/jd-opensource/xllm-service/graphs/contributors) who have contributed to xLLM.\n\u003ca href=\"https://github.com/jd-opensource/xllm-service/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=jd-opensource/xllm-service\" /\u003e\n\u003c/a\u003e\n\n---\n\n## 8. License\n\n[Apache License](LICENSE)\n\n#### xLLM is provided by JD.com \n#### Thanks for your Contributions!\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjd-opensource%2Fxllm-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjd-opensource%2Fxllm-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjd-opensource%2Fxllm-service/lists"}