{"id":30184395,"url":"https://github.com/gpustack/runner","last_synced_at":"2026-02-04T02:17:32.177Z","repository":{"id":309314131,"uuid":"1035810244","full_name":"gpustack/runner","owner":"gpustack","description":null,"archived":false,"fork":false,"pushed_at":"2025-08-11T07:51:32.000Z","size":21,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-11T08:21:49.706Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gpustack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-11T06:02:34.000Z","updated_at":"2025-08-11T07:51:35.000Z","dependencies_parsed_at":"2025-08-11T08:35:01.998Z","dependency_job_id":null,"html_url":"https://github.com/gpustack/runner","commit_stats":null,"previous_names":["gpustack/runner"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/gpustack/runner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Frunner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Frunner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Frunner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Frunner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gpustack","download_url":"https://codeload.github.com/gpustack/runner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Frunner/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270057794,"owners_count":24519527,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-12T12:07:39.327Z","updated_at":"2026-02-04T02:17:32.144Z","avatar_url":"https://github.com/gpustack.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GPUStack Runner\n\nThis repository serves as the Docker image pack center for GPUStack Runner.\nIt provides a collection of Dockerfiles to build images for various inference services across different accelerated\nbackends.\n\n## Agenda\n\n- [Onboard Services](#onboard-services)\n- [Directory Structure](#directory-structure)\n- [Dockerfile Convention](#dockerfile-convention)\n- [Docker Image Naming Convention](#docker-image-naming-convention)\n- [Integration Process](#integration-process)\n\n## Onboard Services\n\n\u003e [!TIP]\n\u003e - The list below shows the accelerated backends and inference services available in the latest release. For support of\n    backends or services not shown here, please refer to previous release tags.\n\u003e - Deprecated inference service versions in the latest release are marked with ~~strikethrough~~ formatting. They may\n    still be available in previous releases, and not recommended for new deployments.\n\u003e - Polished inference service versions in the latest release are marked with **bold** formatting. If they are using in\n    your deployment, it is recommended to pull the latest images and upgrade.\n\nThe following table lists the supported accelerated backends and their corresponding inference services with versions.\n\n### Ascend CANN\n\n\u003e [!WARNING]\n\u003e - The Atlas 300I series is currently experimental in vLLM, only supporting eager mode and float16 data type. And there\n    are some known issues for running vLLM, you can refer to\n    vllm-ascend [#3316](https://github.com/vllm-project/vllm-ascend/issues/3316)\n    and [#2795](https://github.com/vllm-project/vllm-ascend/issues/2795).\n\n| CANN Version \u003cbr/\u003e (Variant) | MindIE    | vLLM                              | SGLang                 |\n|------------------------------|-----------|-----------------------------------|------------------------|\n| 8.5 (A3/910C)                | `2.3.0`   | `0.14.1`, `0.13.0`                | `0.5.8`                |\n| 8.5 (910B)                   | `2.3.0`   | `0.14.1`, `0.13.0`                | `0.5.8`                |\n| 8.5 (310P)                   | `2.3.0`   | `0.14.1`                          |                        |\n| 8.3 (A3/910C)                | `2.2.rc1` | `0.12.0`, `0.11.0`                | `0.5.7`, `0.5.6.post2` |\n| 8.3 (910B)                   | `2.2.rc1` | `0.12.0`, `0.11.0`                | `0.5.7`, `0.5.6.post2` |\n| 8.3 (310P)                   | `2.2.rc1` |                                   |                        |\n| 8.2 (A3/910C)                | `2.1.rc2` | `0.10.2`                          | `0.5.2`, `0.5.1.post3` |\n| 8.2 (910B)                   | `2.1.rc2` | `0.10.2`, `0.10.0`,  \u003cbr/\u003e`0.9.2` | `0.5.2`, `0.5.1.post3` |\n| 8.2 (310P)                   | `2.1.rc2` | `0.10.0`, `0.9.2`                 |                        |\n\n### Iluvatar CoreX\n\n| CoreX Version \u003cbr/\u003e (Variant) | vLLM    |\n|-------------------------------|---------|\n| 4.2                           | `0.8.3` |\n\n### NVIDIA CUDA\n\n\u003e [!NOTE]\n\u003e - CUDA 12.9 supports Compute Capabilities:\n    `7.5 8.0+PTX 8.9 9.0 10.0 10.3 12.0 12.1+PTX`.\n\u003e - CUDA 12.8 supports Compute Capabilities:\n    `7.5 8.0+PTX 8.9 9.0 10.0+PTX 12.0+PTX`.\n\u003e - CUDA 12.6/12.4 supports Compute Capabilities:\n    `7.5 8.0+PTX 8.9 9.0+PTX`.\n\n| CUDA Version \u003cbr/\u003e (Variant) | vLLM                                                                 | SGLang                                              | VoxBox   |\n|------------------------------|----------------------------------------------------------------------|-----------------------------------------------------|----------|\n| 12.9                         | `0.15.0`, `0.14.1`, \u003cbr/\u003e`0.13.0`, `0.12.0`, \u003cbr/\u003e`0.11.2`           | `0.5.8`, `0.5.7`, \u003cbr/\u003e`0.5.6.post2`                |          |\n| 12.8                         | `0.15.0`, `0.14.1`, \u003cbr/\u003e`0.13.0`, `0.12.0`, \u003cbr/\u003e`0.11.2`, `0.10.2` | `0.5.8`, `0.5.7`, \u003cbr/\u003e`0.5.6.post2`, `0.5.5.post3` | `0.0.21` |\n| 12.6                         | `0.15.0`, `0.14.1`, \u003cbr/\u003e`0.13.0`, `0.12.0`, \u003cbr/\u003e`0.11.2`, `0.10.2` |                                                     | `0.0.21` |\n\n### Hygon DTK\n\n| DTK Version \u003cbr/\u003e (Variant) | vLLM                       |\n|-----------------------------|----------------------------|\n| 25.04                       | `0.11.0`, `0.9.2`, `0.8.5` |\n\n### T-Head HGGC\n\n| HGGC Version \u003cbr/\u003e (Variant) | vLLM     | SGLang  |\n|------------------------------|----------|---------|\n| 12.3                         | `0.11.1` | `0.5.5` |\n\n### MetaX MACA\n\n| MACA Version \u003cbr/\u003e (Variant) | vLLM     | SGLang  |\n|------------------------------|----------|---------|\n| 3.3                          | `0.11.2` | `0.5.6` |\n| 3.2                          | `0.10.2` |         |\n| 3.0                          | `0.9.1`  |         |\n\n### MThreads MUSA\n\n| MUSA Version \u003cbr/\u003e (Variant) | vLLM    | SGLang  |\n|------------------------------|---------|---------|\n| 4.3.2                        |         | `0.5.2` |\n| 4.1.0                        | `0.9.2` |         |\n\n### AMD ROCm\n\n\u003e [!NOTE]\n\u003e - ROCm 7.0 supports LLVM targets:\n    `gfx908 gfx90a gfx942 gfx950 gfx1030 gfx1100 gfx1101 gfx1200 gfx1201 gfx1150 gfx1151`.\n\u003e - ROCm 6.4 supports LLVM targets:\n    `gfx908 gfx90a gfx942 gfx1030 gfx1100`.\n\n\u003e [!WARNING]\n\u003e - ROCm 7.0 vLLM `0.11.2` are reusing the official ROCm 6.4 PyTorch 2.9 wheel package rather than a ROCm\n    7.0 specific PyTorch build. Although supports ROCm 7.0 in vLLM `0.11.2`, `gfx1150/gfx1151` are not supported yet.\n\u003e - ROCm 6.4 vLLM `0.13.0` supports `gfx903 gfx90a gfx942` only.\n\u003e - ROCm 6.4 SGLang supports `gfx942` only.\n\u003e - ROCm 7.0 SGLang supports `gfx950` only.\n\n| ROCm Version \u003cbr/\u003e (Variant) | vLLM                                                                 | SGLang                                              |\n|------------------------------|----------------------------------------------------------------------|-----------------------------------------------------|\n| 7.0                          | `0.15.0`, `0.14.1`, \u003cbr/\u003e`0.13.0`, `0.12.0`, \u003cbr/\u003e`0.11.2`           | `0.5.8`, `0.5.7`, \u003cbr/\u003e`0.5.6.post2`                |\n| 6.4                          | `0.15.0`, `0.14.1`, \u003cbr/\u003e`0.13.0`, `0.12.0`, \u003cbr/\u003e`0.11.2`, `0.10.2` | `0.5.8`, `0.5.7`, \u003cbr/\u003e`0.5.6.post2`, `0.5.5.post3` |\n\n## Directory Structure\n\nThe pack skeleton is organized by backend:\n\n```text\npack\n├── {BACKEND 1}\n│   └── Dockerfile\n├── {BACKEND 2}\n│   └── Dockerfile\n├── {BACKEND 3}\n│   └── Dockerfile\n├── ...\n│   └── Dockerfile\n└── {BACKEND N}\n    └── Dockerfile\n\n```\n\n## Dockerfile Convention\n\nEach Dockerfile follows these conventions:\n\n- Begin with comments describing the package logic in steps and usage of build arguments (`ARG`s).\n- Use `ARG` for all required and optional build arguments. If a required argument is unused, mark it as `(PLACEHOLDER)`.\n- Use heredoc syntax for `RUN` commands to improve readability.\n\n### Example Dockerfile Structure\n\n```dockerfile\n\n# Describe package logic and ARG usage.\n#\nARG PYTHON_VERSION=...                                 # REQUIRED\nARG CMAKE_MAX_JOBS=...                                 # REQUIRED\nARG {OTHERS}                                           # OPTIONAL\nARG {BACKEND}_VERSION=...                              # REQUIRED\nARG {BACKEND}_VERSION_EXTRA=...                        # OPTIONAL\nARG {BACKEND}_ARCHS=...                                # REQUIRED\nARG {BACKEND}_{OTHERS}=...                             # OPTIONAL\nARG {SERVICE}_BASE_IMAGE=...                           # REQUIRED\nARG {SERVICE}_VERSION=...                              # REQUIRED\nARG {SERVICE}_{OTHERS}=...                             # OPTIONAL\nARG {SERVICE}_{FRAMEWORK}_VERSION=...                  # REQUIRED\nARG {SERVICE}_{FRAMEWORK}_{OTHERS}=...                 # OPTIONAL\n\n# Stage Bake Runtime\nFROM {BACKEND DEVEL IMAGE} AS runtime\nSHELL [\"/bin/bash\", \"-eo\", \"pipefail\", \"-c\"]\nARG TARGETPLATFORM\nARG TARGETOS\nARG TARGETARCH\nARG ...\nRUN \u003c\u003cEOF\n    # TODO: install runtime dependencies\nEOF\n\n# Stage Install Service\nFROM {BACKEND}_BASE_IMAGE AS {service}\nSHELL [\"/bin/bash\", \"-eo\", \"pipefail\", \"-c\"]\nARG TARGETPLATFORM\nARG TARGETOS\nARG TARGETARCH\nARG ...\nRUN \u003c\u003cEOF\n    # TODO: install service and dependencies\nEOF\n\nWORKDIR /\nENTRYPOINT [ \"tini\", \"--\" ]\n\n```\n\n## Docker Image Naming Convention\n\nThe Docker image naming convention is as follows:\n\n- Multi-architecture image names: `{NAMESPACE}/{REPOSITORY}:{TAG}`.\n- Single-architecture image tags:\n  `{BACKEND}{BACKEND_VERSION%.*}[-{BACKEND_VARIANT}]-{SERVICE}{SERVICE_VERSION}-{OS}-{ARCH}`.\n- Multi-architecture image tags: `{BACKEND}{BACKEND_VERSION%.*}[-{BACKEND_VARIANT}]-{SERVICE}{SERVICE_VERSION}[-dev]`.\n- All names adn tags must be lowercase.\n\n### Example\n\n- NAMESPACE: `gpustack`\n- REPOSITORY: `runner`\n\n| Accelerated Backend | OS/ARCH     | Inference Service | Single-Arch Image Name                                | Multi-Arch Image Name                     |\n|---------------------|-------------|-------------------|-------------------------------------------------------|-------------------------------------------|\n| Ascend CANN 910b    | linux/amd64 | vLLM              | `gpustack/runner:cann8.1-910b-vllm0.9.2-linux-amd64`  | `gpustack/runner:cann8.1-910b-vllm0.9.2`  |\n| Ascend CANN 910b    | linux/arm64 | vLLM              | `gpustack/runner:cann8.1-910b-vllm0.9.2-linux-arm64`  | `gpustack/runner:cann8.1-910b-vllm0.9.2`  |\n| NVIDIA CUDA 12.8    | linux/amd64 | vLLM              | `gpustack/runner:cuda12.8-910b-vllm0.9.2-linux-amd64` | `gpustack/runner:cuda12.8-910b-vllm0.9.2` |\n| NVIDIA CUDA 12.8    | linux/arm64 | vLLM              | `gpustack/runner:cuda12.8-910b-vllm0.9.2-linux-arm64` | `gpustack/runner:cuda12.8-910b-vllm0.9.2` |\n\n### Build and Release Workflow\n\n1. Build single architecture images for OS/ARCH, e.g. `gpustack/runner:cann8.1-910b-vllm0.9.2-linux-amd64`.\n2. Combine single-architecture images into a multiple architectures image, e.g.\n   `gpustack/runner:cann8.1-910b-vllm0.9.2-dev`.\n3. After testing, rename the multi-architecture image to the final tag, e.g. `gpustack/runner:cann8.1-910b-vllm0.9.2`.\n\n## Integration Process\n\n### Ingesting a New Accelerated Backend\n\nTo add support for a new accelerated backend:\n\n1. Create a new directory under `pack/` named with the new backend.\n2. Add a `Dockerfile` in the new directory following the [Dockerfile Convention](#dockerfile-convention).\n3. Update [pack.yml](.github/workflows/pack.yml), [discard.yml](.github/workflows/discard.yml) and [prune.yml](.github/workflows/prune.yml) to include the new backend in the build matrix.\n4. Update [matrix.yml](pack/matrix.yaml) to include the new backend and its variants.\n5. Update `_RE_DOCKER_IMAGE` in [runner.py](gpustack_runner/runner.py) to recognize the new backend.\n6. [Optional] Update [tests](tests/gpustack_runner) if necessary.\n\n### Ingesting a New Inference Service\n\nTo add support for a new inference service:\n\n1. Modify the `Dockerfile` of the relevant backend in `pack/{BACKEND}/Dockerfile` to include the new service.\n2. Update [pack.yml](.github/workflows/pack.yml) to include the new service in the build matrix.\n3. Update [matrix.yml](pack/matrix.yaml) to include the new service.\n4. Update `_RE_DOCKER_IMAGE` in [runner.py](gpustack_runner/runner.py) to recognize the new service.\n5. [Optional] Update [tests](tests/gpustack_runner) if necessary.\n\n## License\n\nCopyright (c) 2025 The GPUStack authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at [LICENSE](./LICENSE) file for details.\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpustack%2Frunner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgpustack%2Frunner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpustack%2Frunner/lists"}