{"id":51533006,"url":"https://github.com/enzymead/reactantserver.jl","last_synced_at":"2026-07-09T05:01:28.280Z","repository":{"id":365290906,"uuid":"1271429585","full_name":"EnzymeAD/ReactantServer.jl","owner":"EnzymeAD","description":"Production inference server for Reactant-compiled models, serving KServe V2 over gRPC","archived":false,"fork":false,"pushed_at":"2026-07-03T19:39:39.000Z","size":1817,"stargazers_count":4,"open_issues_count":17,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-03T21:25:25.010Z","etag":null,"topics":["grpc","inference-server","julia","kserve","mlir","reactant","stablehlo","xla"],"latest_commit_sha":null,"homepage":"https://enzymead.github.io/ReactantServer.jl","language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EnzymeAD.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-16T16:49:02.000Z","updated_at":"2026-06-28T02:34:39.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/EnzymeAD/ReactantServer.jl","commit_stats":null,"previous_names":["csvance/reactantserver.jl"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/EnzymeAD/ReactantServer.jl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EnzymeAD%2FReactantServer.jl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EnzymeAD%2FReactantServer.jl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EnzymeAD%2FReactantServer.jl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EnzymeAD%2FReactantServer.jl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EnzymeAD","download_url":"https://codeload.github.com/EnzymeAD/ReactantServer.jl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EnzymeAD%2FReactantServer.jl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35287405,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grpc","inference-server","julia","kserve","mlir","reactant","stablehlo","xla"],"created_at":"2026-07-09T05:01:26.601Z","updated_at":"2026-07-09T05:01:28.273Z","avatar_url":"https://github.com/EnzymeAD.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ReactantServer.jl\n\n[![Documentation](https://img.shields.io/badge/docs-dev-blue.svg)](https://enzymead.github.io/ReactantServer.jl/dev/)\n[![CI](https://img.shields.io/github/checks-status/enzymead/ReactantServer.jl/main?label=CI)](https://github.com/enzymead/ReactantServer.jl/commits/main)\n\nA production inference server for XLA-accelerated models, compiled through Reactant.jl\n(StableHLO via XLA today). It speaks the KServe V2 inference API natively over gRPC, so standard\nTriton and KServe clients connect unchanged; it scales from a single GPU to many from one\ncontainer; and it squeezes the most models out of each GPU by balancing model **memory** against\n**compute**. It is Julia-first throughout — custom pre/postprocessing is plain Julia, and every\nconvention follows Julia's (column-major, batch-last axes).\n\nIt targets static-graph workloads — computer vision, scientific computing — where many models\nshare a GPU and one model executes at a time.\n\nNot sure which setup fits you? [Common Use Cases](https://enzymead.github.io/ReactantServer.jl/dev/manual/common_use_cases/) walks\nthrough the deployment shapes (single GPU, multi-GPU distributed or replicated, multi-node) with\nan example configuration for each.\n\n## Highlights\n\n- **XLA-accelerated, Reactant-compiled.** Models are compiled ahead of time into device\n  executables through Reactant's PJRT bindings. The runtime is device-agnostic (CUDA today, CPU\n  for dev/fallback); supporting more accelerators is a goal, not a redesign. → [Architecture](https://enzymead.github.io/ReactantServer.jl/dev/design/architecture/), [Philosophy](https://enzymead.github.io/ReactantServer.jl/dev/design/philosophy/)\n- **Julia-first pre/postprocessing.** A bundle's `model.jl` registers `preprocess`/`postprocess`\n  hooks in plain Julia; they run per request, in parallel and overlapped with GPU execution. → [Bundles \u0026 model.jl](https://enzymead.github.io/ReactantServer.jl/dev/manual/bundles/)\n- **Julia-aligned conventions.** Shapes are column-major with the batch axis last, the way Julia\n  and Lux write them; the codec converts to KServe's row-major wire at the boundary, so Triton\n  clients are unchanged and you never reason about row-major order. → [Getting Started](https://enzymead.github.io/ReactantServer.jl/dev/manual/getting_started/)\n- **Elegant configuration.** One typed YAML node file (with environment-variable overrides)\n  describes a machine; manifests declare tensors with an einsum-style named-axis notation. → [Node Configuration](https://enzymead.github.io/ReactantServer.jl/dev/manual/node_config/), [Bundles \u0026 model.jl](https://enzymead.github.io/ReactantServer.jl/dev/manual/bundles/)\n- **Standard inference protocol.** KServe V2 over gRPC. Tensor data travels inline or through the\n  Triton-compatible system-shared-memory extension for zero-copy local clients. → [Client Usage](https://enzymead.github.io/ReactantServer.jl/dev/manual/client_usage/)\n- **One container, single or multi-GPU.** A node supervisor runs one worker per visible GPU: a\n  single worker serves the public ports directly; two or more get an embedded gateway behind one\n  endpoint. The external interface (`:8001` gRPC, `:8002` metrics/health) is identical either way.\n  → [Docker Deployment](https://enzymead.github.io/ReactantServer.jl/dev/manual/docker/), [Scaling to Multiple GPUs](https://enzymead.github.io/ReactantServer.jl/dev/manual/scaling/)\n- **Balances memory and compute.** Every model's weights stay resident in host RAM and stream\n  onto the GPU on demand, evicted LRU under a byte budget — so a card serves far more models than\n  fit in VRAM, paying a single host-to-device transfer on a cold call. → [On-demand Weights](https://enzymead.github.io/ReactantServer.jl/dev/manual/on_demand_weights/)\n- **Batch coalescing.** Concurrent same-model requests are merged into one execution at a compiled\n  batch size, amortizing per-launch overhead and the one-time weight transfer across the batch.\n  → [Architecture](https://enzymead.github.io/ReactantServer.jl/dev/design/architecture/)\n- **Scheduling modes for single and multi-GPU.** On a worker, `fair` (deficit-weighted,\n  cost-aware) or `fifo`; across GPUs, the gateway offers `round_robin` or memory-aware\n  `lpt_packing` that concentrates each model's traffic to fill batches. → [Architecture](https://enzymead.github.io/ReactantServer.jl/dev/design/architecture/), [Multi-GPU Gateway](https://enzymead.github.io/ReactantServer.jl/dev/manual/multi_gpu_gateway/)\n- **Fast iteration.** In `dynamic` mode the server watches the model repository and hot-loads,\n  unloads, and reloads bundles online — weights, MLIR, manifest, and `model.jl` alike — with no\n  restart (`static` and `explicit` control modes are also available). → [Node Configuration](https://enzymead.github.io/ReactantServer.jl/dev/manual/node_config/)\n- **Meta models.** A `kind: meta` bundle chains several models with data-dependent Julia between\n  stages: its `model.jl` registers a `run` hook that calls sub-models, runs off the GPU dispatch\n  loop, and re-enters the scheduler for each sub-call. → [Meta Models](https://enzymead.github.io/ReactantServer.jl/dev/manual/meta_models/)\n\n## Quick start\n\nThe image is built locally (it is not published to a registry), so build it once and then serve a\ndirectory of model bundles from the container (it scales to all visible GPUs):\n\n```\ngit submodule update --init --recursive   # fetch the vendored lib/gRPCServer.jl fork the build needs\nmake image                                 # build reactantserver:latest (or: docker compose build)\n\ndocker run --gpus all --ipc=host -p 8001:8001 -p 8002:8002 \\\n  -v /path/to/bundles:/var/lib/reactantserver/models:ro reactantserver\n```\n\nThe build is large and the first server startup is slow, since every model compiles before the\ngRPC plane accepts traffic. See [Docker Deployment](https://enzymead.github.io/ReactantServer.jl/dev/manual/docker/) for the\n`docker compose` workflow and configuration.\n\nOr from pure Julia:\n\n```julia\nusing ReactantServerNode\nReactantServerNode.supervise(\"docker/node.yaml\")   # one worker per GPU (+ gateway if \u003e1)\n```\n\nClients speak KServe V2 gRPC to `:8001`; health and metrics are on `:8002`. Walk through exporting\na model, configuring a node, and querying it in [Getting Started](https://enzymead.github.io/ReactantServer.jl/dev/manual/getting_started/).\nReactantServer is designed for a trusted network — read\n[Security](https://enzymead.github.io/ReactantServer.jl/dev/manual/docker/#security) before exposing an endpoint.\n\n## Status\n\nThe full serving path is implemented end to end: export a bundle, compile it through Reactant/PJRT,\nschedule and coalesce requests, and serve over the KServe V2 gRPC control plane. The cost-aware\nscheduler, the on-demand GPU weight cache, dynamic model lifecycle, meta-model orchestration (with\na worked object detection example), and the single- and multi-GPU deployment paths all work today\non CUDA (with CPU for development and fallback); broader accelerator support is intended to follow. Deferred to later milestones: dynamic-batch export with server-side\n`stablehlo-refine` specialization, the compiled-executable disk cache, multi-model orchestrators,\nand full StableHLO-signature validation of manifests. See\n[Architecture](https://enzymead.github.io/ReactantServer.jl/dev/design/architecture/) for the full picture.\n\n## Repository layout\n\nA Julia 1.12+ workspace of five packages under `packages/`, split so that talking to a server never\npulls in the heavy Reactant/XLA stack:\n\n- **`ReactantServerCore`** — shared, Reactant-free substrate: dtypes, the KServe V2 protobuf\n  messages, boundary types, the manifest parser, node config, the wire codec, the shared-memory\n  registry, and the staging `BufferPool`.\n- **`ReactantServer`** — the inference worker (registry, runtime, scheduler, KServe V2 gRPC server);\n  the **only** package that depends on Reactant. Exports `serve`, `stop!`, `register_model`.\n- **`ReactantServerGateway`** — the multi-GPU KServe gRPC reverse proxy (`serve_gateway`). No Reactant.\n- **`ReactantServerClient`** — a Reactant-free inference client (`KServeModel`, `infer_sync`,\n  `infer_async`, `InferInput`, `InferOutput`).\n- **`ReactantServerNode`** — the single-container node supervisor (`supervise`): detects GPUs, runs\n  one worker per device, embeds the gateway when there is more than one, and multiplexes their logs.\n\nOffline model export lives in `packages/ReactantServerExport` (a Reactant tracing frontend plus a\nPythonCall-triggered PyTorch extension); it is deliberately **not** a workspace member, so its\nLux/PythonCall weakdeps stay out of the server images. The vendored `gRPCServer` fork is the only\nremaining git submodule under `lib/`.\n\n## Acknowledgments\n\nDevelopment of ReactantServer.jl is sponsored by [Medical Metrics, Inc.](https://medicalmetrics.com/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fenzymead%2Freactantserver.jl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fenzymead%2Freactantserver.jl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fenzymead%2Freactantserver.jl/lists"}