https://github.com/loonghao/renderacre
Rust render-farm controller, worker, OpenJD, and Python ABI3 toolkit for DCC pipelines.
https://github.com/loonghao/renderacre
blender dcc maturin maya openjd pyo3 render-farm renderfarm rust
Last synced: 4 days ago
JSON representation
Rust render-farm controller, worker, OpenJD, and Python ABI3 toolkit for DCC pipelines.
- Host: GitHub
- URL: https://github.com/loonghao/renderacre
- Owner: loonghao
- License: other
- Created: 2026-05-27T14:15:56.000Z (21 days ago)
- Default Branch: main
- Last Pushed: 2026-05-27T16:10:17.000Z (21 days ago)
- Last Synced: 2026-05-27T16:15:53.443Z (21 days ago)
- Topics: blender, dcc, maturin, maya, openjd, pyo3, render-farm, renderfarm, rust
- Language: Rust
- Homepage: https://pypi.org/project/renderacre/
- Size: 75.2 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 11
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Renderacre
[](https://github.com/loonghao/renderacre/actions/workflows/ci.yml)
[](https://github.com/loonghao/renderacre/actions/workflows/dcc-e2e.yml)
[](https://github.com/loonghao/renderacre/actions/workflows/release.yml)
[](https://github.com/loonghao/renderacre/releases)
[](https://pypi.org/project/renderacre/)
[](https://pypi.org/project/renderacre/)
[](https://pepy.tech/project/renderacre)
[](LICENSE)
Renderacre is a Rust render-farm foundation inspired by Deadline: a remote controller API, standalone workers, Python submitter bindings, and Open Job Description (OpenJD) job templates for portable DCC and batch workloads.
The Python package is named `renderacre` and is built as a `cp37-abi3` wheel, so one wheel supports CPython 3.7 and newer on each platform.
## Installation
Install the Python submitter API from PyPI:
```bash
python -m pip install renderacre
```
The published wheels use `cp37-abi3`, which means each platform wheel supports CPython 3.7 and newer.
Install the standalone controller and worker binaries from GitHub Releases.
Linux/macOS:
```bash
curl -fsSL https://raw.githubusercontent.com/loonghao/renderacre/main/scripts/install.sh | bash
```
Windows PowerShell:
```powershell
iwr https://raw.githubusercontent.com/loonghao/renderacre/main/scripts/install.ps1 -UseB | iex
```
Pin a release or install directory when deploying render nodes:
```bash
export RENDERACRE_VERSION=v0.1.4
export RENDERACRE_INSTALL_DIR=/opt/renderacre/bin
curl -fsSL https://raw.githubusercontent.com/loonghao/renderacre/main/scripts/install.sh | bash
```
```powershell
$env:RENDERACRE_VERSION = "v0.1.4"
$env:RENDERACRE_INSTALL_DIR = "$env:LOCALAPPDATA\renderacre\bin"
iwr https://raw.githubusercontent.com/loonghao/renderacre/main/scripts/install.ps1 -UseB | iex
```
After installation, run the farm processes:
```bash
renderacre-controller --bind 0.0.0.0:7878
renderacre-worker --controller http://controller-host:7878 --name render-node-01 --label app=blender
```
For source builds:
```bash
cargo install --path crates/farm-controller --locked
cargo install --path crates/farm-worker --locked
```
## Features
- Rust HTTP controller with job submission, worker registration, leasing, retries, task dependencies, and job state inspection.
- Standalone worker binary that executes direct commands or OpenJD runtime payloads.
- Controller-collected worker logs, controller events, task stdout/stderr tails, downloadable task artifacts, and upstream/downstream task dependency tracking.
- Official `openjd-model` validation and job creation for OpenJD `jobtemplate-2023-09`.
- Official `openjd-sessions` execution for OpenJD step scripts, task parameters, environments, embedded files, progress/status callbacks, and OpenJD stdout directives.
- OpenJD `hostRequirements` routing through worker capabilities such as `attr.worker.os.family`, `attr.worker.cpu.arch`, and `amount.worker.vcpu`.
- PyO3/maturin Python module for pipeline submitters and DCC tools.
- CI for fmt, clippy, unit tests, controller/worker e2e smoke, and ABI3 wheel builds.
- Release workflow for cross-platform wheels, sdist, and PyPI Trusted Publishing.
## Local Development
```powershell
cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings
powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\e2e_smoke.ps1
python .\scripts\e2e_dcc_tasks.py --jobs python,command
python -m maturin build --release -o target\wheels
```
The wheel filename should contain `cp37-abi3`, for example:
```text
renderacre-0.1.4-cp37-abi3-win_amd64.whl
```
## Run a Local Farm
Start the controller:
```powershell
cargo run -p renderacre-controller -- --bind 127.0.0.1:7878
```
The controller defaults to the in-memory scheduler for demos and tests. Use the
SQLite backend for a lightweight durable farm:
```powershell
cargo run -p renderacre-controller -- --storage sqlite --sqlite-path .\renderacre.sqlite3
```
Start a worker:
```powershell
cargo run -p renderacre-worker -- --controller http://127.0.0.1:7878 --name local-worker --label os=windows
```
Use `--slots ` for a worker that can execute multiple compatible tasks
concurrently. The controller will not lease more active tasks to that worker
than its registered slot capacity.
Submit a direct command job:
```powershell
Invoke-RestMethod `
-Method Post `
-Uri http://127.0.0.1:7878/v1/jobs `
-ContentType application/json `
-InFile .\examples\hello_job.json
```
Tasks can optionally declare routing requirements without changing older job
payloads:
```json
{
"name": "maya-render",
"command": { "executable": "mayapy", "args": ["render.py"] },
"requirements": {
"labels": { "os": "windows", "app": "maya" },
"pools": ["lighting"],
"amounts": [{ "name": "amount.worker.vcpu", "min": 2 }],
"attributes": [
{ "name": "attr.worker.os.family", "anyOf": ["windows"] }
]
}
}
```
Workers advertise matching capabilities with labels such as
`--label os=windows --label app=maya --label pool=lighting`. Renderacre workers
also register default OpenJD capabilities for OS family, CPU architecture, and
slot count, so OpenJD `hostRequirements` work out of the box for common cases.
Operators can define shared limits for scarce licenses or farm-wide resources,
then tasks can declare the named limits they need:
```text
POST /v1/limits
GET /v1/limits
```
```json
{
"name": "maya-render",
"command": { "executable": "mayapy", "args": ["render.py"] },
"limits": ["maya"]
}
```
Common queue operations are available as stable action endpoints:
```text
POST /v1/jobs/{job_id}/pause
POST /v1/jobs/{job_id}/resume
POST /v1/jobs/{job_id}/cancel
POST /v1/jobs/{job_id}/priority
POST /v1/tasks/{task_id}/cancel
POST /v1/tasks/{task_id}/requeue
```
## Python Submitter API
Install a local build:
```powershell
python -m pip install maturin
python -m maturin develop
```
Submit a command job:
```python
import renderacre
job_json = renderacre.command_job(
"hello-from-python",
"python",
["-c", "print('hello from renderacre')"],
)
print(renderacre.submit_job("http://127.0.0.1:7878", job_json))
```
Submit an OpenJD template:
```python
import json
import renderacre
template = open("examples/openjd_python_frames.yaml", encoding="utf-8").read()
job_json = renderacre.openjd_job(
"openjd-demo",
template,
json.dumps({"Message": "hello through OpenJD"}),
)
print(renderacre.submit_job("http://127.0.0.1:7878", job_json))
```
## OpenJD Support
Renderacre accepts an OpenJD job template under `openjd.template_yaml`.
The controller:
1. Parses YAML/JSON with the official OpenJD parser.
2. Validates the template and enabled extensions.
3. Preprocesses typed job parameters, including `PATH` values.
4. Creates the resolved OpenJD job model.
5. Expands step parameter spaces into farm tasks.
6. Preserves OpenJD `hostRequirements` as scheduler requirements.
The worker:
1. Creates an OpenJD session.
2. Enters job and step environments.
3. Runs each OpenJD task through `openjd-sessions`.
4. Streams OpenJD action status/progress callbacks and captures stdout/stderr.
5. Exits environments and cleans the session directory.
Supported current extensions default to all extensions known by `openjd-model`: `EXPR`, `FEATURE_BUNDLE_1`, `TASK_CHUNKING`, and `REDACTED_ENV_VARS`.
For a routing-focused example, see `examples/openjd_host_requirements.yaml`.
## Dashboard
Renderacre includes a Vite + React dashboard under `dashboard/`. It uses modular React components, compact queue tables, worker status panels, per-worker live logs, an OpenJD task inspector, React Flow dependency graphs, downloadable artifacts, and stdout/stderr tail views.
Run it during development:
```powershell
cargo run -p renderacre-controller -- --bind 127.0.0.1:7878
cd dashboard
npm ci
npm run dev
```
The Vite dev server proxies `/v1` and `/healthz` to the controller. For a deployed dashboard:
```powershell
cd dashboard
npm run build
```
Serve `dashboard/dist` through the controller with `--dashboard-dir dashboard/dist`, or through your internal web server or reverse proxy next to the controller API.
## Command and DCC Examples
Renderacre can run any executable the worker can spawn: `cmd.exe`, PowerShell,
`pwsh`, `bash`, `sh`, Blender, Maya, `ffmpeg`, ImageMagick, studio launchers, or
your own tools. Generic command frames are the quickest local test because they
submit three frame tasks per shell, write one text artifact per frame, echo the
artifact path with `RENDERACRE_ARTIFACT=...`, and verify the worker logs plus the
download API.
Run command frame tests against the shells available on the current machine:
```powershell
python .\scripts\e2e_dcc_tasks.py --jobs command --shells auto
```
Pin specific shell runners when testing Windows and POSIX wrapper behavior:
```powershell
python .\scripts\e2e_dcc_tasks.py --jobs command --shells cmd,powershell,bash
```
Frame outputs are written under
`target/e2e-dcc/command-frames//*_frame_0001.txt` and uploaded by CI as
`command-frame-artifacts`.
Renderacre also ships real OpenJD examples for Python frame tasks, Blender
background renders, and Maya standalone scene exports. The same templates are
exercised by the `DCC E2E` GitHub Actions workflow: the generic command job uses
the runner shells, Blender uses official Linux tarballs, and Maya uses the
`tahv/mayapy` container images.
Run the full DCC e2e suite on a machine that has Blender and Maya on `PATH`:
```powershell
python .\scripts\e2e_dcc_tasks.py --jobs all --blender-exe blender --maya-python mayapy
```
Run only the jobs available on the current machine:
```powershell
python .\scripts\e2e_dcc_tasks.py --jobs auto
```
Blender:
```powershell
$template = Get-Content .\examples\dcc\blender_render_openjd.yaml -Raw
$params = @{
BlenderExecutable = "blender"
ScriptPath = (Resolve-Path .\examples\dcc\blender_render_task.py).Path
OutputDir = (Join-Path (Get-Location) "renders\blender")
} | ConvertTo-Json
```
Maya:
```powershell
$template = Get-Content .\examples\dcc\maya_render_openjd.yaml -Raw
$params = @{
MayaPython = "mayapy"
ScriptPath = (Resolve-Path .\examples\dcc\maya_render_task.py).Path
OutputDir = (Join-Path (Get-Location) "renders\maya")
} | ConvertTo-Json
```
Pass the template and parameter JSON through `renderacre.openjd_job(...)` or submit the equivalent REST payload to `/v1/jobs`.
For CI parity, the script writes command `.txt` frames, Blender PNGs, and Maya
`.ma` scenes under `target/e2e-dcc/` and fails if any expected frame output,
artifact download, or worker log line is missing.
## Deployment
Recommended first deployment shape:
- Run one controller per farm or queue: `renderacre-controller --config deploy/lightweight/controller.yaml`.
- Run one worker process per render node: `renderacre-worker --controller http://controller-host:7878 --name --label app=blender`.
- Put the controller behind a private network or authenticated reverse proxy.
- Keep render executables and scripts on shared storage, then pass `PATH` parameters through OpenJD.
- Use GitHub Releases or PyPI to distribute the `renderacre` wheel to submitter machines.
SQLite is the default durable profile for small deployments. Start the
controller with `--storage sqlite --sqlite-path ` or set
`RFARM_STORAGE=sqlite` and `RFARM_SQLITE_PATH`. Use `--dashboard-dir ` or
`RFARM_DASHBOARD_DIR` to serve a built dashboard from the controller in the
single-node profile. The scheduler API remains the same for REST workers,
dashboard reads, and Python submitters; future Postgres or managed/cloud storage
backends can replace the same storage boundary without changing submitter
contracts.
Cloud-ready backend, artifact, worker identity, and scheduler extension
contracts are documented in
[docs/extension-contracts.md](docs/extension-contracts.md).
The one-command lightweight profile, service examples, and upgrade/backup
guidance are documented in [docs/deployment.md](docs/deployment.md).
## Release
The release workflow builds Linux, Windows, and macOS wheels plus an sdist. PyPI publishing uses Trusted Publishing through the `pypi` GitHub environment.
It also uploads standalone controller/worker archives to GitHub Releases for Linux, macOS, and Windows.
Manual release dry run:
```powershell
python -m maturin build --release -o dist
python -m pip install twine
python -m twine check dist/*
```
Publish from GitHub:
1. Configure PyPI Trusted Publisher for `loonghao/renderacre`, workflow `release.yml`, environment `pypi`.
2. Push a `vX.Y.Z` tag or publish a GitHub Release.
3. The workflow uploads artifacts to PyPI when the publish gate is active.
See [docs/architecture.md](docs/architecture.md) for the internal layout.