{"id":50360482,"url":"https://github.com/redbase-app/redb-tsak","last_synced_at":"2026-05-30T01:05:57.031Z","repository":{"id":356615744,"uuid":"1233259956","full_name":"redbase-app/redb-tsak","owner":"redbase-app","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-16T11:17:55.000Z","size":650,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-16T13:33:22.030Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/redbase-app.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-08T19:07:49.000Z","updated_at":"2026-05-16T11:17:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/redbase-app/redb-tsak","commit_stats":null,"previous_names":["redbase-app/redb-tsak"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/redbase-app/redb-tsak","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redbase-app%2Fredb-tsak","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redbase-app%2Fredb-tsak/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redbase-app%2Fredb-tsak/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redbase-app%2Fredb-tsak/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/redbase-app","download_url":"https://codeload.github.com/redbase-app/redb-tsak/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redbase-app%2Fredb-tsak/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33676202,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-30T01:05:56.157Z","updated_at":"2026-05-30T01:05:57.020Z","avatar_url":"https://github.com/redbase-app.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# redb.Tsak\n\n\u003e **The runtime container for [redb.Route](https://github.com/redbase-app/redb-route) integration pipelines.**\n\u003e Deploy DLLs into a folder. Start, stop, reload them at runtime. Scale across nodes with leader election. Watch every message, every error, every restart — from a built-in REST API, a 30-command CLI, or a Blazor dashboard. Zero downtime. Production-tested.\n\n[![License: Apache 2.0](https://img.shields.io/badge/license-Apache_2.0-blue)](LICENSE)\n[![.NET](https://img.shields.io/badge/.NET-8%20%7C%209%20%7C%2010-purple)](https://dotnet.microsoft.com)\n[![Tests](https://img.shields.io/badge/tests-351%20passing-brightgreen)](#testing)\n[![Status](https://img.shields.io/badge/status-production-green)](#)\n[![REST API](https://img.shields.io/badge/REST_API-32_endpoints-blue)](#rest-api)\n[![CLI](https://img.shields.io/badge/CLI-30_commands-blue)](#cli)\n[![Web UI](https://img.shields.io/badge/Web_UI-Blazor_Server-512BD4)](#web-dashboard)\n[![Cluster](https://img.shields.io/badge/cluster-leader_election-orange)](#cluster-mode)\n\n---\n\n## What you get out of the box\n\n| | |\n|---|---|\n| **Module-based deployment** | Drop a `.dll` into `Libs/` — Tsak loads it, builds an `IRouteContext`, starts the routes. Update the DLL — Tsak hot-swaps it without dropping a single in-flight message in other contexts. |\n| **Three deployment modes** | `Standalone` (in-memory, no DB) · `Single-node + redb` (durable EAV state) · `Cluster` (leader election + automatic context redistribution across nodes). |\n| **Three API modes** | `Full` (REST API + management) · `Headless` (zero ports, embedded use) · `Readonly` (only `GET` endpoints — perfect for monitoring sidecars). |\n| **5-layer configuration** | Module ships defaults. DevOps overrides. Config hot-reloads. No code changes, no restarts. |\n| **Built-in observability** | Per-process metrics (CPU/RAM/threads/GC, 12h history), per-route metrics (count/error rate/latency), ring-buffer logs, OpenTelemetry traces, optional Prometheus scrape. |\n| **Watchdog** | Detects suspected and hung routes. Optionally auto-restarts them. |\n| **Quartz scheduler** | Built-in `IScheduler` injected into every context. `RAMJobStore` standalone, `AdoJobStore` cluster-safe — schema auto-created on first start. |\n| **Security** | API Key + HMAC-SHA256 + roles + expiry + revocation. Constant-time comparison. EAV-backed key store. Optional user binding (disabled user → key dies). |\n| **Module isolation** | Per-module `AssemblyLoadContext` — dependencies don't conflict between modules. |\n| **Clients ready to ship** | Typed C# `ITsakApiClient`, 30-command `tsak` CLI with profiles and JSON output, Blazor Server dashboard. |\n\n---\n\n## Why redb.Tsak exists\n\nMost .NET teams either:\n\n1. **Bake their integration code into a Worker Service** — every change means redeploy the entire process. Multiple unrelated pipelines fight in the same `Program.cs`.\n2. **Buy a heavyweight enterprise ESB** — pay six figures for a UI, a runtime, a management API, and a deployment workflow they barely use.\n3. **Roll their own management plane** — and re-invent metrics, hot-reload, leader election, and an admin API for the third time this decade.\n\nTsak is the missing piece between \"a Worker Service\" and \"an enterprise ESB\":\n\n| | Hand-rolled Worker | Tsak | Heavy ESB |\n|---|---|---|---|\n| Deploy a new pipeline | Redeploy whole process | `cp module.dll Libs/` (hot-reload) | Vendor wizard, hours |\n| Stop one pipeline without affecting others | Custom code | `tsak context stop orders` | Vendor UI |\n| Distribute pipelines across nodes | Custom coordinator | Built-in leader election + auto-rebalance | Vendor cluster |\n| REST API for ops/CI | Build it yourself | 32 endpoints, typed client | Yes, vendor-locked |\n| CLI for CI/CD | Build it yourself | 30 commands, profiles, JSON output | Maybe |\n| Web dashboard | Build it yourself | Blazor Server, 10 pages | Yes, vendor-locked |\n| Cost | Engineering time | Apache 2.0, free | $$$$ + lock-in |\n| Routing engine | DIY or MassTransit | **redb.Route** — 22 transports, 24 EIP processors, fluent DSL | Vendor's DSL |\n\n---\n\n## For redb.Route users\n\nIf you already write `RouteBuilder` classes for [redb.Route](https://github.com/redbase-app/redb-route), Tsak is what runs them in production. You do not change a single line of route code.\n\n```\n┌──────────────────────┐         ┌──────────────────────┐         ┌──────────────────────┐\n│   Your routes        │  ---\u003e   │   class library      │  ---\u003e   │  Tsak picks it up    │\n│   (RouteBuilder)     │ publish │   (Orders.dll)       │   cp    │  hot-reload + run    │\n└──────────────────────┘         └──────────────────────┘         └──────────────────────┘\n```\n\nWhat Tsak adds **on top** of a plain `redb.Route` worker:\n\n| Pure `redb.Route` worker (`Host.CreateApplicationBuilder`) | `redb.Route` inside Tsak |\n|---|---|\n| One `Program.cs` wires every `RouteBuilder` at compile time. | Drop the DLL into `Libs/` — Tsak loads it. |\n| Add a route → redeploy the whole process. | Hot-reload one module without touching the others. |\n| One process = one bag of routes. | Multiple **named contexts**, each with its own properties, lifecycle and assembly load context. |\n| Stop one route = stop the process. | `tsak route stop \u003cctx\u003e \u003cid\u003e` or `POST /api/routes/{id}/stop`. |\n| Distribute across nodes = roll your own coordinator. | Built-in leader election + per-context assignment across cluster nodes. |\n| Schedule a job = wire Quartz yourself in `Program.cs`. | `IScheduler` injected into every context, schema auto-bootstrapped. |\n| Operate it = parse logs, build a dashboard, expose metrics yourself. | REST + CLI + Blazor + Prometheus + OTel out of the box. |\n\nA `RouteBuilder` you wrote against `redb.Route`'s plain `IHostedService` runs unchanged inside a Tsak module — the same `Configure()` is called, the same `IExchange` flows through, the same `OnException` and `.Transacted()` semantics apply. Tsak only owns **how** the routes are loaded, scheduled, observed and torn down.\n\n```csharp\n// MyRoutes/InitRoute.cs — the only Tsak-specific piece\npublic static class InitRoute\n{\n    public static IRouteContext main(IRouteContext context)\n    {\n        // Plain redb.Route — exactly what you would write outside Tsak\n        ((RouteContext)context).AddRoutes(new OrderRoutes());\n        ((RouteContext)context).AddRoutes(new ShipmentRoutes());\n        return context;\n    }\n}\n```\n\nFor the routing DSL itself (transports, EIP patterns, expressions, transactional pipelines, OnException, error handling), see the [redb.Route README](https://github.com/redbase-app/redb-route). Everything below in this document is about the **container**.\n\n---\n\n## Architecture\n\n### Process layout\n\n```\n┌────────────────────────────────────────────────────────────────────┐\n│                        redb.Tsak.Worker                            │\n│                                                                    │\n│  ┌──────────────────────────────────────────────────────────────┐ │\n│  │  _system context (protected, cannot be stopped or removed)   │ │\n│  │  ────────────────────────────────────────────────────────── │ │\n│  │  HTTP listener  →  AuthMiddleware  →  ControllerDispatcher   │ │\n│  │  /api/auth        Bearer / X-Api-Key   AuthController         │ │\n│  │  /api/contexts    role check           ContextsController     │ │\n│  │  /api/modules                          ModulesController      │ │\n│  │  /api/cluster                          ClusterController      │ │\n│  │  /api/system                           SystemController       │ │\n│  │  /api/scheduler                        SchedulerController    │ │\n│  │  /api/logs                             LogsController         │ │\n│  │  /api/users                            UsersController        │ │\n│  │  /api/watchdog  /api/diagnostics  /api/lifecycle  /api/routes │ │\n│  └──────────────────────────────────────────────────────────────┘ │\n│                                                                    │\n│  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐      │\n│  │   Context A    │  │   Context B    │  │   Context C    │      │\n│  │   \"orders\"     │  │   \"payments\"   │  │   \"analytics\"  │      │\n│  │ ────────────── │  │ ────────────── │  │ ────────────── │      │\n│  │ Module:        │  │ Module:        │  │ Modules:       │      │\n│  │  Api.Orders    │  │  Pay.Stripe    │  │  Etl.Loader    │      │\n│  │ Route 1: Kafka │  │ Route 1: HTTP  │  │  Etl.Reports   │      │\n│  │ Route 2: Cron  │  │ Route 2: AMQP  │  │ Route 1..N     │      │\n│  │ Route N: HTTP  │  │ Route N: Mail  │  │                │      │\n│  │ ALC: isolated  │  │ ALC: isolated  │  │ ALC: isolated  │      │\n│  └────────────────┘  └────────────────┘  └────────────────┘      │\n│                                                                    │\n│  ┌──────────────────────────────────────────────────────────────┐ │\n│  │  Infrastructure services                                      │ │\n│  │  ────────────────────────────────────────────────────────── │ │\n│  │  HotReloadService   ClusterCoordinator    QuartzScheduler    │ │\n│  │  ModuleRegistry     LeaderElection        WatchdogService    │ │\n│  │  ContextManager     NodeRegistry          MetricsService     │ │\n│  │  ConfigMerger       AssignmentManager     LogRingBuffer      │ │\n│  │                                            HealthCheckService │ │\n│  └──────────────────────────────────────────────────────────────┘ │\n└────────────────────────────────────────────────────────────────────┘\n                                  │\n                                  ▼\n                ┌─────────────────────────────────┐\n                │  redb EAV (Postgres or MSSQL)   │\n                │  - API keys (RedbApiKeyStore)   │\n                │  - Cluster topology (Tree)      │\n                │  - Lifecycle events             │\n                │  - Idempotent state             │\n                │  - Quartz QRTZ_* tables         │\n                └─────────────────────────────────┘\n```\n\n### Anatomy of a request\n\n```\nExternal call:  curl -H \"Authorization: Bearer $KEY\" http://node:9090/api/contexts/orders/start\n       │\n       ▼\nHTTP listener (Kestrel) inside _system context — port 9090\n       │\n       ▼\nHeaderBridge processor      → normalizes route.path = \"contexts/orders/start\", route.method = \"POST\"\n       │\n       ▼\nAuthorizeProcessor          → resolves API key (5min cache) → validates role → enriches Exchange with claims\n       │\n       ▼\nControllerDispatcher        → looks up [Route(\"contexts/{name}/start\")] on ContextsController\n       │\n       ▼\nContextsController.Start    → ITsakContextManager.StartAsync(\"orders\")\n       │\n       ▼\n                            → graceful start: load assemblies → wire routes → connect transports → mark Running\n       │\n       ▼\nJSON response               → { \"name\": \"orders\", \"status\": \"Running\", \"startedAt\": \"...\" }\n```\n\nThe same `ContextsController` can also be invoked over **RabbitMQ RPC**, **gRPC**, **SignalR**, or any other `redb.Route` `InOut` transport — the dispatcher is transport-agnostic. This is how Tsak avoids tying its management API to HTTP-only.\n\n---\n\n## Project structure\n\n```\nredb.Tsak/\n├── src/\n│   ├── redb.Tsak.Core/          Kernel: contracts, controllers, services,\n│   │                            security, cluster, hot-reload, monitoring\n│   ├── redb.Tsak.Core.Pro/      Pro extensions (cluster, hooks, monitoring)\n│   ├── redb.Tsak.Worker/        Hosted process: DI wiring, Serilog,\n│   │                            Quartz, appsettings.json, Dockerfile\n│   ├── redb.Tsak.Contracts/     Wire DTOs (System.Text.Json only) shared\n│   │                            between Core, Client, Web, CLI\n│   ├── redb.Tsak.Client/        ITsakApiClient + TsakApiClient (HTTP)\n│   ├── redb.Tsak.CLI/           tsak — 30 commands, profiles, JSON output\n│   ├── redb.Tsak.Web/           Blazor Server dashboard, custom design system\n│   └── redb.Tsak.Web.Pro/       Pro web extensions (auth, node providers)\n├── tests/\n│   ├── redb.Tsak.Tests/         287 unit + integration tests\n│   └── redb.Tsak.CLI.Tests/     64 CLI command tests\n├── docs/\n│   ├── PLAN.md                  Architecture reference\n│   ├── CONFIG_GUIDE.md          5-layer configuration deep-dive\n│   ├── LT_TSAK_ANALYSIS.md      Analysis of the original lt.tsak\n│   ├── ENTERPRISE_AUDIT.md\n│   └── phases/                  Per-phase implementation notes (0..8B)\n├── README.md                    This file\n├── CHANGELOG.md                 Per-version changes\n├── CONTRIBUTING.md              Contribution guide\n├── SECURITY.md                  Security policy\n├── DEPLOYMENT_SECRETS.md        Secrets management for production\n└── LICENSE                      Apache 2.0\n```\n\n---\n\n## Quick start\n\n### Option A — Docker (fastest)\n\nPre-built images are published to GitHub Container Registry — no build step, no .NET SDK required:\n\n```bash\n# Worker only (REST API + cluster, no UI)\ndocker run -d --name tsak \\\n  -p 9090:9090 \\\n  -v $PWD/Libs:/app/Libs \\\n  ghcr.io/redbase-app/redb-tsak-worker:latest\n\n# Stack (Worker + Web UI in one container, like rabbitmq:management)\ndocker run -d --name tsak \\\n  -p 9090:9090 -p 8080:8080 \\\n  -v $PWD/Libs:/app/worker/Libs \\\n  ghcr.io/redbase-app/redb-tsak-stack:latest\n# UI: http://localhost:8080  ·  REST: http://localhost:9090\n```\n\n| Image | Contains | Size | Ports |\n|---|---|---|---|\n| `ghcr.io/redbase-app/redb-tsak-worker` | Worker (REST + cluster) | ~360 MB | `9090` |\n| `ghcr.io/redbase-app/redb-tsak-web`    | Standalone Web UI       | ~250 MB | `8080` |\n| `ghcr.io/redbase-app/redb-tsak-stack`  | Worker + Web (supervisord) | ~430 MB | `9090`, `8080` |\n\nTags: `latest`, `\u003cversion\u003e` (e.g. `2.0.0`), `\u003cversion\u003e-net9` (Worker also `-net8`, `-net10`).\n\n**With PostgreSQL (durable EAV state, multi-node, cluster):**\n\n```bash\ndocker run -d --name tsak \\\n  -p 9090:9090 -p 8080:8080 \\\n  -v $PWD/Libs:/app/worker/Libs \\\n  -e ConnectionStrings__Postgres=\"Host=pg;Port=5432;Database=redb_tsak;Username=postgres;Password=postgres\" \\\n  ghcr.io/redbase-app/redb-tsak-stack:latest\n```\n\nReady-to-use compose files (worker / web / stack / full-with-postgres) live in [`publish/docker/`](publish/docker/).\n\n### Option B — Standalone archive (no Docker)\n\nSelf-contained archives (no .NET runtime required on host) are attached to every [GitHub Release](https://github.com/redbase-app/redb-tsak/releases):\n\n| File | Platform |\n|---|---|\n| `redb-tsak-\u003cversion\u003e-linux-x64.tar.gz` | Linux x64 |\n| `redb-tsak-\u003cversion\u003e-win-x64.zip`      | Windows x64 |\n\nEach archive bundles `worker/`, `web/`, `cli/`, all 20 Route connectors in `worker/Libs/shared/`, and startup scripts (`start-worker.sh`/`.bat`/`.ps1`, same for web and stack).\n\n```bash\ncurl -LO https://github.com/redbase-app/redb-tsak/releases/latest/download/redb-tsak-2.0.2-linux-x64.tar.gz\ntar xzf redb-tsak-2.0.2-linux-x64.tar.gz\ncd redb-tsak-2.0.2-linux-x64\n./scripts/start-stack.sh   # worker on :9090, web on :8080\n```\n\n### Verifying signatures (recommended)\n\nAll images and archives are signed with [cosign](https://docs.sigstore.dev/cosign/installation/). Public key: [`publish/keys/cosign.pub`](publish/keys/cosign.pub).\n\n```bash\n# Image\ncosign verify --key cosign.pub ghcr.io/redbase-app/redb-tsak-worker:2.0.2\n\n# Archive\ncosign verify-blob --key cosign.pub \\\n  --bundle redb-tsak-2.0.2-linux-x64.tar.gz.bundle \\\n  redb-tsak-2.0.2-linux-x64.tar.gz\n```\n\nSHA256 sums for every artifact are in `checksums.txt` on the release page.\n\n### Option C — Build from source\n\n### 1. Run a Tsak worker (no database, in-memory)\n\n```bash\ncd redb.Tsak/src/redb.Tsak.Worker\ndotnet run\n```\n\n```\n[INF] redb.Tsak.Worker starting…\n[INF] Storage: InMemory\n[INF] Cluster: disabled\n[INF] HotReload: enabled (scan every 10s)\n[INF] HTTP API listening on http://0.0.0.0:9090\n[INF] _system context started\n[INF] Discovered 0 modules in Libs/\n[INF] Ready\n```\n\n```bash\ncurl http://localhost:9090/api/system/health\n# { \"status\": \"Healthy\", \"checks\": [...] }\n```\n\n### 2. Deploy your first module\n\nA Tsak module is a plain .NET class library exposing one of two well-defined entry-point shapes: a `public static class InitRoute` with `public static IRouteContext main(IRouteContext ctx)` (Apache Camel-style convention, shown below), or a concrete public type implementing `ITsakModule`. Inside the entry point you wire up your `RouteBuilder` subclasses against the supplied `IRouteContext`.\n\n```csharp\n// MyRoutes/InitRoute.cs\npublic static class InitRoute\n{\n    public static IRouteContext main(IRouteContext context)\n    {\n        ((RouteContext)context).AddRoutes(new OrderRoutes());\n        return context;\n    }\n}\n\n// MyRoutes/OrderRoutes.cs\npublic class OrderRoutes : RouteBuilder\n{\n    protected override void Configure()\n    {\n        From(\"kafka://orders?brokers=localhost:9092\u0026groupId=svc\")\n            .Filter(e =\u003e e.Message.GetHeader\u003cstring\u003e(\"type\") == \"new\")\n            .Process(async (e, ct) =\u003e await ProcessAsync(e, ct))\n            .To(\"rabbitmq://processed?host=localhost\");\n    }\n}\n```\n\n```bash\n# Build and deploy\ndotnet publish MyRoutes -c Release -o publish/\ncp -r publish/* /opt/tsak/Libs/MyRoutes/\n\n# Tsak picks it up within HotReload:ScanIntervalSeconds (default 10s)\n```\n\n```bash\ntsak context list\n# Name      Status   Routes  Endpoints  Modules\n# _system   Running       11          1   (system)\n# MyRoutes  Running        1          2   MyRoutes\n```\n\n### 3. Manage from the CLI\n\n```bash\ntsak login http://localhost:9090 --key $TSAK_KEY\ntsak health\ntsak context restart MyRoutes\ntsak logs --level Error --limit 50\ntsak metrics\ntsak scheduler jobs\ntsak cluster overview\n```\n\n### 4. Open the dashboard\n\n```bash\ncd redb.Tsak/src/redb.Tsak.Web\ndotnet run\n# Browse to http://localhost:5100\n```\n\n---\n\n## Module workflow\n\nTsak supports **two equivalent deployment formats** under `Libs/`. Both are scanned by `HotReloadService`, hot-reloaded on file change, gracefully unloaded on file removal — pick whichever fits your workflow.\n\n| Format | What it is | When to pick it |\n|---|---|---|\n| **Bare directory** | A folder of loose `*.dll` files (+ optional config) | Local dev, fast inner loop, atomic file replace via `cp -r` |\n| **`.tpkg` package** | A single ZIP archive bundling manifest + DLLs + config | CI/CD, immutable artifacts, one-file deploys, atomic rollback |\n\n### What Tsak actually loads\n\nTsak does **not** load arbitrary .NET DLLs. Each candidate assembly is scanned for one of two well-defined module shapes:\n\n1. A concrete public type implementing `ITsakModule`.\n2. A public static class named `InitRoute` exposing a `static IRouteContext main(IRouteContext ctx)` method (Apache Camel-style entry point convention).\n\nIf neither shape is found, the assembly is **classified as a dependency, not a module**:\n\n- It is registered in `LoadedAssemblyTracker` so that other modules in the process can resolve it as a transitive dependency (same `Assembly` instance everywhere — no type-identity drift across ALCs).\n- Its file timestamp is recorded in an internal \"ignored\" set; subsequent scans skip it unless the file is replaced with a newer version (which triggers a re-scan in case the new bits *do* contain a module).\n\nSo the rule is: **random DLLs are not loaded as modules — but DLLs that other modules depend on are loaded and made available, just not started as routes.** This keeps the runtime predictable and prevents random vendor SDKs sitting in `Libs/` from being treated as deployable units.\n\n### Format 1 — Bare directory layout\n\n```\nLibs/\n└── Orders/\n    ├── Orders.dll                  ← module entry point (contains ITsakModule or InitRoute.main)\n    ├── Orders.deps.json            ← dependency graph (from dotnet publish)\n    ├── Orders.Domain.dll           ← companion dependency (no module → loaded as dep only)\n    ├── Newtonsoft.Json.dll         ← any third-party dep that isn't already in the host\n    ├── context.json                ← infrastructure defaults (Layer 3)  — optional\n    └── Orders.config.json          ← module business settings (Layer 4) — optional\n```\n\nHow the loader treats each file:\n\n| File | Role |\n|---|---|\n| `Orders.dll` | Loaded into a per-module isolated `ModuleAssemblyLoadContext`, scanned for `ITsakModule`/`InitRoute.main`, registered as a module |\n| `Orders.Domain.dll` | Scanned, no module shape found → kept as a shared dependency (resolvable by other ALCs) |\n| `*.deps.json` | Used by `dotnet publish` to record the full dep graph (Tsak does not parse it directly, but it is what makes `CopyLocalLockFileAssemblies=true` work) |\n| `context.json` | Layer 3 of the [5-layer config merge](#5-layer-configuration) |\n| `{Module}.config.json` | Layer 4 of the same merge |\n\n### Format 2 — `.tpkg` package (atomic, immutable, isolated)\n\nA `.tpkg` is a plain ZIP archive bundling everything a module needs into a single file. Drop one file into `Libs/`, Tsak picks it up and loads it as one atomic unit. Replace or delete that one file, the whole bundle reloads or shuts down together — no half-state in the middle.\n\n```\nOrders.tpkg                          (ZIP archive, flat structure)\n├── manifest.json                    ← REQUIRED — package metadata\n├── Orders.dll                       ← entry point (declared in manifest.EntryPoints)\n├── Orders.config.json               ← optional Layer 4 config\n├── Orders.Domain.dll                ← companion dependency (loaded into shared tracker)\n├── Orders.Models.dll                ← companion dependency\n└── Newtonsoft.Json.dll              ← any other transitive dep not provided by the host\n```\n\n`manifest.json` schema (matches `redb.Tsak.Core.Modules.ModuleManifest`):\n\n```json\n{\n  \"Name\": \"Orders\",\n  \"Version\": \"1.0.0\",\n  \"EntryPoints\": [ \"Orders.dll\" ],\n  \"Dependencies\": []\n}\n```\n\n| Field | Meaning |\n|---|---|\n| `Name` | Logical package name. Must be unique across all packages loaded in the process. |\n| `Version` | Informational version string used in logs and diagnostics. |\n| `EntryPoints` | List of DLL filenames inside the archive that contain `ITsakModule` or `InitRoute.main`. Each entry is loaded into the **package's isolated ALC** and scanned for modules. |\n| `Dependencies` | Informational list of other packages this one logically depends on (used in diagnostics; not enforced by the loader). |\n\nA single `.tpkg` may declare **multiple entry points** — for example a `Core` module and an `Api` module that ship together and share a private dependency tree:\n\n```json\n{\n  \"Name\": \"tsum\",\n  \"Version\": \"1.0.0\",\n  \"EntryPoints\": [ \"tsum.Core.dll\", \"tsum.Api.dll\" ],\n  \"Dependencies\": []\n}\n```\n\nBoth entry-point DLLs land in the same isolated ALC, so they share static state and can pass non-public types between each other without serialization.\n\n### Isolation model — what is shared, what is not\n\nTsak runs many modules in one .NET process without letting them collide. The isolation model has three layers:\n\n| Layer | What lives here | Sharing semantics |\n|---|---|---|\n| **Default ALC** (host) | Tsak Worker, redb.Core, redb.Route abstractions, all NuGet deps the host ships with | Singleton — all modules see the same `IRouteContext`, `IRedbService`, etc. type identity. This is what makes interfaces work across the boundary. |\n| **`LoadedAssemblyTracker`** (shared dependency layer in Default ALC) | Companion DLLs from `.tpkg` files, bare-DLL non-module dependencies | One `Assembly` instance per assembly name across the whole process. If two packages each ship `MyCommon.dll` v1.5, the second one **reuses** the first load (last-loaded-wins on `forceReload=true` for hot updates). |\n| **Per-package `ModuleAssemblyLoadContext`** | Entry-point DLLs declared in a package's `manifest.json` | **Isolated.** Each package gets its own ALC. Two packages can ship two different versions of the same entry-point assembly without conflict. Resolving a missing reference falls through: package ALC → tracker → host Default ALC. |\n\nConcretely, this means:\n\n- **Host contracts (`IRouteContext`, `ITsakModule`, `IRedbService`, …) always resolve to the host's Default ALC.** A module never accidentally rebinds them to a private copy.\n- **Companion DLLs are de-duplicated.** If your package ships `Microsoft.Extensions.Logging.Abstractions.dll` and the host already has it, the host wins. If the host doesn't have it, the first package to ship it wins for the whole process.\n- **Entry-point code is fully isolated per package.** Module A cannot reach into Module B's static fields, even if they both extend the same base class — different ALC, different `Type` instance.\n- **One bad package does not poison the host.** A type-load failure in `Orders.tpkg` is logged and `Orders.tpkg` is skipped; every other package keeps running.\n\n### How Tsak loads a module\n\n1. `HotReloadService` scans every directory under `Tsak:Modules:AssemblyPaths` for `*.dll` and `*.tpkg`.\n2. For each file the loader checks `LastWriteTimeUtc` against its known snapshot — unchanged files are skipped instantly.\n3. **For a `.tpkg`**: the manifest is read, **companion DLLs load first** (so dependency resolution works), then **entry-point DLLs load into a fresh per-package ALC**.\n4. **For a bare `.dll`**: the file loads into a fresh per-module ALC. If `ITsakModule`/`InitRoute.main` is found → it becomes a module; otherwise it is registered as a shared dependency and remembered as \"not a module\".\n5. `ConfigMerger` deep-merges all 5 config layers into a single `IDictionary\u003cstring, object?\u003e` and exposes it as the context's property bag.\n6. `TsakContextManager` creates an `IRouteContext`, registers Quartz `IScheduler`, and either invokes `InitRoute.main(ctx)` or instantiates the `ITsakModule`.\n7. The context starts (if `AutoStart = true`): transports connect, consumers begin reading, routes go live.\n\n### Hot-swap\n\nReplace the artifact on disk — that is the whole API.\n\n```bash\n# Bare directory: atomic rsync over existing files\nrsync -a ./publish/ /opt/tsak/Libs/Orders/\n\n# .tpkg: replace the single file\ncp ./output/Orders.tpkg /opt/tsak/Libs/\n```\n\n`HotReloadService` detects the timestamp change, performs a **graceful swap** (start new ALC → wait for it to settle → drain old context's in-flight exchanges → stop old context → optionally unload old ALC). With `Cluster:Enabled = true` and `HotReload:RollingUpdate = true`, nodes update sequentially — there is **never a moment when zero nodes are running the new version**, and **never a moment when in-flight messages are dropped**.\n\n### Graceful shutdown when an artifact is removed\n\nDeleting a file from `Libs/` is a first-class deployment operation, not an error condition. The runtime treats it as an explicit \"stop and unload\" command:\n\n| You do this | Tsak does this |\n|---|---|\n| `rm Libs/Orders.tpkg` | After `RemovalDebounceScans` confirm the file is gone, **all modules from that package are unloaded atomically**: each context stops gracefully (drain in-flight, close transports, close DB connections), the package's isolated ALC is disposed. Other packages keep running. |\n| `rm Libs/Orders/Orders.dll` | Same flow scoped to one bare-DLL module: graceful context stop → unregister module → dispose its ALC. |\n| `rm -rf Libs/Orders/` | Same as above for every module file in the directory. |\n\nThe debounce (`HotReload:RemovalDebounceScans`, default `2` scans) protects against false positives during atomic file replacements where a deployment briefly removes the file before writing the new one.\n\n### Hot-reload knobs\n\n| Key | Default | Behavior |\n|---|---|---|\n| `Tsak:Modules:AssemblyPaths` | `[\"Libs\"]` | Directories to scan for `*.dll` and `*.tpkg` files. |\n| `HotReload:ScanIntervalSeconds` | `10` | How often to scan the configured paths. |\n| `HotReload:KeepVersions` | `2` | Old versions kept in memory for one-command rollback (bare-DLL flow). |\n| `HotReload:StartupTimeoutSeconds` | `60` | Wait time for the new version to settle before retiring the old one. |\n| `HotReload:RollingUpdate` | `true` | In a cluster, update nodes sequentially. |\n| `HotReload:Collectible` | `false` | Enable `AssemblyLoadContext.Unload()` for full GC reclamation. **Do not enable** if your modules use `Reflection.Emit` (XmlSerializer, source-gen serializers, compiled regex) — set to `false` (default) and accept that old ALCs stay in memory until process restart. The number of accumulated non-collectible ALCs is exposed as `LeakedAlcCount` for monitoring. |\n| `HotReload:RemovalDebounceScans` | `2` | Number of consecutive scan cycles a missing file must persist before its module is unloaded — protects against false positives during atomic file replacement. |\n\n### Building a `.tpkg`\n\nTwo production-tested approaches ship with the repo. Pick whichever matches your build pipeline.\n\n#### Option A — MSBuild target inside the `.csproj`\n\nPut packing inside the build itself. After every `dotnet build` the module is repackaged and dropped into the local Tsak `Libs/` for instant hot-reload. This is the pattern used by [redb.Route.Demo](https://github.com/redbase-app/redb-route/tree/main/redb.Route.Demo):\n\n```xml\n\u003cPropertyGroup\u003e\n  \u003c!-- Required: ensures NuGet PackageReference DLLs land in bin/ for class libraries --\u003e\n  \u003cCopyLocalLockFileAssemblies\u003etrue\u003c/CopyLocalLockFileAssemblies\u003e\n\n  \u003cTsakModuleName\u003eOrders\u003c/TsakModuleName\u003e\n  \u003cTsakLibsDir\u003e$(MSBuildThisFileDirectory)..\\redb.Tsak\\src\\redb.Tsak.Worker\\Libs\u003c/TsakLibsDir\u003e\n\u003c/PropertyGroup\u003e\n\n\u003cTarget Name=\"PackTpkg\" AfterTargets=\"Build\"\u003e\n  \u003cPropertyGroup\u003e\n    \u003c_TpkgStaging\u003e$(IntermediateOutputPath)tpkg\u003c/_TpkgStaging\u003e\n    \u003c_TpkgFile\u003e$(MSBuildThisFileDirectory)output\\$(TsakModuleName).tpkg\u003c/_TpkgFile\u003e\n  \u003c/PropertyGroup\u003e\n\n  \u003cRemoveDir Directories=\"$(_TpkgStaging)\" /\u003e\n  \u003cMakeDir Directories=\"$(_TpkgStaging);$(MSBuildThisFileDirectory)output\" /\u003e\n\n  \u003cCopy SourceFiles=\"$(MSBuildThisFileDirectory)manifest.json\"\n        DestinationFolder=\"$(_TpkgStaging)\" /\u003e\n  \u003cCopy SourceFiles=\"$(TargetPath)\"\n        DestinationFolder=\"$(_TpkgStaging)\" /\u003e\n  \u003cCopy SourceFiles=\"$(MSBuildThisFileDirectory)$(TsakModuleName).config.json\"\n        DestinationFolder=\"$(_TpkgStaging)\"\n        Condition=\"Exists('$(MSBuildThisFileDirectory)$(TsakModuleName).config.json')\" /\u003e\n\n  \u003cZipDirectory SourceDirectory=\"$(_TpkgStaging)\"\n                DestinationFile=\"$(_TpkgFile)\" Overwrite=\"true\" /\u003e\n\n  \u003cMakeDir Directories=\"$(TsakLibsDir)\" Condition=\"!Exists('$(TsakLibsDir)')\" /\u003e\n  \u003cCopy SourceFiles=\"$(_TpkgFile)\" DestinationFolder=\"$(TsakLibsDir)\" SkipUnchangedFiles=\"false\" /\u003e\n  \u003cTouch Files=\"$(TsakLibsDir)\\$(TsakModuleName).tpkg\" /\u003e\n\n  \u003cMessage Importance=\"high\" Text=\"Packed $(TsakModuleName) → $(_TpkgFile)\" /\u003e\n\u003c/Target\u003e\n```\n\nThe `\u003cCopyLocalLockFileAssemblies\u003etrue\u003c/CopyLocalLockFileAssemblies\u003e` line is critical for class-library SDK projects — without it, NuGet `PackageReference` assemblies are not copied to `bin/` and your `.tpkg` ships with missing transitive dependencies.\n\nThe final `\u003cTouch\u003e` updates `LastWriteTimeUtc` on the copy, which guarantees `HotReloadService` notices the change even when the new file's content hash matches a recently-loaded version.\n\n#### Option B — Standalone PowerShell script\n\nBest when you want to package multiple projects together as one `.tpkg` (e.g. a `Core` + `Api` pair sharing private dependencies), or when you want fine-grained control over which DLLs to include. Pattern from a real production module that bundles two entry-point projects plus three companions into one `tsum.tpkg`:\n\n```powershell\nparam(\n    [ValidateSet(\"Debug\",\"Release\")] [string]$Configuration = \"Release\"\n)\n$ErrorActionPreference = \"Stop\"\n\n$tfm     = \"net9.0\"\n$pkgName = \"tsum\"\n$staging = Join-Path $env:TEMP \"${pkgName}_tpkg_staging\"\n$tpkg    = Join-Path $PSScriptRoot \"output\\$pkgName.tpkg\"\n$libs    = \"..\\redb.Tsak\\src\\redb.Tsak.Worker\\Libs\"\n\n# 1. Build every project that contributes to the package\ndotnet build .\\tsum.Core\\tsum.Core.csproj -c $Configuration --nologo\ndotnet build .\\tsum.Api\\tsum.Api.csproj   -c $Configuration --nologo\n\n# 2. Stage manifest, entry points, companion DLLs, configs\nif (Test-Path $staging) { Remove-Item -Recurse -Force $staging }\nNew-Item -ItemType Directory -Path $staging | Out-Null\n\nCopy-Item .\\tsum.Core\\manifest.json $staging\nCopy-Item \".\\tsum.Core\\bin\\$Configuration\\$tfm\\tsum.Core.dll\" $staging   # entry point\nCopy-Item \".\\tsum.Api\\bin\\$Configuration\\$tfm\\tsum.Api.dll\"   $staging   # entry point\nCopy-Item \".\\tsum.Core\\bin\\$Configuration\\$tfm\\tsum.Models.dll\" $staging # companion\nCopy-Item \".\\tsum.Core\\bin\\$Configuration\\$tfm\\tsum.Domain.dll\" $staging # companion\nCopy-Item \".\\tsum.Core\\tsum.Core.config.json\" $staging -ErrorAction SilentlyContinue\nCopy-Item \".\\tsum.Api\\tsum.Api.config.json\"   $staging -ErrorAction SilentlyContinue\n\n# 3. Zip and deploy\nif (Test-Path $tpkg) { Remove-Item $tpkg -Force }\nAdd-Type -AssemblyName System.IO.Compression.FileSystem\n[IO.Compression.ZipFile]::CreateFromDirectory($staging, $tpkg)\n\nCopy-Item $tpkg $libs -Force\n(Get-Item (Join-Path $libs \"$pkgName.tpkg\")).LastWriteTime = Get-Date  # trigger hot-reload\nRemove-Item -Recurse -Force $staging\n```\n\nThe corresponding `manifest.json` declares both entry points so Tsak loads each one as a separately-managed module inside the shared package ALC:\n\n```json\n{\n  \"Name\": \"tsum\",\n  \"Version\": \"1.0.0\",\n  \"EntryPoints\": [ \"tsum.Core.dll\", \"tsum.Api.dll\" ],\n  \"Dependencies\": []\n}\n```\n\n#### Option C — Plain ZIP (any tool, any pipeline)\n\nA `.tpkg` is just a ZIP. Any tool that can write a ZIP can produce one:\n\n```powershell\nCompress-Archive -Path manifest.json,Orders.dll,Orders.Domain.dll,Orders.config.json `\n                 -DestinationPath Orders.zip -Force\nMove-Item Orders.zip Orders.tpkg -Force\n```\n\nOr `zip -j Orders.tpkg manifest.json *.dll *.config.json` on Linux.\n\n#### Excluding host-provided DLLs (recommended for shipping)\n\nTsak Worker already loads dozens of assemblies (`redb.Core`, `redb.Route.*`, every transport it ships, Quartz, Serilog, ASP.NET Core). Repacking those into every `.tpkg` wastes disk, bloats the archive, and occasionally causes version skew if your module references a different version than the host.\n\nThe recommended pattern is: build an exclude set from `redb.Tsak.Worker\\bin\\\u003ctfm\u003e\\*.dll` and skip any DLL in that set when staging — except for a small force-include list of packages you know are version-incompatible with the host's copy.\n\n```powershell\n$hostBin   = \"..\\redb.Tsak\\src\\redb.Tsak.Worker\\bin\\$Configuration\\$tfm\"\n$exclude   = New-Object 'System.Collections.Generic.HashSet[string]' (\n                 [System.StringComparer]::OrdinalIgnoreCase)\nGet-ChildItem \"$hostBin\\*.dll\" | ForEach-Object { [void]$exclude.Add($_.Name) }\n\n# Always include these even if the host ships an older copy (binary-incompatible)\n$forceInclude = @( 'Microsoft.IdentityModel.*.dll' )\n\nGet-ChildItem \".\\bin\\$Configuration\\$tfm\\*.dll\" | ForEach-Object {\n    $force = $false\n    foreach ($pat in $forceInclude) { if ($_.Name -like $pat) { $force = $true; break } }\n    if ($force -or -not $exclude.Contains($_.Name)) {\n        Copy-Item $_.FullName $staging\n    }\n}\n```\n\nThe module's per-package ALC then resolves a force-included DLL from the package itself before falling through to the host's older copy in the Default ALC.\n\n---\n\n## 5-layer configuration\n\nThe most powerful feature in Tsak's configuration model: deep merge across five layers, with predictable precedence.\n\n```\nLayer 1: Tsak:Contexts:default              ← base for all contexts (lowest priority)\nLayer 2: Tsak:Contexts:{name}               ← named context overrides\nLayer 3: Libs/{Module}/context.json         ← module infrastructure defaults\nLayer 4: Libs/{Module}/{Module}.config.json ← module business settings\nLayer 5: Tsak:Contexts:{name}:Override      ← DevOps final word (highest priority)\n```\n\nLater layers **deep-merge** over earlier layers. Nested objects merge recursively — they do not replace each other wholesale.\n\n### Worked example\n\n```json\n// appsettings.json\n{\n  \"Tsak\": {\n    \"Contexts\": {\n      \"default\": {\n        \"AutoStart\": true,\n        \"RabbitMQ\": { \"Host\": \"localhost\", \"Port\": 5672 }\n      },\n      \"orders\": {\n        \"Modules\": [\"Orders\"],\n        \"RabbitMQ\": { \"Host\": \"rabbit-orders.local\" },\n        \"Override\": {\n          \"RabbitMQ\": { \"Password\": \"from-secret-manager\" }\n        }\n      }\n    }\n  }\n}\n```\n\n```json\n// Libs/Orders/context.json\n{ \"RabbitMQ\": { \"Vhost\": \"/orders\", \"Username\": \"orders-svc\" } }\n```\n\n```json\n// Libs/Orders/Orders.config.json\n{ \"MaxRetries\": 10 }\n```\n\n**Effective configuration for the `orders` context:**\n\n```json\n{\n  \"AutoStart\": true,\n  \"MaxRetries\": 10,\n  \"RabbitMQ\": {\n    \"Host\": \"rabbit-orders.local\",       // from Layer 2 (overrode Layer 1 \"localhost\")\n    \"Port\": 5672,                         // from Layer 1 (preserved through merge)\n    \"Vhost\": \"/orders\",                   // from Layer 3 (module ships default)\n    \"Username\": \"orders-svc\",             // from Layer 3\n    \"Password\": \"from-secret-manager\"     // from Layer 5 (DevOps wins)\n  }\n}\n```\n\n### Hot-reload of config\n\nEdit `context.json` or `{Module}.config.json` while Tsak is running. The hot-reload scan re-merges all five layers and restarts the affected context with the new properties. **No worker restart required.**\n\n### Named vs anonymous contexts\n\n- **Named** — explicitly defined in `Tsak:Contexts:{name}` with a `Modules` array. Multiple modules share one `IRouteContext`.\n- **Anonymous** — for any module not assigned to a named context. One module = one context, named after the module.\n\n```json\n{\n  \"Tsak\": { \"Contexts\": {\n    \"api\": { \"Modules\": [\"Api.Orders\", \"Api.Catalog\"], \"AutoStart\": true }\n  }}\n}\n```\n\n`Api.Orders` and `Api.Catalog` share the `api` context (and its property bag). Any other module gets its own anonymous context.\n\nFull reference: [CONFIG_GUIDE.md](CONFIG_GUIDE.md).\n\n---\n\n## REST API\n\n32 endpoints organized into 12 controllers. Every endpoint speaks JSON. Auth is opt-in (`Tsak:Auth:Enabled`) — when enabled, all endpoints except `/api/system/health` require an API key.\n\n### Controllers\n\n| Group | Endpoints | Purpose |\n|---|---:|---|\n| `/api/auth` | 3 | Create / list / revoke API keys |\n| `/api/users` | 5 | User CRUD (Pro feature, EAV-backed) |\n| `/api/contexts` | 7 | List / get / start / stop / restart / reset-route-states / remove |\n| `/api/routes` | 6 | List / get / start / stop / force-stop / inflight per route |\n| `/api/modules` | 3 | List / get / remove loaded modules |\n| `/api/cluster` | 4 | Status / nodes / rebalance / remove-node |\n| `/api/system` | 4 | Health / metrics / metrics-history / info |\n| `/api/scheduler` | 9 | Status / jobs / running / start / standby / pause-job / resume-job / fire-job |\n| `/api/watchdog` | 2 | State / alerts |\n| `/api/lifecycle` | 1 | Recent lifecycle events (filtered) |\n| `/api/diagnostics` | 2 | Dump (cluster-wide) / route-level diagnostics |\n| `/api/logs` | 3 | Tail / list-files / download-file |\n\n### Sample calls\n\n```bash\n# Authenticate\nKEY=\"$(tsak auth keys create --name ci --roles admin --output json | jq -r .rawKey)\"\n\n# Health\ncurl -s http://localhost:9090/api/system/health | jq\n# {\n#   \"status\": \"Healthy\",\n#   \"checks\": [\n#     { \"name\": \"Contexts\", \"status\": \"Healthy\", \"data\": { \"running\": 4, \"failed\": 0 } },\n#     { \"name\": \"Memory\", \"status\": \"Healthy\", \"data\": { \"workingSetMB\": 184 } },\n#     { \"name\": \"Cluster\", \"status\": \"Healthy\", \"data\": { \"leader\": \"node-1\", \"epoch\": 7 } }\n#   ]\n# }\n\n# Start a context\ncurl -s -X POST -H \"Authorization: Bearer $KEY\" \\\n  http://localhost:9090/api/contexts/orders/start | jq\n\n# Live in-flight exchanges in a route (debugging stuck routes)\ncurl -s -H \"Authorization: Bearer $KEY\" \\\n  http://localhost:9090/api/contexts/orders/routes/route-1/inflight | jq\n\n# Force-stop a hung route\ncurl -s -X POST -H \"Authorization: Bearer $KEY\" \\\n  http://localhost:9090/api/contexts/orders/routes/route-1/force-stop\n\n# Recent lifecycle events\ncurl -s -H \"Authorization: Bearer $KEY\" \\\n  \"http://localhost:9090/api/lifecycle?contextName=orders\u0026limit=20\" | jq\n\n# Diagnostic dump (CPU profiles, GC info, thread states)\ncurl -s -H \"Authorization: Bearer $KEY\" \\\n  http://localhost:9090/api/diagnostics/dump \u003e tsak-dump.json\n```\n\n### Typed C# client\n\n```csharp\nservices.AddTsakClient(opts =\u003e\n{\n    opts.BaseUrl = \"http://tsak-prod:9090\";\n    opts.ApiKey = builder.Configuration[\"TsakKey\"];\n});\n\npublic class MyService(ITsakApiClient tsak)\n{\n    public async Task RestartAllAsync(CancellationToken ct)\n    {\n        var contexts = await tsak.ListContextsAsync(ct);\n        foreach (var c in contexts.Where(c =\u003e c.Status == \"Failed\"))\n            await tsak.RestartContextAsync(c.Name, ct);\n    }\n}\n```\n\n---\n\n## CLI\n\n`tsak` is a single binary that talks to any Tsak worker. It supports multiple connection profiles, JSON output for CI, and rich tabular output for humans.\n\n```bash\n# Profile management\ntsak login http://prod-1:9090 --key $PROD_KEY --profile prod\ntsak login http://stg-1:9090 --key $STG_KEY  --profile staging\ntsak profile use prod\ntsak profile list\n\n# Per-call override\ntsak --server http://other:9090 --key $OTHER_KEY  context list\n\n# Output format\ntsak context list                  # tabular (default)\ntsak context list --output json    # JSON\ntsak context list --output plain   # raw lines (grep-friendly)\n```\n\n### Command groups (30 commands)\n\n| Group | Commands |\n|---|---|\n| **profile** | `login`, `logout`, `use`, `list` |\n| **auth** | `auth keys list`, `auth keys create`, `auth keys revoke` |\n| **context** | `context list`, `context get`, `context start`, `context stop`, `context restart`, `context reset-routes`, `context delete` |\n| **route** | `route list`, `route get`, `route start`, `route stop`, `route force-stop`, `route inflight` |\n| **module** | `module list`, `module get`, `module deploy`, `module delete` |\n| **scheduler** | `scheduler status`, `scheduler jobs`, `scheduler running`, `scheduler start`, `scheduler standby`, `scheduler pause-job`, `scheduler resume-job`, `scheduler fire-job` |\n| **cluster** | `cluster overview`, `cluster nodes`, `cluster rebalance`, `cluster remove-node` |\n| **monitoring** | `health`, `metrics`, `metrics history`, `info`, `logs`, `logs files`, `logs download`, `lifecycle`, `diagnostics`, `route-diagnostics` |\n| **watchdog** | `watchdog status`, `watchdog alerts` |\n| **users** | `users list`, `users get`, `users create`, `users update`, `users delete` |\n\n---\n\n## Web dashboard\n\nA separate Blazor Server process (`redb.Tsak.Web`) — works in two modes:\n\n| Mode | Storage | Node discovery |\n|---|---|---|\n| **Standalone** | None | Static node list in `appsettings.json` |\n| **Cluster** | Required (Postgres / MSSQL) | Discovered dynamically from EAV cluster topology |\n\n### Pages\n\n| Page | Highlights |\n|---|---|\n| **Dashboard** | Cluster overview, status donut chart, sparkline metrics, node list grid (sortable, filterable, paginated). |\n| **Cluster** | 3-level topology tree (cluster → group → node), module assignment, per-node health, click-through to NodeDetail. |\n| **NodeDetail** | 5 tabs: Contexts · Scheduler · Modules · Monitoring · Logs. Live Chart.js graphs (CPU, GC, memory, threads), 10s auto-refresh. |\n| **Routes** | All routes across all contexts. Status, message count, error rate, click-through to route detail. |\n| **RouteView** | Per-route deep-dive: definition, current state, in-flight exchanges, recent diagnostics. |\n| **Endpoints** | Consumer / producer endpoints per route. |\n| **Watchdog** | Suspected and hung route alerts with manual stop/restart actions. |\n| **Logs** | Searchable ring-buffer log viewer with level filter and tail mode. |\n| **Auth** | API key management UI — create, revoke (with confirmation). |\n| **Login** | Credential-based dashboard access (cluster mode = EAV users; standalone = config). |\n\n### Custom design system\n\nThe dashboard uses a custom CSS design system — **no Bootstrap, no MUI, no Tailwind**. Built on CSS variables, supports dark/light theme, system fonts, inline SVG icons.\n\nReusable components: `TsakGrid\u003cT\u003e`, `TsakChart`, `TsakCard`, `TsakBadge`, `TsakIcon`, `TsakConfirmDialog`, `TsakToast`, `TsakPageHeader`, `TsakErrorBoundary`.\n\n---\n\n## Cluster mode\n\nEnable with `Tsak:Cluster:Enabled = true` and a Postgres/MSSQL connection string. Tsak handles the rest.\n\n### What you get\n\n- **Leader election** — distributed lock in redb EAV with TTL and epoch fencing. A new leader is elected automatically when the current one dies or loses its lock.\n- **Node registry** — each worker registers itself with periodic heartbeats. Dead nodes are evicted after `DeadNodeTimeoutSeconds`.\n- **Automatic context assignment** — the leader distributes contexts across live nodes (currently `round-robin`; weighted strategies on the roadmap). Contexts are reassigned automatically when nodes join or leave.\n- **Rolling hot-reload** — module updates roll across nodes in sequence, never updating multiple nodes concurrently.\n- **Cluster-wide diagnostics** — `tsak cluster overview` aggregates state from every node.\n\n### Topology in EAV\n\nStored as a polymorphic 3-level tree using `redb.Tree` (so it shows up nicely in any redb-aware tool):\n\n```\ncluster:default                     ← scheme: _tsak_clusters\n └── group:default:default          ← scheme: _tsak_groups\n      ├── node:default:worker-1     ← scheme: _tsak_nodes\n      ├── node:default:worker-2     ← scheme: _tsak_nodes\n      └── node:default:worker-3     ← scheme: _tsak_nodes\n```\n\nCluster operations (assignment, leader change, rebalance) mutate this tree atomically. Every operation is fenced by the leader's epoch token — a stale leader cannot corrupt state after losing election.\n\n### Cluster configuration\n\n```json\n{\n  \"Tsak\": {\n    \"Cluster\": {\n      \"Enabled\": true,\n      \"ClusterName\": \"default\",\n      \"GroupName\": \"default\",\n      \"NodeId\": \"\",\n      \"ApiEndpoint\": \"http://node-1.local:9090\",\n      \"HeartbeatIntervalSeconds\": 15,\n      \"DeadNodeTimeoutSeconds\": 60,\n      \"LeaderLockTtlSeconds\": 30,\n      \"Strategy\": \"round-robin\"\n    }\n  }\n}\n```\n\n### Pluggable cluster backends\n\nAll cluster coordination is hidden behind interfaces in `redb.Tsak.Core.Pro`:\n\n| Interface | Default implementation (redb EAV) | Drop-in alternative |\n|---|---|---|\n| `ILeaderElection` | `RedbLeaderElection` (epoch-fenced lock in EAV) | `KubernetesLeaderElection` (Lease API), `EtcdLeaderElection`, ZK |\n| `IDistributedLock` | `RedbDistributedLock` (TTL row in EAV) | `RedisDistributedLock`, `KubernetesLeaseLock` |\n| `INodeRegistry` | `RedbNodeRegistry` (heartbeat rows) | `KubernetesPodRegistry` (label selector), Consul |\n| `IClusterCoordinator` | `ClusterCoordinator` (background loop) | implementation owns the loop |\n| `IClusterBootstrap` | `RedbClusterBootstrap` | bootstrap from K8s ConfigMap |\n| `IAssignmentManager` | `RoundRobinAssignmentManager` | `WeightedAssignmentManager`, custom |\n\nReplace any of them with a single DI registration **before** `AddTsakCluster()`. Nothing else in the codebase changes — the `ClusterCoordinator` only talks to these interfaces.\n\n```csharp\nbuilder.Services\n    .AddSingleton\u003cILeaderElection, KubernetesLeaderElection\u003e()  // override default\n    .AddTsakCluster(builder.Configuration);                     // everything else stays\n```\n\nThis is the design path for native Kubernetes integration without ever touching redb EAV for coordination — handy when the cluster runs against an external operational database that you do not want to use as a synchronization primitive.\n\n---\n\n## Watchdog\n\n`WatchdogService` continuously inspects every route and classifies it into one of three states:\n\n```\n                                 (no progress for SuspectedThresholdMinutes)\n       ┌────────────┐                            │\n       │  Healthy   │ ─────────────────────────────────────► ┌─────────────┐\n       └────────────┘                                         │  Suspected  │\n              ▲                                               └─────────────┘\n              │                                                       │\n              │ (progress resumes)            (no progress for HungThresholdMinutes)\n              │                                                       │\n              │                                                       ▼\n              │                                                ┌─────────────┐\n              └─────────────────────────────────────────────── │    Hung     │\n                                                                └─────────────┘\n                                                                       │\n                                            (AutoRestartHungRoutes = true)\n                                                                       │\n                                                                       ▼\n                                                       Force-stop + restart route\n```\n\nConfigurable:\n\n```json\n{\n  \"Tsak\": {\n    \"Watchdog\": {\n      \"Enabled\": true,\n      \"CheckIntervalSeconds\": 10,\n      \"SuspectedThresholdMinutes\": 0.5,\n      \"HungThresholdMinutes\": 1.5,\n      \"AutoRestartHungRoutes\": false\n    }\n  }\n}\n```\n\nAlerts surface in `/api/watchdog/alerts`, in the CLI (`tsak watchdog alerts`), and in the Web dashboard's Watchdog page.\n\n---\n\n## Lifecycle \u0026 graceful shutdown\n\nTsak's startup and shutdown sequences are **deterministic** — `LifecycleHookOrdering` enforces the order so operators can rely on what is up before what depends on it, and what drains before what stops.\n\n### Startup order (Worker DI)\n\n1. **`QuartzSchemaInitializer`** — applies Quartz DDL (Postgres / MSSQL) idempotently via raw ADO.NET. No redb dependency at this stage to avoid bootstrap deadlocks.\n2. **`QuartzHostedService`** — starts the scheduler (`RAMJobStore` standalone, `AdoJobStore` cluster).\n3. **`MetricsCollectionService`** — periodic process / GC sampling into the circular buffer.\n4. **`TsakHostedService`** — main coordinator: shared assembly loader → module discovery → context start → cluster register.\n5. **`RouteWatchdogService`** — hung-exchange detector (Pro).\n\n### Shutdown sequence (`TsakHostedService.StopAsync`)\n\n1. Logs `\"Tsak graceful shutdown, stopping all contexts...\"`.\n2. Runs every `ITsakLifecycleHook.OnStoppingAsync()` **in reverse registration order** — last hook to start is first to stop. **Cluster deregistration is a hook**, so leadership is released *before* contexts begin draining.\n3. For each context, calls `ITsakContextManager.StopContextAsync(ctx, CancellationToken.None)` — deliberately **not** the host shutdown token. One slow context cannot cancel the drain of the others. Each context owns its own `context:graceful-stop-timeout` for in-flight exchanges.\n4. `base.StopAsync()` releases hosted-service resources.\n5. Quartz scheduler stops, Serilog sinks flush.\n\nStop / restart REST responses include `DrainTimeMs` and `InflightAfter` — operators see exactly how long the drain took and whether anything was force-killed. SIGTERM handling is delegated to the standard .NET `IHostApplicationLifetime`.\n\n### Lifecycle hooks\n\n```csharp\npublic class MyHook : ITsakLifecycleHook\n{\n    public Task OnStartingAsync(CancellationToken ct)  =\u003e /* before module discovery */;\n    public Task OnStartedAsync (CancellationToken ct)  =\u003e /* after all contexts running */;\n    public Task OnStoppingAsync(CancellationToken ct)  =\u003e /* before context drain (reverse) */;\n    public Task OnStoppedAsync (CancellationToken ct)  =\u003e /* after all contexts stopped */;\n}\n\nbuilder.Services.AddSingleton\u003cITsakLifecycleHook, MyHook\u003e();\n```\n\nEvery start / stop / error event is also persisted by `LifecycleAuditService` and queryable through `/api/lifecycle`.\n\n---\n\n## Storage modes\n\n| Mode | API keys | Modules | Cluster | State | Use case |\n|---|---|---|---|---|---|\n| **InMemory** | `ConfigApiKeyStore` (read-only, from `appsettings`) | In-process registry | Not supported | Lost on restart | Dev, CI, embedded scenarios |\n| **Redb (Postgres)** | `RedbApiKeyStore` (EAV, runtime CRUD) | Persistent | Supported | Survives restart | Single-node production, lightweight clusters |\n| **Redb (MSSQL)** | `RedbApiKeyStore` (EAV, runtime CRUD) | Persistent | Supported | Survives restart | Single-node production, MSSQL shops |\n\nSwitch modes with one config setting:\n\n```json\n{ \"Tsak\": { \"Storage\": { \"Type\": \"Redb\" }, \"Redb\": { \"Provider\": \"postgres\" } } }\n```\n\nWhen `UsePro = true`, [redb.Core.Pro](https://github.com/redbase-app/redb) is enabled — gives you EAV change tracking (faster writes), distributed locking primitives, and the cluster topology features.\n\n---\n\n## Security\n\n| Layer | Mechanism |\n|---|---|\n| **Wire** | API Key in `Authorization: Bearer` or `X-Api-Key` header |\n| **Storage** | SHA-256 hash, raw key never persisted |\n| **Comparison** | `CryptographicOperations.FixedTimeEquals` (timing-attack safe) |\n| **Authorization** | Per-endpoint role checks (`admin`, `reader`, custom) |\n| **Lifecycle** | Optional expiry per key, runtime revocation, immediate cache invalidation |\n| **User binding** | Optional `UserId` link — when the user is disabled or deleted, the key dies (via `IUserProvider`) |\n| **Cache** | 5-minute TTL key lookup cache, invalidated on revoke |\n| **Cluster trust** | Inter-node calls use the same API key auth — no implicit trust between nodes |\n| **Protected resources** | The `_system` context cannot be stopped or removed by any caller, including admins |\n\nFull policy: [SECURITY.md](SECURITY.md). Production secrets handling: [DEPLOYMENT_SECRETS.md](DEPLOYMENT_SECRETS.md).\n\n---\n\n## Observability\n\n### Metrics\n\n- `MetricsService` — circular buffer with CPU, working set, managed memory, threads, GC pressure. Default window: **12 hours × 10s sample = 4320 points**.\n- `ContextMetricsCollector` — per-context aggregation: messages/sec, error rate, in-flight count.\n- Per-route metrics from `redb.Route`'s `.Metered()` step.\n- Cluster-wide periodic state report: `ClusterReportIntervalSeconds`.\n\n### OpenTelemetry \u0026 Prometheus\n\nFirst-class OTel integration via the standard NuGet packages:\n\n- `OpenTelemetry.Extensions.Hosting`\n- `OpenTelemetry.Exporter.Prometheus.HttpListener`\n- `OpenTelemetry.Instrumentation.Process`\n- `OpenTelemetry.Instrumentation.Runtime`\n\nWhen `Tsak:Metrics:Prometheus:Enabled = true`, Tsak calls\n`AddOpenTelemetry().WithMetrics(b =\u003e b.AddPrometheusHttpListener(...))` and exposes a Prometheus-format endpoint on **`http://*:9464/metrics`** (port configurable via `Tsak:Metrics:Prometheus:Port`).\n\nThe OTel pipeline registers the redb.Route `ActivitySource` (`RouteActivitySource.SourceName`) so any tracing emitted by route processors is collected automatically — point an OTLP collector / Jaeger / Tempo at the activity source, or scrape `:9464/metrics` from Prometheus.\n\n```jsonc\n\"Tsak\": {\n  \"Metrics\": {\n    \"Prometheus\": { \"Enabled\": true, \"Port\": 9464 }\n  }\n}\n```\n\n### Health probes — Kubernetes-ready\n\nDistinct probes for the three K8s lifecycle phases. All three are **auth-exempt** by default (`Tsak:Api:AuthExempt`):\n\n| Endpoint | Probe type | Returns 200 when |\n|---|---|---|\n| `GET /api/system/health/startup` | startup | process is up |\n| `GET /api/system/health/live` | liveness | process is not deadlocked |\n| `GET /api/system/health/ready` | **readiness** | contexts are running **and** healthy (stricter than liveness) |\n\n`HealthCheckService` aggregates probes (worst status wins: Unhealthy \u003e Degraded \u003e Healthy) and never throws — exceptions inside a probe become Unhealthy, never a 500. Modules can contribute custom probes by implementing `IModuleHealthContributor`. Pro ships `ClusterHealthContributor` reporting leader / member health.\n\n### Logs\n\n- `LogRingBuffer` — Serilog in-memory sink, default 2000 entries, queryable via REST and Web UI.\n- File logs via Serilog (rolling files, configurable path).\n- `MemoryUsageEnricher` — every log event carries `{MemoryUsage}` for cheap memory pressure correlation.\n\n### Tracing\n\n- OpenTelemetry traces via `redb.Route`'s `.Traced()` step — per-route, per-step spans.\n- Standard OTLP exporter (configurable in `appsettings.json`).\n\n### Diagnostics\n\n- Per-route diagnostic dumps via `/api/contexts/{ctx}/routes/{id}/diagnostics`.\n- Cluster-wide dumps via `/api/diagnostics/dump`.\n- Lifecycle events feed (route start/stop/restart, hot-swap, cluster reassignment, watchdog alerts).\n- In-flight exchange tracking — see exactly which messages are sitting where right now.\n\n---\n\n## Quartz scheduler\n\nEvery Tsak context gets an `IScheduler` injected for free. Modules use it via `redb.Route.Quartz`:\n\n```csharp\nFrom(\"cron:0 */5 * * * ?\")           // every 5 minutes\n    .Setbody(() =\u003e DateTime.UtcNow)\n    .To(\"rabbitmq://heartbeats\");\n```\n\n| Mode | Storage | Cluster-safe | Use |\n|---|---|---|---|\n| `RAMJobStore` | In-memory | No | Standalone, dev |\n| `AdoJobStore` | Postgres / MSSQL `QRTZ_*` tables | Yes | Production |\n\nSchema is auto-created on first start by `QuartzSchemaInitializer` — embedded SQL scripts for both Postgres and MSSQL, idempotent, runs **before** Quartz initializes its own connection pool. No DBA action required.\n\nREST endpoints under `/api/scheduler` cover: status, listing scheduled jobs, listing currently-running jobs, start/standby, pause/resume/fire-now per job key.\n\n---\n\n## Three deployment recipes\n\n### Recipe A — Standalone, single binary\n\nFor development, demos, embedded use.\n\n```json\n{\n  \"Tsak\": {\n    \"Storage\": { \"Type\": \"InMemory\" },\n    \"Cluster\": { \"Enabled\": false },\n    \"Auth\":    { \"Enabled\": false }\n  }\n}\n```\n\nJust `dotnet run`. No DB, no auth, no cluster. Fastest path to running a Tsak module locally.\n\n### Recipe B — Single node + redb (Postgres)\n\nFor production single-node deployments where state must survive restarts.\n\n```json\n{\n  \"ConnectionStrings\": { \"Postgres\": \"Host=db;Database=redb_tsak;Username=tsak;Password=$$\" },\n  \"Tsak\": {\n    \"Storage\": { \"Type\": \"Redb\" },\n    \"Redb\":    { \"Provider\": \"postgres\", \"UsePro\": true, \"License\": \"$$\" },\n    \"Cluster\": { \"Enabled\": false },\n    \"Auth\":    { \"Enabled\": true, \"Secret\": \"$$\" }\n  }\n}\n```\n\nAPI keys persist. Quartz jobs persist. Lifecycle events persist. Module assignments persist. One node, durable state.\n\n### Recipe C — Cluster\n\nFor HA and horizontal scaling. Same `appsettings` on every node, only `NodeId` and `ApiEndpoint` differ:\n\n```json\n{\n  \"ConnectionStrings\": { \"Postgres\": \"Host=db.cluster;Database=redb_tsak;Username=tsak;Password=$$\" },\n  \"Tsak\": {\n    \"Storage\": { \"Type\": \"Redb\" },\n    \"Redb\":    { \"Provider\": \"postgres\", \"UsePro\": true },\n    \"Cluster\": {\n      \"Enabled\": true,\n      \"ClusterName\": \"production\",\n      \"GroupName\": \"default\",\n      \"ApiEndpoint\": \"http://node-1.local:9090\",\n      \"Strategy\": \"round-robin\"\n    },\n    \"HotReload\": { \"RollingUpdate\": true },\n    \"Auth\":     { \"Enabled\": true, \"Secret\": \"$$\" }\n  }\n}\n```\n\nStart three workers. They'll discover each other through the shared EAV store, elect a leader, distribute your modules, and roll updates without downtime.\n\n---\n\n## Docker\n\nPre-built images are published to GitHub Container Registry for every release — no build step required. See [Quick start → Option A](#option-a--docker-fastest) for the basic flow. This section covers production deployment.\n\n### Image variants\n\n| Image | Best for | Default ports |\n|---|---|---|\n| `ghcr.io/redbase-app/redb-tsak-worker` | Headless workers (k8s `Deployment`/`StatefulSet`), one process per pod | `9090` (REST) |\n| `ghcr.io/redbase-app/redb-tsak-web`    | Separate management UI pod talking to a worker cluster | `8080` |\n| `ghcr.io/redbase-app/redb-tsak-stack`  | Single-host install (Worker + Web in one container, like `rabbitmq:management`) | `9090`, `8080` |\n\nAvailable tags:\n- `latest`, `\u003cversion\u003e` — net9 build (default).\n- `\u003cversion\u003e-net8`, `\u003cversion\u003e-net9`, `\u003cversion\u003e-net10` — Worker only; pick a TFM that matches your shared connector ABI.\n- `\u003cversion\u003e-net9` — Web and Stack are net9 only.\n\n### Production run (Worker + external Postgres)\n\n```bash\ndocker run -d --name tsak \\\n  -p 9090:9090 -p 9464:9464 \\\n  -v /opt/tsak/Libs:/app/Libs \\\n  -e ConnectionStrings__Postgres=\"$PG_CONN\" \\\n  -e Tsak__Auth__Secret=\"$TSAK_SECRET\" \\\n  -e Tsak__Metrics__Prometheus__Enabled=true \\\n  ghcr.io/redbase-app/redb-tsak-worker:2.0.2\n```\n\nMount `Libs/` from the host (or a shared volume) so module updates can be deployed without rebuilding the image. `appsettings.json` overrides flow through environment variables in the standard ASP.NET pattern (`__` for nesting).\n\n### docker compose templates\n\nReady-to-use compose files are shipped in [`publish/docker/`](publish/docker/) — copy and edit, no monorepo paths:\n\n| File | What it stands up |\n|---|---|\n| `compose.worker.yml` | Worker only |\n| `compose.web.yml`    | Web only (talks to an existing worker) |\n| `compose.stack.yml`  | Worker + Web in one container |\n| `compose.full.yml`   | Stack + PostgreSQL (durable EAV, single-node) |\n\nEach has a matching `compose.*.env.example` — copy to `.env` and fill in.\n\n### Verifying images\n\n```bash\ncosign verify --key cosign.pub ghcr.io/redbase-app/redb-tsak-worker:2.0.2\n```\n\nPublic key: [`publish/keys/cosign.pub`](publish/keys/cosign.pub) in the repo, or downloadable from any [release](https://github.com/redbase-app/redb-tsak/releases).\n\n### Building images yourself\n\nIf you need a custom build (e.g. proprietary connectors baked in), see [`publish/HOW_TO_PUBLISH.md`](publish/HOW_TO_PUBLISH.md) for the full pipeline (`pwsh publish/build.ps1 -All`).\n\n---\n\n## Kubernetes deployment\n\nTsak is built from the ground up for container deployment — distinct K8s probes, OTel/Prometheus exporter, env-var configuration, and pluggable cluster backends are all in the box.\n\n### Probes\n\nMap the three health endpoints to the matching K8s probe types:\n\n```yaml\nstartupProbe:\n  httpGet: { path: /api/system/health/startup, port: 9090 }\n  failureThreshold: 30\n  periodSeconds: 4           # up to 120s to boot\n\nlivenessProbe:\n  httpGet: { path: /api/system/health/live, port: 9090 }\n  periodSeconds: 10\n  failureThreshold: 3        # 30s before pod restart\n\nreadinessProbe:\n  httpGet: { path: /api/system/health/ready, port: 9090 }\n  periodSeconds: 5\n  failureThreshold: 2        # 10s before removal from load balancer\n```\n\n**Liveness** intentionally does *not* check module health — that avoids restart loops during rolling updates. **Readiness** is stricter: any context in a non-running state → pod is removed from service endpoints (no restart), allowing the cluster to rebalance via `IAssignmentManager`.\n\n### Graceful termination\n\nKubernetes sends SIGTERM, then SIGKILL after `terminationGracePeriodSeconds`. Set `Tsak:Shutdown:TimeoutSeconds` to **`terminationGracePeriodSeconds − 5`** so cluster deregistration has its buffer. The shutdown sequence then becomes:\n\n```\nSIGTERM → cluster deregister (lifecycle hook, reverse-order) →\ncontext drain (isolated CT, up to TimeoutSeconds each) →\nQuartz shutdown → log flush → SIGKILL never needed\n```\n\n```yaml\nspec:\n  terminationGracePeriodSeconds: 60\n  containers:\n    - name: tsak\n      env:\n        - name: Tsak__Shutdown__TimeoutSeconds\n          value: \"55\"\n```\n\n### Pod identity in cluster mode\n\nStableNode IDs across pod restarts matter for assignment continuity. Inject pod metadata via the downward API and bind `NodeId` to `metadata.name`:\n\n```yaml\nenv:\n  - name: POD_NAME\n    valueFrom: { fieldRef: { fieldPath: metadata.name } }\n  - name: POD_NAMESPACE\n    valueFrom: { fieldRef: { fieldPath: metadata.namespace } }\n  - name: POD_IP\n    valueFrom: { fieldRef: { fieldPath: status.podIP } }\n  - name: Tsak__Cluster__NodeId\n    valueFrom: { fieldRef: { fieldPath: metadata.name } }\n  - name: Tsak__Cluster__ApiEndpoint\n    value: http://$(POD_IP):9090\n```\n\nUse a **StatefulSet** (or a Deployment + headless service) for predictable pod names. Tsak treats `Tsak:Cluster:NodeId` as the cluster identity — mapping it to `metadata.name` keeps assignment stable across restarts.\n\n### Prometheus scraping\n\n```yaml\nmetadata:\n  annotations:\n    prometheus.io/scrape: \"true\"\n    prometheus.io/port:   \"9464\"\n    prometheus.io/path:   \"/metrics\"\n```\n\nSet `Tsak:Metrics:Prometheus:Enabled = true` to activate the OTel Prometheus exporter on port 9464.\n\n### Native cluster integration\n\nAll coordination interfaces (`ILeaderElection`, `IDistributedLock`, `INodeRegistry`, `IClusterCoordinator`, `IClusterBootstrap`, `IAssignmentManager`) are pluggable — see [Pluggable cluster backends](#pluggable-cluster-backends). A Kubernetes-native Lease implementation (`KubernetesLeaderElection`) can be dropped in without touching redb EAV for coordination, leaving redb only for module / lifecycle / API key state.\n\n---\n\n## Testing\n\n| Suite | Count | What's covered |\n|---|---:|---|\n| `redb.Tsak.Tests` | **287** | Module loading, context lifecycle, config merge, security, controllers, monitoring, scheduler, cluster (leader election, heartbeat, rebalance, epoch fencing), hot-reload (ALC isolation, rolling update, rollback, removal debounce), in-memory and Redb stores, host startup, full API integration |\n| `redb.Tsak.CLI.Tests` | **64** | All 30 CLI commands — output format (table/JSON/plain), error handling, auth failure, profile resolution |\n| **Total** | **351** | All passing on net8.0, net9.0, net10.0 |\n\n```bash\ndotnet test redb.Tsak/tests/redb.Tsak.Tests\ndotnet test redb.Tsak/tests/redb.Tsak.CLI.Tests\n```\n\n---\n\n## Implementation status\n\nAll 9 phases are complete and merged. See [STATUS.md](STATUS.md) for the per-phase breakdown.\n\n| # | Phase | Status | Tests |\n|---|---|---|---|\n| 0 | Infrastructure | Done | 1 |\n| 1 | Controller Dispatcher (transport-agnostic) | Done | 20 |\n| 2 | Container Core (modules, contexts, coordinator) | Done | 32 |\n| 3 | Cluster (leader election, registry, assignment) | Done | 9 |\n| 4 | Hot Reload (collectible ALC, rolling update, rollback) | Done | 25 |\n| 5 | Monitoring (metrics, health, logs, watchdog) | Done | 50 |\n| 6 | REST API \u0026 Auth (12 controllers, EAV key store) | Done | 42 |\n| 7 | Quartz Scheduler (DI, schema initializer, controller) | Done | 30 |\n| 8A | CLI (30 commands, profiles, JSON) | Done | 64 |\n| 8B | Web UI (Blazor Server, 10 pages, design system) | Done | — |\n\n**Total: 351 tests passing.**\n\n---\n\n## Frequently asked questions\n\n**Is Tsak a routing engine?**\nNo. Routing is [redb.Route](https://github.com/redbase-app/redb-route). Tsak is the runtime container that hosts route contexts, manages their lifecycle, and exposes a management surface.\n\n**Can Tsak load any .NET DLL?**\nTsak loads class libraries that follow its module convention: a static `InitRoute.main(IRouteContext)` method, or one or more public `RouteBuilder` subclasses. Random DLLs are ignored.\n\n**Is module isolation a sandbox?**\nNo. It is a dependency isolation boundary (separate `AssemblyLoadContext` per module). A malicious module runs with the same OS privileges as the Tsak process. Restrict write access to `Libs/` accordingly. See [SECURITY.md](SECURITY.md).\n\n**Why is `Collectible: false` the default for hot-reload?**\nBecause `Reflection.Emit`-based code (XmlSerializer, source-gen serializers, compiled regex, MEF) crashes inside collectible `AssemblyLoadContext`s. The .NET runtime forbids non-collectible dynamic assemblies (generated by Emit) from referencing types in a collectible ALC. Default is the safe choice; opt in only when you know your modules are Emit-free.\n\n**What happens to in-flight messages during hot-swap?**\nThe old context drains (existing exchanges complete). The new context starts in parallel. Once the new context is healthy and `StartupTimeoutSeconds` elapses, the old context is retired. Zero message loss for transports that support graceful shutdown.\n\n**Can I write my own assignment strategy?**\nYes — `IAssignmentManager` is the extensibility point. The `round-robin` strategy is the only one shipped today; weighted strategies are on the roadmap.\n\n**Does Tsak support multi-region clusters?**\nOut of the box, no. The cluster coordination assumes low-latency access to the shared EAV database. For multi-region, run one cluster per region and federate above the Tsak layer (e.g. via `redb.Route.RabbitMQ` shovels).\n\n**Why a custom design system instead of Bootstrap or MUI?**\nThe dashboard is small and focused. Custom CSS keeps the bundle tiny, eliminates a major source of UI churn (vendor breaking changes), and gives full control over theming. CSS variables enable dark/light themes with no JS.\n\n**Is there an OpenAPI spec?**\nNot yet. The 12 controllers are documented in [STATUS.md](STATUS.md) and exposed via the typed `ITsakApiClient`. OpenAPI / Swagger generation is on the roadmap.\n\n---\n\n## Roadmap\n\n- OpenAPI / Swagger generation for the REST API\n- Weighted cluster assignment strategies (CPU + memory composite)\n- Batch operations (`POST /api/contexts/batch/start`)\n- Webhook subscriptions for lifecycle events (push instead of poll)\n- Live config editor in the dashboard\n- Multi-region federation primitives\n\nSee [docs/](docs/) for design notes on each.\n\n---\n\n## Part of\n\n- [redb.Route](https://github.com/redbase-app/redb-route) — ESB \u0026 EIP framework for .NET (the routing engine Tsak hosts)\n- [redb.Core / redb.Core.Pro](https://github.com/redbase-app/redb) — EAV storage backend (the persistence layer Tsak uses)\n- [RedBase](https://github.com/redbase-app) — full ecosystem\n\n---\n\n## License\n\nApache License 2.0 — see [LICENSE](LICENSE) and [NOTICE](NOTICE).\n\nCopyright © 2024–2026 RedBase.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredbase-app%2Fredb-tsak","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredbase-app%2Fredb-tsak","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredbase-app%2Fredb-tsak/lists"}