https://github.com/czhao-dev/control-plane
A declarative infrastructure control plane: Deployment scheduling, Node registration and heartbeating, desired-state reconciliation, failure recovery, label-based Service discovery, and health-aware proxy routing — with Prometheus/Grafana observability.
https://github.com/czhao-dev/control-plane
container-orchestration control-plane distributed-systems docker docker-compose fault-tolerance golang grafana health-check infrastructure load-balancer microservices node-agent orchestration prometheus reconciler rest-api reverse-proxy scheduler service-discovery
Last synced: about 22 hours ago
JSON representation
A declarative infrastructure control plane: Deployment scheduling, Node registration and heartbeating, desired-state reconciliation, failure recovery, label-based Service discovery, and health-aware proxy routing — with Prometheus/Grafana observability.
- Host: GitHub
- URL: https://github.com/czhao-dev/control-plane
- Owner: czhao-dev
- License: mit
- Created: 2026-06-23T05:45:49.000Z (19 days ago)
- Default Branch: main
- Last Pushed: 2026-07-08T13:09:44.000Z (3 days ago)
- Last Synced: 2026-07-08T14:17:22.718Z (3 days ago)
- Topics: container-orchestration, control-plane, distributed-systems, docker, docker-compose, fault-tolerance, golang, grafana, health-check, infrastructure, load-balancer, microservices, node-agent, orchestration, prometheus, reconciler, rest-api, reverse-proxy, scheduler, service-discovery
- Language: Go
- Size: 12.6 MB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Distributed Container Orchestration Platform
A declarative infrastructure control plane that accepts Deployment specs, schedules Pods onto registered Nodes, tracks Node health via heartbeats, reconciles desired vs. actual state, recovers from Node failures, and drives a health-aware reverse proxy — with Prometheus/Grafana observability across the full stack.
```
infractl / curl
│
▼
┌─────────────────┐
│ control-plane │ deployments, pods, nodes, services
│ :7070 │ scheduler + controller loops
└────────┬────────┘
┌───────────────┼───────────────┐
▼ ▼ ▼
worker-1 worker-2 worker-3 (register, heartbeat,
(:9100 metrics) (:9101 metrics) (:9102 metrics) poll, execute)
▲ ▲ ▲
└───────────────┼───────────────┘
┌─────────────────┐
client ───▶ │ proxy │ fetches healthy nodes from the
│ :8080 │ control plane, routes/fails over
└─────────────────┘
│
┌────────┴────────┐
▼ ▼
Prometheus :9090 Grafana :3000
```
Four binaries make up the system: `control-plane` (the API server plus its scheduler and controller loops), `worker` (a node agent that registers, heartbeats, and executes pods), `proxy` (a health-aware reverse proxy that discovers backends dynamically from the control plane), and `infractl` (the CLI client). See [`docs/`](docs/) for the design of each component.
## Repository Layout
```text
.
├── go.mod # single Go module for the whole system
├── docker-compose.yml # control-plane + 3 workers + proxy + Prometheus/Grafana
├── Dockerfile # multi-stage build producing all 4 binaries
├── cmd/
│ ├── control-plane/ # API server entrypoint
│ ├── worker/ # node agent entrypoint
│ ├── proxy/ # reverse proxy entrypoint
│ └── infractl/ # CLI entrypoint
├── pkg/
│ ├── apis/ # domain types: Deployment, Pod, Node, Service
│ ├── api/ # REST server: routes, handlers, middleware
│ ├── scheduler/ # assigns PENDING pods to HEALTHY nodes
│ ├── controller/ # reconciliation + node failure detection
│ ├── agent/ # node agent: register/heartbeat/poll/execute
│ │ └── metrics/
│ ├── proxy/ # reverse proxy: balancing, health checks, discovery
│ │ └── balancer/ backend/ health/ admin/ config/ metrics/
│ ├── registry/ # state store: BoltDB or in-memory
│ ├── metrics/ # control-plane Prometheus metrics
│ ├── logging/ # shared structured logger
│ └── config/ # control-plane env config
├── deployments/ # Prometheus config, proxy config, Grafana dashboard/provisioning
├── docs/ # architecture, scheduler, controller, node-model, proxy design docs
├── examples/ # example Deployment/Service manifests
├── scripts/ # demo and benchmark scripts
└── tests/integration/ # black-box integration suite
```
## Quickstart
```bash
./scripts/run-local-cluster.sh
```
Builds and starts the control plane, 3 node agents, the proxy, Prometheus, and Grafana.
- **Control plane API**: http://localhost:7070
- **Proxy**: http://localhost:8080 (backend status: `/admin/backends`)
- **Prometheus**: http://localhost:9090
- **Grafana** (admin/admin): http://localhost:3000
Then walk through the demo scripts in [`scripts/`](scripts/):
```bash
./scripts/submit-demo-jobs.sh # submit a 20-pod deployment, watch it schedule across nodes
./scripts/demo-worker-failure.sh # kill a node mid-pod, watch the controller detect and reschedule
./scripts/demo-proxy-failover.sh # kill a node, watch the proxy route around it and recover
./scripts/benchmark-scheduler.sh # throughput: deployments submitted to all pods scheduled
```
Or drive it directly with `infractl` (`go run ./cmd/infractl ...`; `INFRACTL_SERVER` defaults to `http://localhost:7070`):
```bash
infractl deployment submit examples/batch-job.yaml
infractl deployment status
infractl node list
infractl cluster status
```
**Known limitation:** a node that crashes and never comes back stays `UNHEALTHY` in the control plane's store forever — the heartbeat timeout only ever flips `HEALTHY → UNHEALTHY`; there's no automatic garbage-collection of permanently-dead nodes. Its pods are correctly rescheduled onto the remaining healthy nodes, so this is a bookkeeping gap, not a scheduling one. A `drain`-then-timeout path does fully remove a node (see [`docs/controller.md`](docs/controller.md)).
## Component overview
| Component | Package/binary |
|---|---|
| API Server | `cmd/control-plane` + `pkg/api` |
| Persistent state store | `pkg/registry` (BoltDB or in-memory) |
| Scheduler | `pkg/scheduler` |
| Controller | `pkg/controller` |
| Node agent | `cmd/worker` + `pkg/agent` |
| Proxy | `cmd/proxy` + `pkg/proxy` |
| CLI client | `cmd/infractl` |
## Domain model
A `Deployment` is a declarative template (image or command, replicas, retry policy, resources, namespace, labels). The controller creates `Pod`s from it — one execution unit per replica — and reconciles their count against `Replicas` every tick. Pods are scheduled onto `Node`s (registered, heartbeating agent processes) by the scheduler, which picks the least-loaded healthy node with enough capacity. A `Service` describes path-based routing and a label `Selector` that resolves to a set of `Node` backends, which the proxy discovers dynamically. All four resource types share the same pattern: a `Status` field, an `allowedTransitions` map, and a pure `Transition` validator.
## Labels and namespaces
`Deployment`, `Pod`, and `Service` are namespaced (`Namespace` field, defaults to `default`); `Node`s are cluster-scoped (no `Namespace` field — a node can run pods from any namespace). Pods inherit their Deployment's `Namespace` and `Labels` at creation time. List endpoints accept label-selector query params (`?label=key=value`), matched via exact-match `Service.Selector`-style matching.
## Persistent state
`CTRLPLANE_DB_PATH` toggles the state store: unset uses an in-memory store (data lost on restart), set to a file path uses BoltDB (persists across restarts, single-node, no replication — see [`docs/architecture.md`](docs/architecture.md) for the tradeoff).
## Local development
```bash
go build ./...
go vet ./...
go test ./... -race
```
All four binaries live in one Go module (`go.mod` at the repo root), so these commands work directly — no per-component build steps needed.
## Documented simplifications
These are intentional design decisions for the scope of this project, not bugs:
| Production orchestrators | This project |
|---|---|
| Distributed consensus store (e.g. etcd with Raft, gRPC) | BoltDB: embedded single-file ACID store, no network |
| Watch/informer event streams from the state store | Node agent polls on a timer (500 ms); controller ticks on a timer (2 s) |
| Services select Pods by label via overlay networking | Services select Nodes by label (Pods have no network address; nodes are the routable entities) |
| Full container runtime (containerd, runc, CNI networking) | Docker SDK — pods run as Docker containers when `image:` is set, or OS subprocesses otherwise |
| RBAC, admission webhooks, CRDs | Not implemented |
| Metrics-server-backed autoscaling on real CPU/memory utilization | `pkg/autoscaler` — cluster-wide PENDING-pod-count proxy signal instead (see [docs/autoscaling.md](docs/autoscaling.md)) |
| Dedicated load-balancing layer as a separate, mature project | `pkg/proxy` — a from-scratch reverse proxy, not a replacement for NGINX/HAProxy/Envoy/Traefik |
## Docs
- [`docs/architecture.md`](docs/architecture.md) — component mapping, topology, responsibilities
- [`docs/scheduler.md`](docs/scheduler.md) — scheduling policy and capacity bookkeeping
- [`docs/controller.md`](docs/controller.md) — reconciliation passes and failure recovery
- [`docs/autoscaling.md`](docs/autoscaling.md) — replica autoscaling policy, signal, and observability
- [`docs/node-model.md`](docs/node-model.md) — node agent lifecycle, execution, graceful shutdown
- [`docs/proxy.md`](docs/proxy.md) — proxy design, load-balancing strategies, health checking
- [`docs/security-hardening.md`](docs/security-hardening.md) — before/after hardening checklist with real Trivy findings
## References
- [Go Workspaces](https://go.dev/doc/workspaces) — a mechanism this project previously used to tie multiple modules together; the system is now a single module
- [Prometheus Go client](https://github.com/prometheus/client_golang) — instrumentation library used by the control plane and node agent
- [Prometheus](https://prometheus.io/docs/introduction/overview/) — metrics collection and alerting
- [Grafana](https://grafana.com/docs/grafana/latest/) — metrics visualization and dashboards
- [Docker Compose](https://docs.docker.com/compose/) — local multi-service orchestration for the demo stack
- Controller / reconcile-loop pattern — the desired-vs-actual-state design that inspired `pkg/controller`
- [go.etcd.io/bbolt](https://pkg.go.dev/go.etcd.io/bbolt) — embedded single-file ACID key-value store used as the persistent state backend
- [gopkg.in/yaml.v3](https://pkg.go.dev/gopkg.in/yaml.v3) — YAML parsing used for declarative deployment spec files and proxy config
- [testify](https://github.com/stretchr/testify) — assertion library used across the test suite
- [Docker Go SDK](https://pkg.go.dev/github.com/docker/docker/client) — official Docker client used by the node agent to pull images and manage container lifecycle when `image:` is set in a Deployment spec
## License
This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.