{"id":48439891,"url":"https://github.com/bearbinary/omni-infra-provider-truenas","last_synced_at":"2026-04-23T02:02:55.664Z","repository":{"id":349552408,"uuid":"1201509394","full_name":"bearbinary/omni-infra-provider-truenas","owner":"bearbinary","description":"TrueNAS SCALE infrastructure provider for Sidero Omni — automatically provisions Talos Linux VMs via JSON-RPC 2.0","archived":false,"fork":false,"pushed_at":"2026-04-22T18:52:44.000Z","size":13132,"stargazers_count":0,"open_issues_count":16,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-22T20:11:09.195Z","etag":null,"topics":["bare-metal","gitops","home-server","homelab","iac","infrastructure-as-code","json-rpc","k8s","kubernetes","omni","proxmox-alternative","self-hosted","sidero","talos","talos-linux","truenas","truenas-scale","virtualization","vm-provisioning","zfs"],"latest_commit_sha":null,"homepage":"https://bearbinary.github.io/omni-infra-provider-truenas/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bearbinary.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"bearbinary"}},"created_at":"2026-04-04T19:18:55.000Z","updated_at":"2026-04-22T18:51:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"55f200e3-8493-4d20-9e85-c610d1983205","html_url":"https://github.com/bearbinary/omni-infra-provider-truenas","commit_stats":null,"previous_names":["bearbinary/omni-infra-provider-truenas"],"tags_count":42,"template":false,"template_full_name":null,"purl":"pkg:github/bearbinary/omni-infra-provider-truenas","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bearbinary%2Fomni-infra-provider-truenas","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bearbinary%2Fomni-infra-provider-truenas/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bearbinary%2Fomni-infra-provider-truenas/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bearbinary%2Fomni-infra-provider-truenas/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bearbinary","download_url":"https://codeload.github.com/bearbinary/omni-infra-provider-truenas/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bearbinary%2Fomni-infra-provider-truenas/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32162614,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-22T17:06:48.269Z","status":"online","status_checked_at":"2026-04-23T02:00:06.710Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bare-metal","gitops","home-server","homelab","iac","infrastructure-as-code","json-rpc","k8s","kubernetes","omni","proxmox-alternative","self-hosted","sidero","talos","talos-linux","truenas","truenas-scale","virtualization","vm-provisioning","zfs"],"created_at":"2026-04-06T15:01:13.706Z","updated_at":"2026-04-23T02:02:55.653Z","avatar_url":"https://github.com/bearbinary.png","language":"Go","funding_links":["https://github.com/sponsors/bearbinary"],"categories":["Table of Contents"],"sub_categories":[],"readme":"\u003c!-- omni-infra-provider-truenas — TrueNAS SCALE infrastructure provider for Sidero Omni --\u003e\n\u003c!-- SPDX-License-Identifier: MIT --\u003e\n\u003c!-- keywords: truenas, omni, talos, kubernetes, infrastructure-provider, vm, zfs, json-rpc --\u003e\n\u003c!-- category: infrastructure, kubernetes, virtualization --\u003e\n\u003c!-- language: go --\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"assets/logo_with_name.png\" alt=\"Omni Infrastructure for TrueNAS\" width=\"420\" /\u003e\n\n\u003cbr /\u003e\n\u003cbr /\u003e\n\n**Automatically provision and manage Talos Linux VMs on TrueNAS SCALE through [Sidero Omni](https://omni.siderolabs.com/).**\n\n[![CI](https://github.com/bearbinary/omni-infra-provider-truenas/actions/workflows/ci.yaml/badge.svg)](https://github.com/bearbinary/omni-infra-provider-truenas/actions/workflows/ci.yaml)\n[![Release](https://github.com/bearbinary/omni-infra-provider-truenas/actions/workflows/release.yaml/badge.svg)](https://github.com/bearbinary/omni-infra-provider-truenas/actions/workflows/release.yaml)\n[![Go Version](https://img.shields.io/github/go-mod/go-version/bearbinary/omni-infra-provider-truenas?style=flat\u0026color=00ADD8)](go.mod)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![GitHub Release](https://img.shields.io/github/v/release/bearbinary/omni-infra-provider-truenas?style=flat\u0026color=blue)](https://github.com/bearbinary/omni-infra-provider-truenas/releases/latest)\n[![Docker Image](https://img.shields.io/badge/ghcr.io-omni--infra--provider--truenas-blue?logo=docker\u0026logoColor=white)](https://ghcr.io/bearbinary/omni-infra-provider-truenas)\n[![Quality Gate](https://sonarcloud.io/api/project_badges/measure?project=bearbinary_omni-infra-provider-truenas\u0026metric=alert_status)](https://sonarcloud.io/summary/new_code?id=bearbinary_omni-infra-provider-truenas)\n[![Tests](https://img.shields.io/badge/tests-572-brightgreen?logo=testcafe\u0026logoColor=white)](docs/testing.md)\n\n\u003cbr /\u003e\n\n[Getting Started](docs/getting-started.md) ·\n[Quick Start](#quick-start) ·\n[Configuration](#configuration) ·\n[Usage](#usage) ·\n[Architecture](#architecture) ·\n[FAQ](#faq) ·\n[AI/LLM Reference](llms.txt)\n\n\u003c/div\u003e\n\n\u003cbr /\u003e\n\n\u003e [!IMPORTANT]\n\u003e **Requires TrueNAS SCALE 25.04+ (Fangtooth).** This provider uses the JSON-RPC 2.0 API exclusively. The legacy REST v2.0 API is **not supported**.\n\n\u003e **New to Kubernetes?** Start with the [Getting Started guide](docs/getting-started.md) — a step-by-step tutorial from NAS to running cluster, no prior experience required.\n\n---\n\n## Overview\n\n**What is omni-infra-provider-truenas?** It is an open-source infrastructure provider that automatically provisions and manages Talos Linux virtual machines on TrueNAS SCALE through [Sidero Omni](https://omni.siderolabs.com/) — turning your NAS into a fully automated Kubernetes platform. It bridges Omni and [TrueNAS SCALE](https://www.truenas.com/truenas-scale/), enabling fully automated Kubernetes cluster provisioning on your own hardware. When Omni requests a machine, this provider creates a Talos Linux VM on TrueNAS — complete with ZFS-backed storage, network configuration, and automatic Omni enrollment.\n\n### Key Features\n\n- **Zero-touch VM lifecycle** — provision, start, stop, and destroy VMs automatically in response to Omni MachineRequests\n- **WebSocket JSON-RPC 2.0** — connects to TrueNAS via authenticated WebSocket with API key\n- **ZFS-native storage** — zvols for VM disks, automatic ISO caching with SHA-256 deduplication\n- **Multi-arch support** — `amd64` and `arm64` VM images via [Talos Image Factory](https://factory.talos.dev/)\n- **Flexible networking** — bridges, VLANs, or physical NICs\n- **Startup health checks** — validates pool, NIC, and API connectivity before accepting work\n- **Persistent storage** — Longhorn-ready with dedicated data disks (`storage_disk_size`). See [storage guide](docs/storage.md)\n- **Background cleanup** — automatically removes stale ISOs and orphan VMs/zvols\n- **OpenTelemetry observability** — traces, metrics, and profiling (optional)\n\n---\n\n## How It Works\n\n```mermaid\nflowchart TD\n    A[User scales cluster] --\u003e|MachineRequest| B[Sidero Omni]\n    B --\u003e|MachineRequest| C[omni-infra-provider-truenas]\n    C --\u003e|JSON-RPC 2.0| D[TrueNAS SCALE]\n    D --\u003e E[VM + zvol created]\n    E --\u003e|SideroLink · WireGuard| F[Joins Omni cluster]\n```\n\n1. **Omni creates a MachineRequest** — user scales a cluster or creates a MachineSet\n2. **Provider generates a Talos schematic** — defines the OS image with extensions (e.g., `qemu-guest-agent`)\n3. **Provider downloads the Talos ISO** — from [Image Factory](https://factory.talos.dev/), cached on TrueNAS to avoid re-downloading\n4. **Provider creates a VM** — zvol for disk, CDROM with ISO, NIC on your bridge, starts the VM\n5. **VM boots Talos, joins Omni** — via SideroLink (outbound WireGuard tunnel)\n\nWhen machines are removed, the provider stops the VM, deletes it, and cleans up the zvol.\n\n---\n\n## Transport\n\nThis provider communicates with TrueNAS via **JSON-RPC 2.0 over WebSocket** — the same protocol used by TrueNAS's own CLI (`midclt`) and web UI.\n\nAll deployments require `TRUENAS_HOST` and `TRUENAS_API_KEY`. TrueNAS 25.10 removed implicit authentication on the Unix socket, so an API key is required whether the provider runs on the TrueNAS host or elsewhere.\n\n\u003e The legacy REST v2.0 API (`/api/v2.0/`) is **not supported**. TrueNAS deprecated it in 25.04 and will remove it in 26.x.\n\n---\n\n## Quick Start\n\n### Prerequisites\n\n1. **TrueNAS SCALE 25.04+** — the JSON-RPC 2.0 API is required\n2. **Omni instance** with an infrastructure provider service account\n3. **ZFS pool** with available space\n4. **Network interface** for VM traffic — a bridge (e.g., `br0`), VLAN (e.g., `vlan100`), or physical NIC\n\n### Create the Omni Service Account\n\n```bash\nomnictl serviceaccount create --role=InfraProvider infra-provider:truenas\n# Save the output — it contains OMNI_SERVICE_ACCOUNT_KEY\n```\n\n### Option 1: Docker Compose on TrueNAS (Recommended)\n\nRun the container directly on your TrueNAS host via **Apps \u003e Discover \u003e Install via YAML**. Create an API key first — see [TrueNAS Setup \u003e API Key](docs/truenas-setup.md#5-api-key) for the recommended scoped-role setup (dedicated non-root user, minimum 11 roles). Do **not** use the `root` user's API key.\n\n```yaml\n# Paste into TrueNAS Apps \u003e Discover \u003e Install via YAML\nservices:\n  omni-infra-provider-truenas:\n    image: ghcr.io/bearbinary/omni-infra-provider-truenas:latest\n    restart: unless-stopped\n    network_mode: host\n    environment:\n      OMNI_ENDPOINT: \"https://omni.example.com\"\n      OMNI_SERVICE_ACCOUNT_KEY: \"\u003cyour-key\u003e\"\n      TRUENAS_HOST: \"localhost\"\n      TRUENAS_API_KEY: \"\u003cyour-truenas-api-key\u003e\"\n      TRUENAS_INSECURE_SKIP_VERIFY: \"true\"\n      DEFAULT_POOL: \"default\"\n      DEFAULT_NETWORK_INTERFACE: \"br0\"\n```\n\n### Option 2: Kubernetes (Helm)\n\n```bash\nhelm install omni-infra-provider deploy/helm/omni-infra-provider-truenas \\\n  --namespace omni-infra-provider --create-namespace \\\n  --set omniEndpoint=\"https://omni.example.com\" \\\n  --set truenasHost=\"truenas.local\" \\\n  --set secrets.omniServiceAccountKey=\"\u003cyour-key\u003e\" \\\n  --set secrets.truenasApiKey=\"\u003cyour-api-key\u003e\" \\\n  --set defaults.pool=\"default\"\n```\n\nSee [`deploy/helm/`](deploy/helm/omni-infra-provider-truenas/) for the chart and `values.yaml`.\n\n### Option 3: Docker Compose (Remote)\n\nFor running on a separate host via WebSocket:\n\n```bash\ncp .env.example .env\n# Fill in OMNI_ENDPOINT, OMNI_SERVICE_ACCOUNT_KEY, TRUENAS_HOST, TRUENAS_API_KEY\ndocker compose -f deploy/docker-compose.yaml up -d\n```\n\n---\n\n## Configuration\n\nAll configuration is via environment variables. A `.env` file is loaded automatically if present.\n\n### Required\n\n| Variable | Description |\n|---|---|\n| `OMNI_ENDPOINT` | Omni instance URL (e.g., `https://omni.example.com`) |\n| `OMNI_SERVICE_ACCOUNT_KEY` | Omni infra provider service account key |\n\n### TrueNAS Connection\n\n| Variable | Default | Description |\n|---|---|---|\n| `TRUENAS_HOST` | — | **Required.** TrueNAS hostname or IP (e.g., `truenas.local`, or `localhost` when running the container on the TrueNAS host itself) |\n| `TRUENAS_API_KEY` | — | **Required.** TrueNAS API key — create a dedicated non-root user with scoped roles ([setup](docs/truenas-setup.md#5-api-key)). Do not use the `root` user's key. |\n| `TRUENAS_INSECURE_SKIP_VERIFY` | `false` | Skip TLS verification for self-signed certs |\n\n### Provider Defaults\n\n| Variable | Default | Description |\n|---|---|---|\n| `DEFAULT_POOL` | `default` | ZFS pool for VM zvols and ISO cache |\n| `DEFAULT_NETWORK_INTERFACE` | — | Network interface for VM NICs (bridge, VLAN, or physical) |\n| `DEFAULT_BOOT_METHOD` | `UEFI` | VM boot method (`UEFI` or `BIOS`) |\n| `CONCURRENCY` | `4` | Max parallel provision/deprovision workers |\n| `LOG_LEVEL` | `info` | Log level: `debug`, `info`, `warn`, `error` |\n| `TRUENAS_MAX_CONCURRENT_CALLS` | `8` | Max concurrent JSON-RPC calls to TrueNAS |\n| `GRACEFUL_SHUTDOWN_TIMEOUT` | `30` | Seconds to wait for ACPI shutdown before force-stop on deprovision |\n| `MAX_ERROR_RECOVERIES` | `5` | Max consecutive ERROR recoveries before auto-replacing a VM (`-1` to disable) |\n| `HEALTH_LISTEN_ADDR` | `:8081` | Address for the HTTP health endpoint (`/healthz`, `/readyz`) |\n\n### Provider Identity (Optional)\n\n| Variable | Default | Description |\n|---|---|---|\n| `PROVIDER_ID` | `truenas` | Provider ID registered with Omni |\n| `PROVIDER_NAME` | `TrueNAS` | Display name in Omni UI |\n| `PROVIDER_DESCRIPTION` | `TrueNAS SCALE infrastructure provider` | Description in Omni UI |\n| `OMNI_INSECURE_SKIP_VERIFY` | `false` | Skip TLS verification for Omni connection |\n\n### Singleton Enforcement (Optional)\n\nPrevents two processes with the same `PROVIDER_ID` from racing on VM/zvol/ISO\noperations. On by default; see [`docs/architecture.md`](docs/architecture.md#singleton-enforcement)\nand [`docs/troubleshooting.md`](docs/troubleshooting.md#singleton-lease-acquire-failed--another-provider-instance-holds-the-singleton-lease).\n\n| Variable | Default | Description |\n|---|---|---|\n| `PROVIDER_SINGLETON_ENABLED` | `true` | Claim a distributed lease on startup; fail fast if another instance holds it |\n| `PROVIDER_SINGLETON_REFRESH_INTERVAL` | `15s` | How often to refresh the lease heartbeat |\n| `PROVIDER_SINGLETON_STALE_AFTER` | `45s` | Heartbeat age at which another instance may take over (must be `\u003e= 2x` refresh) |\n\n### Observability (Optional)\n\n| Variable | Description |\n|---|---|\n| `OTEL_EXPORTER_OTLP_ENDPOINT` | OpenTelemetry collector endpoint (e.g., `localhost:4317`) |\n| `OTEL_EXPORTER_OTLP_INSECURE` | Use insecure gRPC to collector |\n| `OTEL_SERVICE_NAME` | Override service name (default: `omni-infra-provider-truenas`) |\n| `PYROSCOPE_URL` | Pyroscope endpoint for continuous profiling (e.g., `http://localhost:4040`) |\n\nSee [`deploy/observability/`](deploy/observability/) for a ready-to-use Prometheus + Tempo + Grafana stack.\n\n### ISO Handling\n\nTalos ISOs are downloaded automatically from [Image Factory](https://factory.talos.dev/) based on the schematic generated for each machine request. ISOs are cached on the TrueNAS filesystem at `\u003cpool\u003e/talos-iso/` and deduplicated by SHA-256 hash — the same image is never downloaded twice.\n\n---\n\n## Usage\n\nOnce the provider is running and connected to Omni, create MachineClasses to trigger VM provisioning.\n\n### Via omnictl (CLI)\n\n**Create a MachineClass:**\n\n```bash\ncat \u003c\u003c'EOF' | omnictl apply -f -\nmetadata:\n  namespace: default\n  type: MachineClasses.omni.sidero.dev\n  id: truenas-small\nspec:\n  autoprovision:\n    providerid: truenas\n    grpcendpoint: \"\"\n    icon: \"\"\n    configpatch: |\n      cpus: 2\n      memory: 4096\n      disk_size: 40\nEOF\n```\n\n**With custom pool and NIC (overrides provider defaults):**\n\n```bash\ncat \u003c\u003c'EOF' | omnictl apply -f -\nmetadata:\n  namespace: default\n  type: MachineClasses.omni.sidero.dev\n  id: truenas-large\nspec:\n  autoprovision:\n    providerid: truenas\n    grpcendpoint: \"\"\n    icon: \"\"\n    configpatch: |\n      cpus: 8\n      memory: 16384\n      disk_size: 200\n      pool: \"fast-nvme\"\n      network_interface: \"vlan100\"\nEOF\n```\n\nAssign the MachineClass when creating a cluster or MachineSet in Omni.\n\n**Multi-pool setup (e.g., NVMe for control plane, HDD for workers):**\n\nCreate separate MachineClasses targeting different ZFS pools. Each VM's zvol and ISO cache are created on the specified pool.\n\n```bash\n# Control plane on fast NVMe pool\ncat \u003c\u003c'EOF' | omnictl apply -f -\nmetadata:\n  namespace: default\n  type: MachineClasses.omni.sidero.dev\n  id: truenas-cp-nvme\nspec:\n  autoprovision:\n    providerid: truenas\n    grpcendpoint: \"\"\n    icon: \"\"\n    configpatch: |\n      cpus: 2\n      memory: 2048\n      disk_size: 10\n      pool: \"fast-nvme\"\nEOF\n\n# Workers on bulk HDD pool (with storage disk for Longhorn)\ncat \u003c\u003c'EOF' | omnictl apply -f -\nmetadata:\n  namespace: default\n  type: MachineClasses.omni.sidero.dev\n  id: truenas-worker-hdd\nspec:\n  autoprovision:\n    providerid: truenas\n    grpcendpoint: \"\"\n    icon: \"\"\n    configpatch: |\n      cpus: 4\n      memory: 8192\n      disk_size: 100\n      pool: \"bulk-hdd\"\n      storage_disk_size: 100\nEOF\n```\n\nTo move a VM to a different pool, update the `pool` field in its MachineClass and let Omni deprovision/reprovision — Talos nodes are stateless, so this is safe and automatic.\n\n### Via Omni Web UI\n\n1. Navigate to **Clusters \u003e Create Cluster** (or edit an existing cluster)\n2. In the machine selection step, choose **Auto Provision** and select the `truenas` provider\n3. Configure CPUs, Memory, Disk Size, and optional overrides\n4. Set the desired replica count and create the cluster\n\nFields left blank use the provider's defaults (`DEFAULT_POOL`, `DEFAULT_NETWORK_INTERFACE`, etc.).\n\n### MachineClass Config Reference\n\nThese fields go in the MachineClass `configpatch` (CLI) or the provider config form (UI):\n\n| Field | Type | Required | Default | Description |\n|---|---|---|---|---|\n| `cpus` | int | Yes | `2` | Virtual CPUs (min: 1) |\n| `memory` | int | Yes | `4096` | Memory in MiB (min: 1024) |\n| `disk_size` | int | Yes | `40` | Root disk in GiB (min: 10) |\n| `pool` | string | Yes | — | ZFS **pool** name (top-level only — not a dataset path, see below) |\n| `network_interface` | string | Yes | — | Bridge, VLAN, or physical interface for VM NIC |\n| `boot_method` | string | Yes | `UEFI` | `UEFI` or `BIOS` |\n| `architecture` | string | Yes | `amd64` | `amd64` or `arm64` |\n| `dataset_prefix` | string | No | — | Dataset path under pool for isolation (see below) |\n| `advertised_subnets` | string | No | — | Comma-separated CIDRs to pin etcd/kubelet (required for multi-NIC) |\n| `encrypted` | bool | No | `false` | Enable ZFS AES-256-GCM encryption (passphrase auto-generated per zvol) |\n| `extensions` | list | No | — | Additional Talos extensions (merged with defaults) |\n| `additional_disks` | list | No | — | Extra data disks beyond root (each: `size` required, optional `pool`, `dataset_prefix`, `encrypted`) |\n| `additional_nics` | list | No | — | Extra NICs for network segmentation (each: `network_interface` required, optional `type`, `mtu`) |\n| `storage_disk_size` | int | No | — | Dedicated data disk (GiB) for Longhorn persistent storage |\n\n#### Understanding `pool` vs `dataset_prefix`\n\nZFS has a hierarchy: **pools** contain **datasets**, which contain **zvols** (virtual disks). The provider needs to know both where your storage lives:\n\n- **`pool`** — The **top-level ZFS pool name** only (e.g., `default`, `tank`, `fast-nvme`). This is NOT a dataset path. Run `zpool list` or check TrueNAS UI under **Storage** to see your pool names.\n- **`dataset_prefix`** — An optional **dataset path within the pool** where the provider should create VM storage. Use this when you want VMs organized under an existing dataset hierarchy.\n\nThe provider creates zvols at `\u003cpool\u003e/\u003cdataset_prefix\u003e/omni-vms/\u003cvm-id\u003e` and caches ISOs at `\u003cpool\u003e/\u003cdataset_prefix\u003e/talos-iso/`.\n\n**Example:** If your ZFS layout looks like this:\n\n```\ndefault                    ← pool\n  └── previewk8            ← dataset (created by you)\n        └── previewcluster ← zvol (existing VM disk)\n```\n\nThe correct MachineClass config is:\n\n```yaml\npool: \"default\"              # The pool name — NOT \"previewk8\" or \"default/previewk8\"\ndataset_prefix: \"previewk8\"  # The dataset path under the pool\n```\n\nThis creates VMs at `default/previewk8/omni-vms/...` — right alongside your existing `previewcluster` zvol.\n\n**Common mistake:** Setting `pool: \"previewk8\"` or `pool: \"default/previewk8\"` — both will fail with \"pool not found\" because `previewk8` is a dataset inside the `default` pool, not a pool itself.\n\n| Your ZFS layout | `pool` | `dataset_prefix` | VMs created at |\n|---|---|---|---|\n| `tank` (flat) | `tank` | _(empty)_ | `tank/omni-vms/...` |\n| `default/myproject` | `default` | `myproject` | `default/myproject/omni-vms/...` |\n| `default/prod/k8s` | `default` | `prod/k8s` | `default/prod/k8s/omni-vms/...` |\n| `fast-nvme` (separate pool) | `fast-nvme` | _(empty)_ | `fast-nvme/omni-vms/...` |\n\n### Recommended MachineClasses\n\n| Class | CPUs | Memory | Disk | Use Case |\n|---|---|---|---|---|\n| `truenas-control-plane` | 2 | 2048 MiB | 10 GiB | Control plane (etcd + API server) |\n| `truenas-worker` | 4 | 8192 MiB | 100 GiB | Workers (application workloads) |\n\n\u003e **Note:** Talos requires a minimum of 2 GiB RAM for control plane nodes. Control plane disks only need ~10 GiB (OS + etcd). Workers need more disk for container images.\n\n### Talos System Extensions\n\nEvery VM automatically includes these extensions:\n\n- `siderolabs/qemu-guest-agent` — hypervisor-to-guest communication\n- `siderolabs/util-linux-tools` — mount/block device operations\n- `siderolabs/iscsi-tools` — iSCSI initiator (required by Longhorn; also used by democratic-csi iSCSI mode)\n\nIf you need NFS client support (for democratic-csi NFS mode or manual NFS mounts), add `siderolabs/nfs-utils` to your MachineClass `extensions` field.\n\nAdd more via the `extensions` field in MachineClass config:\n\n```yaml\nextensions:\n  - \"siderolabs/iscsi-tools\"\n```\n\n---\n\n## Architecture\n\n```\ncmd/omni-infra-provider-truenas/\n├── main.go                     # Entry point, env config, transport auto-detection\n└── data/\n    ├── schema.json             # MachineClass config schema (served to Omni UI)\n    └── icon.svg                # Provider icon for Omni UI\n\ninternal/\n├── client/                     # TrueNAS JSON-RPC 2.0 client\n│   ├── transport.go            # Transport interface\n│   ├── ws.go                   # WebSocket transport (JSON-RPC 2.0, API key auth)\n│   ├── truenas.go              # Client constructor\n│   ├── vm.go                   # VM CRUD operations\n│   ├── device.go               # Device attachment (CDROM, DISK, NIC)\n│   ├── storage.go              # ZFS storage operations (zvols, ISOs)\n│   └── jsonrpc.go              # JSON-RPC 2.0 protocol implementation\n├── provisioner/\n│   ├── provisioner.go          # Provisioner struct, infra.Provisioner interface\n│   ├── steps.go                # 4 provision steps (schematic → ISO → VM → health)\n│   ├── deprovision.go          # VM teardown and cleanup\n│   └── data.go                 # MachineClass config parsing + validation\n├── singleton/\n│   └── singleton.go            # Distributed lease preventing duplicate instances\n├── cleanup/\n│   └── cleanup.go              # Background stale ISO / orphan VM cleanup\n├── health/\n│   └── health.go               # HTTP health endpoint (/healthz, /readyz)\n├── monitor/\n│   └── monitor.go              # Host health monitoring (OTEL gauges)\n├── resources/\n│   ├── machine.go              # COSI Machine typed resource\n│   └── meta/meta.go            # Provider ID constant\n└── telemetry/\n    ├── telemetry.go            # OpenTelemetry + Pyroscope init\n    └── metrics.go              # Custom metrics\n\ndeploy/\n├── docker-compose.yaml         # Docker Compose for remote deployment\n├── helm/                       # Helm chart\n└── observability/              # Prometheus + Tempo + Grafana stack\n\napi/specs/\n├── specs.proto                 # Protobuf definition for Machine resource\n└── specs.pb.go                 # Generated Go code\n```\n\n### Provision Flow\n\n```mermaid\nflowchart LR\n    A[MachineRequest] --\u003e B[\"createSchematic\\n\\nGenerate Talos image\\nschematic with extensions\"]\n    B --\u003e C[\"uploadISO\\n\\nDownload from Image Factory\\nSHA-256 dedup · upload to pool\"]\n    C --\u003e D[\"createVM\\n\\nCreate zvol · create VM\\nattach devices · start + poll\"]\n    D --\u003e E[\"healthCheck\\n\\nVerify VM exists\\non TrueNAS\"]\n```\n\n---\n\n## Supply Chain Security\n\nAll Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/) (keyless, via Sigstore). Every release includes an SBOM (SPDX) and binary signatures.\n\n### Verify Docker Image\n\n```bash\ncosign verify --certificate-identity-regexp=\"https://github.com/bearbinary/omni-infra-provider-truenas\" --certificate-oidc-issuer=\"https://token.actions.githubusercontent.com\" ghcr.io/bearbinary/omni-infra-provider-truenas:v0.13.0\n```\n\n### Verify Binary\n\n```bash\ncosign verify-blob --certificate omni-infra-provider-truenas-linux-amd64.cert --signature omni-infra-provider-truenas-linux-amd64.sig --certificate-identity-regexp=\"https://github.com/bearbinary/omni-infra-provider-truenas\" --certificate-oidc-issuer=\"https://token.actions.githubusercontent.com\" omni-infra-provider-truenas-linux-amd64\n```\n\n### View SBOM\n\nDownload `sbom.spdx.json` from the [release assets](https://github.com/bearbinary/omni-infra-provider-truenas/releases/latest).\n\n---\n\n## Grafana Dashboards\n\nFour ready-to-import Grafana dashboards ship with each release. They cover VM health, provisioning latency, TrueNAS API performance, and cleanup metrics.\n\n### Quick install (recommended)\n\nDownload the dashboard bundle from the latest release and import via the Grafana UI:\n\n```bash\ncurl -L https://github.com/bearbinary/omni-infra-provider-truenas/releases/latest/download/grafana-dashboards.zip -o dashboards.zip\nunzip dashboards.zip -d dashboards/\n```\n\nThen in Grafana: **Dashboards → New → Import → Upload JSON file**. When prompted, select your Prometheus, Tempo, Loki, and Pyroscope data sources.\n\n### Import from grafana.com\n\nOnce approved on [grafana.com/dashboards](https://grafana.com/grafana/dashboards/), you can import by ID from **Dashboards → New → Import → Paste ID**.\n\n| Dashboard | ID | Description |\n|---|---|---|\n| Omni TrueNAS Provider — Overview | _pending_ | VM count, host health, pool status, and provisioning rate |\n| Omni TrueNAS Provider — VM Provisioning | _pending_ | Per-step provision/deprovision latency, error categories, ISO cache hits |\n| Omni TrueNAS Provider — TrueNAS API Performance | _pending_ | JSON-RPC call latency by method, WebSocket reconnects, rate limit queue depth |\n| Omni TrueNAS Provider — Cleanup \u0026 Maintenance | _pending_ | Stale ISO cleanup, orphan VM detection, zvol reclaim metrics |\n\n### Local dev stack\n\nRun `docker compose -f deploy/observability/docker-compose.yaml up -d` to start Prometheus, Tempo, Loki, Pyroscope, and Grafana locally. Dashboards are autoloaded at [localhost:3000](http://localhost:3000).\n\n---\n\n## Development\n\n```bash\nmake build              # Build binary to _out/\nmake test               # Run unit tests\nmake test-v             # Verbose unit tests\nmake test-integration   # Integration tests (requires TrueNAS instance)\nmake test-e2e           # Full E2E tests (requires TrueNAS instance)\nmake lint               # Run linters\nmake image              # Build Docker image\nmake generate           # Regenerate protobuf from specs.proto\n```\n\nIntegration and E2E tests require a real TrueNAS SCALE instance. See [`docs/testing.md`](docs/testing.md) for setup instructions.\n\nFor detailed system design, see [`docs/architecture.md`](docs/architecture.md). For networking (bridges, DHCP, MetalLB, VIP, UniFi), see [`docs/networking.md`](docs/networking.md). For persistent storage (Longhorn setup), see [`docs/storage.md`](docs/storage.md). For common issues, see [`docs/troubleshooting.md`](docs/troubleshooting.md). For version upgrades, see [`docs/upgrading.md`](docs/upgrading.md).\n\n### Binary Releases\n\nMulti-platform binaries are built automatically on every release:\n\n- `linux/amd64`, `linux/arm64`\n- `darwin/amd64`, `darwin/arm64`\n\nDocker images are published to `ghcr.io/bearbinary/omni-infra-provider-truenas` with multi-arch support.\n\n---\n\n## FAQ\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eDoes Omni cost money?\u003c/strong\u003e\u003c/summary\u003e\n\nOmni has a free tier for personal/homelab use. Check [omni.siderolabs.com](https://omni.siderolabs.com/) for current pricing. You can also self-host Omni.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eWill this affect my NAS performance?\u003c/strong\u003e\u003c/summary\u003e\n\nYes — VMs share your NAS hardware (CPU, RAM, disk). Start small and monitor. You can remove VMs anytime if things slow down.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eHow much disk space do the VMs use?\u003c/strong\u003e\u003c/summary\u003e\n\nTalos ISO: ~100 MB (cached once). Control plane: ~10 GB each. Worker: 40-100 GB each. All ZFS-compressed — actual usage is often less.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eCan I use this without internet?\u003c/strong\u003e\u003c/summary\u003e\n\nNo. VMs need outbound HTTPS for Talos ISO download (first time, then cached) and SideroLink (WireGuard on port 443) to Omni. No inbound ports required.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eCan I SSH into the Kubernetes nodes?\u003c/strong\u003e\u003c/summary\u003e\n\nNo. Talos Linux has no SSH by design. Manage nodes through `kubectl`, `talosctl`, and the Omni UI.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eWhat if my NAS reboots?\u003c/strong\u003e\u003c/summary\u003e\n\nVMs restart with TrueNAS. The provider auto-recovers and reconnects to Omni. Kubernetes restarts your workloads automatically.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eWhat's the minimum hardware?\u003c/strong\u003e\u003c/summary\u003e\n\nA 1-node cluster needs ~4 cores, 16 GB RAM, and 50 GB free disk (including what TrueNAS uses). See the [Getting Started guide](docs/getting-started.md#hardware-requirements) for full sizing.\n\u003c/details\u003e\n\nFor more questions, see the [Getting Started FAQ](docs/getting-started.md#faq).\n\n---\n\n## Related Projects\n\n- [Sidero Omni](https://github.com/siderolabs/omni) — SaaS Kubernetes management platform\n- [Talos Linux](https://github.com/siderolabs/talos) — Immutable Kubernetes OS\n- [TrueNAS SCALE](https://github.com/truenas/middleware) — Open-source storage and virtualization platform\n\n---\n\n## Contributing\n\nWe use an **issues-only** contribution model — no pull requests. Open an issue describing what you'd like, and optionally prototype in a fork. See [CONTRIBUTING.md](CONTRIBUTING.md) for details.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n\nBuilt by [Bear Binary](https://github.com/bearbinary).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbearbinary%2Fomni-infra-provider-truenas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbearbinary%2Fomni-infra-provider-truenas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbearbinary%2Fomni-infra-provider-truenas/lists"}