{"id":38115702,"url":"https://github.com/rajsinghtech/garage-operator","last_synced_at":"2026-05-03T02:05:28.191Z","repository":{"id":332891263,"uuid":"1134774906","full_name":"rajsinghtech/garage-operator","owner":"rajsinghtech","description":"A Kubernetes operator for managing Garage - a distributed S3-compatible object storage system designed for self-hosting.","archived":false,"fork":false,"pushed_at":"2026-04-26T19:47:33.000Z","size":1217,"stargazers_count":182,"open_issues_count":4,"forks_count":11,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-26T20:27:05.439Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://deepwiki.com/rajsinghtech/garage-operator","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rajsinghtech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-15T07:24:48.000Z","updated_at":"2026-04-26T19:36:10.000Z","dependencies_parsed_at":"2026-03-11T23:01:01.135Z","dependency_job_id":null,"html_url":"https://github.com/rajsinghtech/garage-operator","commit_stats":null,"previous_names":["rajsinghtech/garage-operator"],"tags_count":69,"template":false,"template_full_name":null,"purl":"pkg:github/rajsinghtech/garage-operator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajsinghtech%2Fgarage-operator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajsinghtech%2Fgarage-operator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajsinghtech%2Fgarage-operator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajsinghtech%2Fgarage-operator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rajsinghtech","download_url":"https://codeload.github.com/rajsinghtech/garage-operator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajsinghtech%2Fgarage-operator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32312505,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T19:15:34.056Z","status":"ssl_error","status_checked_at":"2026-04-26T19:15:15.467Z","response_time":129,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-16T22:20:33.380Z","updated_at":"2026-05-03T02:05:28.178Z","avatar_url":"https://github.com/rajsinghtech.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Garage Kubernetes Operator\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"logo.svg\" alt=\"Garage Kubernetes Operator\" width=\"128\" height=\"128\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003eS3-Compatible Object Storage on Kubernetes\u003c/strong\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/rajsinghtech/garage-operator/actions/workflows/test.yml\"\u003e\u003cimg src=\"https://github.com/rajsinghtech/garage-operator/actions/workflows/test.yml/badge.svg\" alt=\"CI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://goreportcard.com/report/github.com/rajsinghtech/garage-operator\"\u003e\u003cimg src=\"https://goreportcard.com/badge/github.com/rajsinghtech/garage-operator\" alt=\"Go Report Card\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/rajsinghtech/garage-operator/releases/latest\"\u003e\u003cimg src=\"https://img.shields.io/github/v/release/rajsinghtech/garage-operator\" alt=\"Latest Release\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://deepwiki.com/rajsinghtech/garage-operator\"\u003e\u003cimg src=\"https://deepwiki.com/badge.svg\" alt=\"Ask DeepWiki\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nA Kubernetes operator for [Garage](https://garagehq.deuxfleurs.fr/) - distributed, self-hosted object storage with multi-cluster federation.\n\n- **Declarative cluster lifecycle** — StatefulSet, config, and layout managed via CRDs\n- **Bucket \u0026 key management** — create buckets, quotas, and S3 credentials with kubectl\n- **Multi-cluster federation** — span storage across Kubernetes clusters with automatic node discovery\n- **Gateway clusters** — stateless S3 proxies that scale independently from storage\n- **Scale subresource** — `kubectl scale` and VPA/HPA support for GarageCluster\n- **COSI driver** — optional Kubernetes-native object storage provisioning\n\n## Custom Resources\n\n| CRD | Description |\n|-----|-------------|\n| `GarageCluster` | Deploys and manages a Garage cluster (storage or gateway) |\n| `GarageBucket` | Creates buckets with quotas and website hosting |\n| `GarageKey` | Provisions S3 access keys with per-bucket permissions |\n| `GarageNode` | Fine-grained node layout control (zone, capacity, tags) |\n| `GarageAdminToken` | Manages admin API tokens |\n| `GarageReferenceGrant` | Grants cross-namespace access to clusters and buckets |\n\n## Install\n\n```bash\nhelm install garage-operator oci://ghcr.io/rajsinghtech/charts/garage-operator \\\n  --namespace garage-operator-system \\\n  --create-namespace\n```\n\n## Quick Start\n\nFirst, create an admin token secret for the operator to manage Garage resources:\n\n```bash\nkubectl create secret generic garage-admin-token \\\n  --from-literal=admin-token=$(openssl rand -hex 32)\n```\n\nCreate a 3-node Garage cluster ([full example](config/samples/garage_v1beta1_garagecluster.yaml)):\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageCluster\nmetadata:\n  name: garage\nspec:\n  replicas: 3\n  zone: us-east-1\n  replication:\n    factor: 3\n  storage:\n    data:\n      size: 100Gi\n  network:\n    rpcBindPort: 3901\n    service:\n      type: ClusterIP\n  admin:\n    adminTokenSecretRef:\n      name: garage-admin-token\n      key: admin-token\n```\n\nWait for the cluster to be ready:\n\n```bash\nkubectl wait --for=condition=Ready garagecluster/garage --timeout=300s\n```\n\nCreate a bucket:\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageBucket\nmetadata:\n  name: my-bucket\nspec:\n  clusterRef:\n    name: garage\n  quotas:\n    maxSize: 10Gi\n```\n\nCreate access credentials:\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageKey\nmetadata:\n  name: my-key\nspec:\n  clusterRef:\n    name: garage\n  bucketPermissions:\n    - bucketRef:\n        name: my-bucket\n      read: true\n      write: true\n```\n\nOr grant access to **all buckets** in the cluster — useful for admin tools, monitoring, or [mountpoint-s3](https://github.com/awslabs/mountpoint-s3) workloads that span multiple buckets:\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageKey\nmetadata:\n  name: admin-key\nspec:\n  clusterRef:\n    name: garage\n  allBuckets:\n    read: true\n    write: true\n    owner: true\n```\n\nPer-bucket overrides layer on top of `allBuckets`, so you can combine cluster-wide read with owner on a specific bucket:\n\n```yaml\n  allBuckets:\n    read: true\n  bucketPermissions:\n    - bucketRef:\n        name: metrics-bucket\n      owner: true\n```\n\nImport existing credentials from an inline spec or a Kubernetes secret:\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageKey\nmetadata:\n  name: imported-key\nspec:\n  clusterRef:\n    name: garage\n  importKey:\n    accessKeyId: \"GKxxxxxxxxxxxxxxxxxxxxxxxx\"\n    secretAccessKey: \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n```\n\nOr reference an existing secret — use `accessKeyIdKey`/`secretAccessKeyKey` to specify which keys to read from the source secret (defaults to `access-key-id`/`secret-access-key`):\n\n```yaml\n  importKey:\n    secretRef:\n      name: my-existing-creds\n    accessKeyIdKey: AWS_ACCESS_KEY_ID\n    secretAccessKeyKey: AWS_SECRET_ACCESS_KEY\n```\n\n### Secret Template\n\nBy default the generated secret includes `access-key-id`, `secret-access-key`, `endpoint`, `host`, `scheme`, and `region`. Use `secretTemplate` to customize what gets included and how keys are named:\n\n```yaml\nsecretTemplate:\n  accessKeyIdKey: AWS_ACCESS_KEY_ID\n  secretAccessKeyKey: AWS_SECRET_ACCESS_KEY\n  endpointKey: AWS_ENDPOINT_URL_S3\n  regionKey: AWS_REGION\n  includeEndpoint: false   # omit endpoint/host/scheme\n  includeRegion: false     # omit region\n```\n\nThis is useful when mounting the secret directly as environment variables with `envFrom` — only the keys your app expects will be present.\n\nGet S3 credentials:\n\n```bash\nkubectl get secret my-key -o jsonpath='{.data.access-key-id}' | base64 -d \u0026\u0026 echo\nkubectl get secret my-key -o jsonpath='{.data.secret-access-key}' | base64 -d \u0026\u0026 echo\nkubectl get secret my-key -o jsonpath='{.data.endpoint}' | base64 -d \u0026\u0026 echo\n```\n\n## Gateway Clusters\n\nGateway clusters handle S3 API requests without storing data. They connect to a storage cluster and scale independently, ideal for edge deployments or handling high request volumes. See [gateway examples](config/samples/garage_v1beta1_garagecluster_gateway.yaml) for more configurations.\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageCluster\nmetadata:\n  name: garage-gateway\nspec:\n  replicas: 5\n  gateway: true\n  connectTo:\n    clusterRef:\n      name: garage  # Reference to storage cluster\n  replication:\n    factor: 3       # Must match storage cluster\n  admin:\n    adminTokenSecretRef:\n      name: garage-admin-token\n      key: admin-token\n```\n\nKey differences from storage clusters:\n- Uses a **StatefulSet with metadata PVC** for node identity persistence (no data PVC)\n- Registers pods as **gateway nodes** in the layout (capacity=null)\n- Requires `connectTo` to reference a storage cluster\n- Lightweight and horizontally scalable\n\nFor cross-namespace or external storage clusters, use `rpcSecretRef` and `adminApiEndpoint`:\n\n```yaml\nconnectTo:\n  rpcSecretRef:\n    name: garage-rpc-secret\n    key: rpc-secret\n  adminApiEndpoint: \"http://garage.storage-namespace.svc.cluster.local:3903\"\n  adminTokenSecretRef:\n    name: storage-admin-token\n    key: admin-token\n```\n\n### External Storage (NAS, Bare Metal)\n\nTo connect a gateway to a Garage instance running outside Kubernetes (e.g., on a NAS or bare-metal server), use `bootstrapPeers` instead of `clusterRef`. Get the node ID from your external Garage with `garage node id`.\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageCluster\nmetadata:\n  name: garage-gateway\nspec:\n  replicas: 2\n  gateway: true\n  replication:\n    factor: 3  # Must match the external cluster\n  connectTo:\n    rpcSecretRef:\n      name: garage-rpc-secret\n      key: rpc-secret\n    bootstrapPeers:\n      - \"563e1ac825ee3323aa441e72c26d1030d6d4414aeb3dd25287c531e7fc2bc95d@nas.local:3901\"\n  admin:\n    adminTokenSecretRef:\n      name: garage-admin-token\n      key: admin-token\n```\n\nThe gateway pods will connect to the external nodes via the RPC port and register as gateway nodes in the existing cluster layout.\n\n## Manual Node Layout (GarageNode)\n\nBy default, GarageCluster uses `layoutPolicy: Auto` — the operator assigns every pod to the Garage layout using the cluster's zone and PVC-derived capacity. For fine-grained control over individual nodes, set `layoutPolicy: Manual` and create GarageNode resources.\n\nEach GarageNode creates a single-replica StatefulSet and manages that node's layout entry (zone, capacity, tags).\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageNode\nmetadata:\n  name: storage-node-a\nspec:\n  clusterRef:\n    name: garage\n  zone: zone-a\n  capacity: 500Gi\n  tags: [\"ssd\", \"high-performance\"]\n  storage:\n    metadata:\n      size: 10Gi\n    data:\n      size: 500Gi\n      storageClassName: fast-ssd\n```\n\n### Gateway Nodes\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageNode\nmetadata:\n  name: gateway-node\nspec:\n  clusterRef:\n    name: garage\n  zone: zone-a\n  gateway: true\n  storage:\n    metadata:\n      size: 1Gi\n```\n\n### External Nodes\n\nFor nodes running outside Kubernetes (bare-metal, NAS, other clusters):\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageNode\nmetadata:\n  name: external-node\nspec:\n  clusterRef:\n    name: garage\n  nodeId: \"563e1ac825ee3323aa441e72c26d1030d6d4414aeb3dd25287c531e7fc2bc95d\"\n  zone: dc-1\n  capacity: 1Ti\n  external:\n    address: nas.local\n    port: 3901\n```\n\nExternal nodes require `nodeId` (64-hex-char Ed25519 public key). No StatefulSet is created — the operator only manages the layout entry.\n\n### Per-Node Overrides\n\nGarageNode supports overriding cluster defaults: `image`, `resources`, `nodeSelector`, `tolerations`, `affinity`, `podAnnotations`, `podLabels`, and `priorityClassName`.\n\n### Status\n\n```bash\nkubectl get garagenodes\n# NAME              CLUSTER   ZONE    CAPACITY   GATEWAY   CONNECTED   INLAYOUT   AGE\n# storage-node-a    garage    zone-a  500Gi      false     true        true       5m\n```\n\nThe controller auto-discovers node IDs from pods, reconciles layout drift (zone/capacity/tags), and handles node removal with replication-safe finalization.\n\n## Scaling\n\nGarageCluster supports the Kubernetes [scale subresource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource), enabling `kubectl scale` and compatibility with autoscalers like VPA and HPA.\n\n```bash\nkubectl scale garagecluster garage --replicas=5\n```\n\nThe operator populates `status.replicas`, `status.readyReplicas`, and `status.selector` for the scale subresource to function correctly.\n\n## PVC Retention Policy\n\nBy default, PVCs created by a GarageCluster's StatefulSet are **not deleted** when the cluster is deleted or scaled down. This is intentional: Garage stores your data in those volumes, and automatic deletion would be irreversible.\n\nThe behavior is controlled by `spec.storage.pvcRetentionPolicy`:\n\n| Field | Value | Behavior |\n|-------|-------|----------|\n| `whenDeleted` | `Retain` (default) | PVCs survive GarageCluster deletion — manual cleanup required |\n| `whenDeleted` | `Delete` | PVCs are deleted automatically when the GarageCluster is deleted |\n| `whenScaled` | `Retain` (default) | PVCs for scaled-down pods are kept (allows scaling back up) |\n| `whenScaled` | `Delete` | PVCs for removed replicas are deleted on scale-down |\n\nFor dev/test clusters where you want automatic cleanup:\n\n```yaml\nspec:\n  storage:\n    pvcRetentionPolicy:\n      whenDeleted: Delete\n      whenScaled: Delete\n```\n\nRequires Kubernetes 1.23+. For production clusters, leave this unset (defaults to `Retain`) or set `whenScaled: Delete` only if you're confident scaled-down nodes won't need their data again.\n\n## Operational Annotations\n\nOne-shot operational commands are triggered by setting annotations on the resource. The operator processes the annotation, acts on it, removes it, and records the result in `status.lastOperation`. If the operation fails, the annotation is retained so the next reconcile retries.\n\n### Maintenance Mode\n\nTo suspend reconciliation during planned maintenance, use `spec.maintenance.suspended`:\n\n```yaml\nspec:\n  maintenance:\n    suspended: true\n```\n\nThe operator requeues every 5 minutes but makes no changes while suspended. Clear the field to resume.\n\n\u003e **Deprecated:** The `garage.rajsingh.info/pause-reconcile: \"true\"` annotation still works but `spec.maintenance.suspended` is preferred — it is version-controlled, visible in `kubectl get`, and works with GitOps tools.\n\n### GarageCluster\n\n| Annotation | Value | Action |\n|---|---|---|\n| `garage.rajsingh.info/trigger-snapshot` | `\"true\"` | Trigger a metadata database snapshot on all nodes. Keeps the 2 most recent snapshots. |\n| `garage.rajsingh.info/trigger-repair` | repair type | Launch a repair operation on all nodes. Valid types: `Tables`, `Blocks`, `Versions`, `MultipartUploads`, `BlockRefs`, `BlockRc`, `Rebalance`, `Aliases`, `ClearResyncQueue`. |\n| `garage.rajsingh.info/scrub-command` | command | Control the block integrity scrub worker on all nodes. Valid commands: `start`, `pause`, `resume`, `cancel`. |\n| `garage.rajsingh.info/revert-layout` | `\"true\"` | Discard all staged layout changes. Does **not** undo an already-applied layout version — only clears the pending staging area. |\n| `garage.rajsingh.info/retry-block-resync` | `\"true\"` or hashes | Clear the resync backoff for blocks so they are retried immediately. Use `\"true\"` to retry all errored blocks, or a comma-separated list of 64-hex-char block hashes to retry specific ones. |\n| `garage.rajsingh.info/purge-blocks` | hashes | **Irreversible.** Permanently delete all S3 objects that reference the listed blocks. Value is a comma-separated list of 64-hex-char block hashes. Only use when you are certain the data is unrecoverable and must be removed from the cluster. |\n| `garage.rajsingh.info/force-layout-apply` | `\"true\"` | Force-apply a staged layout version. |\n| `garage.rajsingh.info/connect-nodes` | `nodeId@addr:port,...` | Connect to external nodes (one-shot federation bootstrap). |\n\n**Example — trigger a Tables repair and check the result:**\n```bash\nkubectl annotate garagecluster garage garage.rajsingh.info/trigger-repair=Tables\nkubectl get garagecluster garage -o jsonpath='{.status.lastOperation}'\n# {\"type\":\"Repair:Tables\",\"triggeredAt\":\"2026-05-02T10:00:00Z\",\"succeeded\":true}\n```\n\n**Example — discard staged layout changes:**\n```bash\nkubectl annotate garagecluster garage garage.rajsingh.info/revert-layout=true\n```\n\n**Example — retry all block resync errors:**\n```bash\nkubectl annotate garagecluster garage garage.rajsingh.info/retry-block-resync=true\n# Or retry specific blocks:\nkubectl annotate garagecluster garage \\\n  'garage.rajsingh.info/retry-block-resync=abc123...,def456...'\n```\n\n**Example — purge a lost block (last resort):**\n```bash\n# First confirm the block is truly unrecoverable with: garage block list-errors\nkubectl annotate garagecluster garage \\\n  'garage.rajsingh.info/purge-blocks=abc123def456...'\n```\n\n**Example — run and then pause a scrub:**\n```bash\nkubectl annotate garagecluster garage garage.rajsingh.info/scrub-command=start\n# Later...\nkubectl annotate garagecluster garage garage.rajsingh.info/scrub-command=pause\n```\n\n\u003e **Note:** `trigger-repair: Scrub` is not supported — use `scrub-command: start` instead.\n\n### Operation Status\n\nAll triggered operations record their outcome in `status.lastOperation`:\n\n```yaml\nstatus:\n  lastOperation:\n    type: \"Repair:Blocks\"\n    triggeredAt: \"2026-05-02T10:00:00Z\"\n    succeeded: true\n```\n\nOn failure, `succeeded: false` and `error` contains the message. The annotation is kept so the next reconcile retries automatically.\n\n### GarageBucket\n\n| Annotation | Value | Action |\n|---|---|---|\n| `garage.rajsingh.info/cleanup-mpu` | `\"true\"` | Delete incomplete multipart uploads older than the threshold (default: 24h). |\n| `garage.rajsingh.info/cleanup-mpu-older-than` | duration | Age threshold for MPU cleanup (e.g. `\"12h\"`, `\"30m\"`). Only used with `cleanup-mpu`. Defaults to `24h` if absent or invalid. |\n\n**Example — clean up stale uploads older than 48 hours:**\n```bash\nkubectl annotate garagebucket my-bucket \\\n  garage.rajsingh.info/cleanup-mpu=true \\\n  garage.rajsingh.info/cleanup-mpu-older-than=48h\n```\n\n## Worker Tuning\n\nGarage runs several background workers that can be tuned at runtime. Set `spec.workers` to configure them — the operator applies the values on every reconcile so they persist across pod restarts.\n\n```yaml\nspec:\n  workers:\n    scrubTranquility: 4      # default: 2, higher = slower scrub, less disk pressure\n    resyncWorkerCount: 2     # default: 1, range: 1-8\n    resyncTranquility: 4     # default: 2, higher = slower resync\n```\n\n| Field | Garage variable | Default | Notes |\n|---|---|---|---|\n| `scrubTranquility` | `scrub-tranquility` | 2 | Pauses between block integrity checks. Higher = less disk I/O. |\n| `resyncWorkerCount` | `resync-worker-count` | 1 | Parallel block resync goroutines. Max 8. |\n| `resyncTranquility` | `resync-tranquility` | 2 | Pauses between block resyncs. Higher = less disk I/O. |\n\nCurrent values are visible in `status.workers.variables`. Unset fields leave the corresponding Garage default unchanged.\n\n## Website Hosting\n\nWebsite hosting is **enabled by default** on every GarageCluster. Buckets with website hosting enabled are served at `\u003cbucket\u003e.\u003croot-domain\u003e` on port 3902.\n\nThe default `rootDomain` is `.\u003ccluster-name\u003e.\u003cnamespace\u003e.svc`, so a bucket named `my-site` on a cluster named `garage` in namespace `default` is accessible at `my-site.garage.default.svc:3902`.\n\nTo use a custom domain:\n\n```yaml\nspec:\n  webApi:\n    rootDomain: \".web.garage.example.com\"\n```\n\nThen enable website hosting on a bucket:\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageBucket\nmetadata:\n  name: my-site\nspec:\n  clusterRef:\n    name: garage\n  website:\n    enabled: true\n    indexDocument: index.html\n    errorDocument: error.html\n```\n\nThe site is served at `my-site.web.garage.example.com:3902`. Point DNS (wildcard CNAME or per-bucket) at the Garage service, and optionally front it with an ingress or HTTPRoute.\n\nOnce website hosting is enabled and the bucket has a global alias, the operator populates `status.websiteUrl`:\n\n```bash\nkubectl get garagebucket my-site -o jsonpath='{.status.websiteUrl}'\n# http://my-site.web.garage.example.com\n```\n\nOther options:\n\n```yaml\nspec:\n  webApi:\n    rootDomain: \".web.garage.example.com\"\n    bindPort: 8080           # default: 3902\n    addHostToMetrics: true   # adds domain to Prometheus labels\n```\n\nTo disable website hosting entirely:\n\n```yaml\nspec:\n  webApi:\n    disabled: true\n```\n\n## K2V API\n\nThe [K2V API](https://garagehq.deuxfleurs.fr/documentation/reference-manual/k2v/) provides a key-value store on top of Garage. Add `k2vApi` to enable it:\n\n```yaml\nspec:\n  k2vApi:\n    bindPort: 3904  # default\n```\n\nOmit `k2vApi` entirely to disable. The K2V endpoint is exposed on the same Service as the S3 API.\n\n## Namespace Isolation\n\nBy default, all cross-namespace references are **denied**. A `GarageKey` in namespace `team-b` cannot reference a `GarageCluster` or `GarageBucket` in namespace `storage-admin` unless the admin of `storage-admin` explicitly grants it.\n\n### GarageReferenceGrant\n\n`GarageReferenceGrant` (short: `grg`) lives in the **destination** namespace — the one that owns the `GarageCluster` or `GarageBucket`. Only admins of that namespace can create it, so tenants cannot self-grant access.\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageReferenceGrant\nmetadata:\n  name: allow-team-b\n  namespace: storage-admin      # destination namespace\nspec:\n  from:\n    - kind: GarageKey\n      namespace: team-b         # who is allowed to reference\n    - kind: GarageBucket\n      namespace: team-b\n  to:\n    - kind: GarageCluster\n      name: my-cluster          # specific cluster (omit name to allow all)\n```\n\nOnce this grant exists, `team-b` can create a `GarageKey` that references the cluster cross-namespace:\n\n```yaml\napiVersion: garage.rajsingh.info/v1beta1\nkind: GarageKey\nmetadata:\n  name: my-key\n  namespace: team-b\nspec:\n  clusterRef:\n    name: my-cluster\n    namespace: storage-admin    # cross-namespace — requires the grant above\n  bucketPermissions:\n    - bucketRef:\n        name: my-bucket\n      read: true\n      write: true\n```\n\nThe same grant mechanism applies to:\n- `GarageKey.spec.clusterRef` — which cluster the key belongs to\n- `GarageKey.spec.bucketPermissions[].bucketRef.namespace` — cross-namespace bucket references\n- `GarageBucket.spec.clusterRef` — cross-namespace cluster for a bucket\n- `GarageAdminToken.spec.clusterRef` — cross-namespace cluster for an admin token\n\n`GarageNode` does **not** support cross-namespace cluster references — node management is always same-namespace.\n\n### Generated Secrets\n\nSecrets generated by `GarageKey` and `GarageAdminToken` are always written to the same namespace as the resource. To make a secret available in another namespace, use a tool like [ExternalSecrets](https://external-secrets.io/) or [Reflector](https://github.com/emberstack/kubernetes-reflector).\n\n### Multi-Tenant Setup Example\n\nA typical setup: the platform team owns `storage-admin`, tenants live in their own namespaces.\n\n```\nstorage-admin/\n  GarageCluster: main-cluster\n  GarageReferenceGrant: allow-team-a (→ team-a GarageKey + GarageBucket)\n  GarageReferenceGrant: allow-team-b (→ team-b GarageKey)\n\nteam-a/\n  GarageBucket: team-a-bucket   (clusterRef.namespace: storage-admin)\n  GarageKey: team-a-key         (clusterRef.namespace: storage-admin)\n\nteam-b/\n  GarageKey: team-b-key         (clusterRef.namespace: storage-admin)\n```\n\nTenants can only access what the platform team grants them. Revoking access is as simple as deleting the `GarageReferenceGrant`.\n\n## Multi-Cluster Federation\n\nGarage supports federating clusters across Kubernetes clusters for geo-distributed storage. All clusters share the same RPC secret and Garage distributes replicas across zones automatically.\n\n1. Create the same RPC secret in every Kubernetes cluster:\n   ```bash\n   SECRET=$(openssl rand -hex 32)\n   kubectl create secret generic garage-rpc-secret --from-literal=rpc-secret=$SECRET\n   ```\n\n2. Configure `remoteClusters` and `publicEndpoint` on each GarageCluster:\n   ```yaml\n   apiVersion: garage.rajsingh.info/v1beta1\n   kind: GarageCluster\n   metadata:\n     name: garage\n   spec:\n     replicas: 3\n     zone: us-east-1\n     replication:\n       factor: 3\n     network:\n       rpcSecretRef:\n         name: garage-rpc-secret\n         key: rpc-secret\n     publicEndpoint:\n       type: LoadBalancer\n       loadBalancer:\n         perNode: true\n     remoteClusters:\n       - name: eu-west\n         zone: eu-west-1\n         connection:\n           adminApiEndpoint: \"http://garage-eu.example.com:3903\"\n           adminTokenSecretRef:\n             name: garage-admin-token\n             key: admin-token\n     admin:\n       adminTokenSecretRef:\n         name: garage-admin-token\n         key: admin-token\n   ```\n\nThe operator handles node discovery, layout coordination, and health monitoring across clusters. Each cluster needs a `publicEndpoint` so remote nodes can reach it on the RPC port. See the [Garage documentation](https://garagehq.deuxfleurs.fr/documentation/cookbook/real-world/) for networking requirements.\n\n## Monitoring\n\nThe operator integrates with Prometheus Operator for metrics scraping and alerting.\n\n### ServiceMonitor\n\nEnable `spec.monitoring` on a `GarageCluster` to create a `ServiceMonitor` targeting the admin API `/metrics` endpoint. Covers both Auto-mode pods and Manual-mode `GarageNode` pods via the `garage.rajsingh.info/cluster` label selector.\n\n```yaml\nspec:\n  monitoring:\n    enabled: true\n    interval: 30s          # optional, defaults to Prometheus global interval\n    additionalLabels:\n      release: monitoring  # match your Prometheus serviceMonitorSelector\n```\n\nIf the cluster uses `metricsTokenSecretRef`, the generated ServiceMonitor will include `Authorization: Bearer` from that secret. Ensure your Prometheus instance has RBAC to `get` secrets in the Garage namespace.\n\n### PrometheusRules\n\nThe Helm chart includes alerting rules covering node availability, cluster health (quorum, partitions, disconnected nodes), RPC error rate, block resync errors, and low disk space:\n\n```yaml\n# values.yaml\nprometheusRules:\n  enabled: true\n  labels:\n    release: monitoring\n```\n\n### Grafana Dashboard\n\nThe Helm chart ships the official [Garage Prometheus dashboard](https://garagehq.deuxfleurs.fr/documentation/cookbook/monitoring/) as a ConfigMap:\n\n```yaml\n# values.yaml\ngrafanaDashboard:\n  enabled: true\n  labels:\n    grafana_dashboard: \"1\"    # Grafana sidecar pattern\n```\n\nIf you use the **Grafana Operator** (`grafana.integreatly.org`), create a `GrafanaDashboard` CR in the same namespace as your cluster pointing at the ConfigMap:\n\n```yaml\napiVersion: grafana.integreatly.org/v1beta1\nkind: GrafanaDashboard\nmetadata:\n  name: garage\n  namespace: garage           # same namespace as the ConfigMap\nspec:\n  allowCrossNamespaceImport: true\n  instanceSelector:\n    matchLabels:\n      grafana.internal/instance: grafana\n  folder: Garage\n  configMapRef:\n    name: \u003ccluster-name\u003e-garage-dashboard\n    key: garage-prometheus.json\n  datasources:\n    - inputName: DS_PROMETHEUS\n      datasourceName: Prometheus\n```\n\n\u003e **Note**: `grafanaDashboard` in the Helm chart creates a single cluster-agnostic ConfigMap (`\u003crelease\u003e-garage-dashboard`). The `GrafanaDashboard` CR pointing at it can live anywhere with `allowCrossNamespaceImport: true`.\n\n## CSI-S3: Mount Buckets as Persistent Volumes\n\nYou can use [k8s-csi-s3](https://github.com/yandex-cloud/k8s-csi-s3) to mount Garage buckets as PersistentVolumes via FUSE. This is useful for workloads that need filesystem-style access to S3 data (e.g., shared config, static assets, ML datasets).\n\n1. Create a dedicated bucket and key:\n   ```yaml\n   apiVersion: garage.rajsingh.info/v1beta1\n   kind: GarageBucket\n   metadata:\n     name: csi-s3\n   spec:\n     clusterRef:\n       name: garage\n     globalAlias: csi-s3\n     quotas:\n       maxSize: 5Ti\n       maxObjects: 10000000\n     keyPermissions:\n       - keyRef:\n           name: csi-s3-key\n         read: true\n         write: true\n   ---\n   apiVersion: garage.rajsingh.info/v1beta1\n   kind: GarageKey\n   metadata:\n     name: csi-s3-key\n   spec:\n     clusterRef:\n       name: garage\n     name: \"CSI-S3 Storage Key\"\n     secretTemplate:\n       name: csi-s3-secret\n       accessKeyIdKey: accessKeyID\n       secretAccessKeyKey: secretAccessKey\n       additionalData:\n         endpoint: \"http://garage.garage.svc.cluster.local:3900\"\n         region: \"garage\"\n     bucketPermissions:\n       - bucketRef:\n           name: csi-s3\n         read: true\n         write: true\n   ```\n\n   The `additionalData` fields on the secret template provide the S3 endpoint and region that the CSI driver expects in the secret.\n\n2. Install the CSI driver via Helm:\n   ```bash\n   helm repo add csi-s3 https://yandex-cloud.github.io/k8s-csi-s3/charts\n   helm install csi-s3 csi-s3/csi-s3 \\\n     --namespace csi-s3 --create-namespace \\\n     --set storageClass.singleBucket=csi-s3 \\\n     --set 'storageClass.mountOptions=--memory-limit 1000 --dir-mode 0777 --file-mode 0666' \\\n     --set secret.create=false\n   ```\n\n   Setting `secret.create=false` tells the chart to use the `csi-s3-secret` created by the GarageKey controller.\n\n3. Create a PVC and use it:\n   ```yaml\n   apiVersion: v1\n   kind: PersistentVolumeClaim\n   metadata:\n     name: my-s3-pvc\n   spec:\n     accessModes:\n       - ReadWriteMany\n     storageClassName: csi-s3\n     resources:\n       requests:\n         storage: 10Gi\n   ---\n   apiVersion: v1\n   kind: Pod\n   metadata:\n     name: test-s3-mount\n   spec:\n     containers:\n       - name: app\n         image: busybox\n         command: [\"sleep\", \"infinity\"]\n         volumeMounts:\n           - name: data\n             mountPath: /data\n     volumes:\n       - name: data\n         persistentVolumeClaim:\n           claimName: my-s3-pvc\n   ```\n\n\u003e **Note:** FUSE-backed S3 mounts have limitations — no true random writes, no `fsync`, and higher latency than block storage. The csi-s3 namespace requires the `pod-security.kubernetes.io/enforce: privileged` label. For native S3 API access, use GarageKey secrets directly.\n\n## COSI Support (Optional)\n\nThe operator includes an optional COSI (Container Object Storage Interface) driver that provides Kubernetes-native object storage provisioning.\n\n### Enabling COSI\n\n1. Install the COSI CRDs:\n   ```bash\n   for crd in bucketclaims bucketaccesses bucketclasses bucketaccessclasses buckets; do\n     kubectl apply -f \"https://raw.githubusercontent.com/kubernetes-sigs/container-object-storage-interface/main/client/config/crd/objectstorage.k8s.io_${crd}.yaml\"\n   done\n   ```\n\n2. Deploy the COSI controller:\n   ```bash\n   kubectl apply -k \"github.com/kubernetes-sigs/container-object-storage-interface/controller?ref=main\"\n   ```\n\n3. Install the operator with COSI enabled:\n   ```bash\n   helm install garage-operator oci://ghcr.io/rajsinghtech/charts/garage-operator \\\n     --namespace garage-operator-system \\\n     --create-namespace \\\n     --set cosi.enabled=true\n   ```\n\n### Using COSI\n\n1. Create a BucketClass:\n   ```yaml\n   apiVersion: objectstorage.k8s.io/v1alpha2\n   kind: BucketClass\n   metadata:\n     name: garage-standard\n   spec:\n     driverName: garage.rajsingh.info\n     deletionPolicy: Delete\n     parameters:\n       clusterRef: garage\n       clusterNamespace: garage-operator-system\n   ```\n\n2. Create a BucketAccessClass:\n   ```yaml\n   apiVersion: objectstorage.k8s.io/v1alpha2\n   kind: BucketAccessClass\n   metadata:\n     name: garage-readwrite\n   spec:\n     driverName: garage.rajsingh.info\n     authenticationType: Key\n     parameters:\n       clusterRef: garage\n       clusterNamespace: garage-operator-system\n   ```\n\n3. Request a bucket:\n   ```yaml\n   apiVersion: objectstorage.k8s.io/v1alpha2\n   kind: BucketClaim\n   metadata:\n     name: my-bucket\n   spec:\n     bucketClassName: garage-standard\n     protocols:\n     - S3\n   ```\n\n4. Request access credentials:\n   ```yaml\n   apiVersion: objectstorage.k8s.io/v1alpha2\n   kind: BucketAccess\n   metadata:\n     name: my-bucket-access\n   spec:\n     bucketAccessClassName: garage-readwrite\n     protocol: S3\n     bucketClaims:\n       - bucketClaimName: my-bucket\n         accessMode: ReadWrite\n         accessSecretName: my-bucket-creds\n   ```\n\n5. Use the credentials in your application:\n   ```yaml\n   env:\n   - name: S3_ENDPOINT\n     valueFrom:\n       secretKeyRef:\n         name: my-bucket-creds\n         key: COSI_S3_ENDPOINT\n   - name: AWS_ACCESS_KEY_ID\n     valueFrom:\n       secretKeyRef:\n         name: my-bucket-creds\n         key: COSI_S3_ACCESS_KEY_ID\n   - name: AWS_SECRET_ACCESS_KEY\n     valueFrom:\n       secretKeyRef:\n         name: my-bucket-creds\n         key: COSI_S3_ACCESS_SECRET_KEY\n   ```\n\n### COSI Limitations\n\n- Only S3 protocol is supported\n- Only Key authentication is supported (no IAM)\n- Bucket deletion requires the bucket to be empty first\n- Upstream COSI controller does not yet implement deletion — `DriverDeleteBucket` and `DriverRevokeBucketAccess` are implemented but won't be called until upstream adds support\n\n## Documentation\n\n- [Helm Chart](charts/garage-operator/) - Installation and configuration\n- [Garage Docs](https://garagehq.deuxfleurs.fr/) - Garage project documentation\n\n## Development\n\n```bash\nmake dev-up       # Start kind cluster with operator\nmake dev-test     # Apply test resources\nmake dev-status   # View cluster status\nmake dev-logs     # Stream operator logs\nmake dev-down     # Tear down\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frajsinghtech%2Fgarage-operator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frajsinghtech%2Fgarage-operator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frajsinghtech%2Fgarage-operator/lists"}