{"id":51595222,"url":"https://github.com/hyperized/silo","last_synced_at":"2026-07-11T18:01:48.062Z","repository":{"id":361929947,"uuid":"1252817313","full_name":"hyperized/silo","owner":"hyperized","description":"Cloud-native distributed storage system in Go for Kubernetes workloads","archived":false,"fork":false,"pushed_at":"2026-06-28T15:47:23.000Z","size":836,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-28T17:22:33.606Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hyperized.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-28T22:43:19.000Z","updated_at":"2026-06-28T15:47:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hyperized/silo","commit_stats":null,"previous_names":["hyperized/silo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hyperized/silo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperized%2Fsilo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperized%2Fsilo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperized%2Fsilo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperized%2Fsilo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hyperized","download_url":"https://codeload.github.com/hyperized/silo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperized%2Fsilo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35370428,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-11T02:00:05.354Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-11T18:01:47.390Z","updated_at":"2026-07-11T18:01:48.056Z","avatar_url":"https://github.com/hyperized.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# silo\n\n**Distributed block storage for Kubernetes that you can actually operate.**\n\nsilo is a symmetric storage system in Go. One binary per node, no metadata tier,\nno quorum to lose, no rebalance ceremony. It gives your pods `ReadWriteOnce`\nvolumes through an ordinary `PersistentVolumeClaim`, and it heals itself through\nnetwork partitions without paging you.\n\n**Jump to:** [Design principles](#design-principles) · [What you get](#what-you-get) · [Try locally](#try-it-in-5-minutes-local) · [Deploy](#deploy) · [Kubernetes](#use-it-on-kubernetes) · [Performance](#performance) · [What works today](#what-works-today) · [Docs](#documentation)\n\n---\n\n## Design principles\n\nsilo's bet is that storage you can operate beats storage with every feature: one\ncomponent, deployed the same way everywhere, with as few decisions as possible\nbetween you and a working volume. Five choices follow, and a feature that would\nviolate one doesn't ship.\n\n- **One binary, symmetric nodes.** Every node runs the same `silod` and can do\n  every job. There are no monitor/metadata/OSD roles to size, place, and\n  babysit. You deploy one thing and scale it by adding more of the same.\n- **No quorum to lose.** Membership is SWIM gossip and the namespace is a CRDT, so\n  a partition keeps serving on both sides and **converges automatically** when the\n  network heals. No Raft, no \"2 of 3 monitors up.\" A node joins by pointing at one\n  peer and leaves by dying. Re-replication is a paced background task, not a\n  stop-the-world rebalance.\n- **Recoverable by default.** No protocol step needs a human to break a tie. A\n  volume is an extent map of immutable, encrypted chunks under a **fenced**\n  single-writer lease, so a split brain can't corrupt it and snapshots are free.\n- **Errors are instructions.** Every error tells you what to do next, not just\n  what broke.\n- **Stdlib-first.** External dependencies require justification: the FUSE protocol\n  and CSI bindings are built in-tree, and volumes attach over **NBD** from the\n  mainline kernel. There is no client to install and little to audit.\n\nsilo trades a few things for this: volumes are `ReadWriteOnce`, the FUSE surface\nis close-to-open coherent (NFS-style), and there's no erasure coding yet. If you\nneed RWX or EC today, silo isn't there; see [docs/known-gaps.md](docs/known-gaps.md).\nHow this compares to Ceph, in operational surface and write latency, is in\n[docs/performance.md](docs/performance.md).\n\n---\n\n## What you get\n\n- **Block volumes for Kubernetes.** An ordinary `PersistentVolumeClaim` against\n  the `silo` StorageClass becomes a `ReadWriteOnce` device in your pod. CSI-native\n  snapshots and clones; a fenced lease so a rescheduled pod takes over its disk\n  instead of corrupting it.\n- **Self-healing replication.** Every chunk lives on N nodes. Lose one and its\n  chunks re-replicate onto survivors in the background. There is no rebalance to\n  babysit.\n- **Encrypted at rest, always.** Chunks are AES-GCM encrypted with per-chunk keys;\n  cluster traffic is mTLS. Keys come from a file or a cloud KMS (AWS/GCP/Azure).\n- **Partition-tolerant namespace.** `mkdir`/`touch`/`ls`/`rm` keep working on both\n  sides of a split and converge afterward, with conflicts surfaced rather than\n  silently dropped.\n- **A shared filesystem too.** A from-scratch FUSE surface gives RWX,\n  close-to-open coherent access where you need many readers.\n- **Operability built in.** Prometheus metrics, a ready-made Grafana overview\n  dashboard (health markers over per-subject graphs), and a `siloctl` CLI for\n  status, draining, and capacity rebalancing.\n\n---\n\n## Try it in 5 minutes (local)\n\nNo Kubernetes required. This boots a real 3-node cluster with Prometheus and\nGrafana, then creates and snapshots a volume.\n\n```sh\ngit clone https://github.com/hyperized/silo \u0026\u0026 cd silo\n\nmake up        # build + boot 3 silod nodes (+ Prometheus on :9090, Grafana on :3030)\nmake build     # compile ./bin/{silod,siloctl,silo-csi}\n```\n\n`make up` scrapes silo-a's bootstrap token from the container logs and prints a\npaste-ready `siloctl auth init` command. Run it to claim your operator\ncredentials (this writes the cluster CA + your client cert to `~/.config/silo/`\nand remembers the server):\n\n```sh\n./bin/siloctl auth init \\\n  --token \u003cTOKEN\u003e \\\n  --server 127.0.0.1:7001 \\\n  --server-fingerprint \u003cFINGERPRINT\u003e\n```\n\nNow drive the cluster; every command authenticates over mTLS automatically:\n\n```sh\n./bin/siloctl volume create /db --size 10G        # create a 10 GiB block volume\n./bin/siloctl volume snapshot /db /db-backup      # instant copy-on-write snapshot\n./bin/siloctl ns ls /                             # list the namespace\n./bin/siloctl auth status                         # show your cluster credentials\n```\n\nWatch it heal: `docker kill silo-b`, then `make status` and re-list. The cluster\nstays available and re-replicates in the background. `make down` tears it all down.\n\n\u003e Block-mounting a volume locally needs silod's NBD server (`SILO_NBD_ADDR`) plus\n\u003e `nbd-client` on the host. On Kubernetes the CSI node plugin does this for you.\n\n---\n\n## Deploy\n\nThe same binary runs everywhere; what changes is *where* it runs. There are two\nshapes, and you can start in one and grow into the other with no data migration.\n\n- **Co-located (small scale).** Run silod on machines you already have: your\n  Kubernetes control-plane or worker nodes, or a couple of existing hosts. In\n  Kubernetes it runs as a **DaemonSet** on node-local disk; outside Kubernetes\n  it's the **standalone binary** under systemd. No dedicated storage hardware.\n  This is the right starting point and is production-capable across 3+ nodes.\n- **Dedicated (larger scale).** Run silod on its own set of machines sized for\n  storage and point your clients at them. Same binary, same wiring; only the\n  disks move off the compute nodes, so storage and compute scale\n  independently.\n\n| Shape | Nodes | Survives node loss? | For |\n|---|---|---|---|\n| **Standalone** | 1 | No (one copy; [back it up](docs/operations.md#backups)) | edge, a single host, dev |\n| **Co-located cluster** | 3+ | Yes (replication factor N) | silod alongside your existing nodes |\n| **Dedicated cluster** | 3+ | Yes | a storage fleet serving many clients |\n\nStandalone is one line; adding nodes (same CA, same encryption key) grows it into\neither cluster shape:\n\n```sh\nSILO_ENCRYPTION_KEY_SOURCE=file SILO_ENCRYPTION_KEY_PATH=/etc/silo/key \\\nSILO_DATA_DIR=/var/lib/silo SILO_NBD_ADDR=0.0.0.0:10809 ./bin/silod\n```\n\nFull recipes (co-located DaemonSet, dedicated fleet, CA seeding, advertise\naddresses, KMS, backups) are in\n**[docs/operations.md → Deployment paths](docs/operations.md#deployment-paths)**.\n\n---\n\n## Use it on Kubernetes\n\nThe [`silo-csi` Helm chart](deploy/helm/silo-csi) installs the driver: a\ncontroller `Deployment` (with the standard external-provisioner/attacher/\nsnapshotter sidecars) and a node `DaemonSet`. You point it at a running silo\ncluster, whether that's co-located on your nodes or a dedicated fleet.\n**Prerequisite:** the `nbd` kernel module on each node (`modprobe nbd`).\n\n```sh\nhelm install silo-csi deploy/helm/silo-csi \\\n  --namespace silo-system --create-namespace \\\n  --set silod.address=silo.silo-system.svc.cluster.local:7000\n\nkubectl apply -f deploy/helm/silo-csi/examples/pvc.yaml\n```\n\nFrom there it's the path your app teams already know: an ordinary PVC against the\n`silo` StorageClass:\n\n```yaml\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: data\nspec:\n  accessModes: [\"ReadWriteOnce\"]\n  storageClassName: silo\n  resources:\n    requests:\n      storage: 10Gi\n```\n\nSnapshots are CSI-native (`VolumeSnapshot`), and a snapshot can seed a new PVC\n(clone/restore). Image building, StorageClass parameters, the snapshot workflow,\nand troubleshooting are in **[docs/kubernetes.md](docs/kubernetes.md)**.\n\n---\n\n## Performance\n\nsilo's writes are slower than Ceph's at default settings, and the trade is\ndeliberate: **every write is flushed to real disk on multiple nodes before silod\nsays OK**. There is no journal to drain and no in-flight state to recover after\na node loss. The number you benchmark on day one is the number production sees.\n\nThat costs you on small synchronous writes (transaction logs, `fsync`-heavy\nfilesystems); bulk sequential I/O is much closer. `SILO_REPLICATION=2` and the\nper-volume `chunk-size` are the knobs. The full explanation (what each system\ndoes on a write and why) is in **[docs/performance.md](docs/performance.md)**.\n\n---\n\n## What works today\n\n| Capability | Status |\n|---|---|\n| Encrypted, replicated chunk store (AES-GCM, N=3, consistent-hash placement) | ✅ |\n| Gossip membership (SWIM) + mTLS, token-based operator join | ✅ |\n| CRDT namespace: partition-tolerant `mkdir`/`touch`/`ls`/`rm`, conflict surfacing | ✅ |\n| Writer/reader SDKs, writer-owned chunks (no metadata round-trip on writes) | ✅ |\n| Block volumes: extent map, fenced single-writer lease, NBD server, COW snapshots | ✅ |\n| **Kubernetes CSI driver**: provision, attach, mount, snapshot, clone; Helm chart | ✅ |\n| Restart-resilient attachments: volumes pause and reconnect across silod rollouts, no I/O errors | ✅ |\n| FUSE (RWX) filesystem surface: from-scratch protocol library, close-to-open | ✅ |\n| Observability/ops: Prometheus metrics, **Grafana overview dashboard**, `siloctl status`, drain, capacity rebalance | ✅ |\n| Security hardening: cloud KMS keys (AWS/GCP/Azure), cert auto-rotation, CRL revocation, scoped capability tokens | ✅ |\n| Durability/upgrades: encrypted backup to S3/GCS/Azure, rolling-upgrade protocol handshake | ✅ |\n| nvme-tcp block transport | ⏳ v1.1 (kernel-bound) |\n\n---\n\n## Documentation\n\n- **[docs/kubernetes.md](docs/kubernetes.md)**: install and operate silo-csi on Kubernetes (Helm values, StorageClass, snapshots, troubleshooting)\n- **[docs/operations.md](docs/operations.md)**: operator guide covering configuration, deployment paths (co-located and dedicated), mTLS/credentials, NBD, troubleshooting\n- **[docs/performance.md](docs/performance.md)**: the trade vs Ceph (operational surface, the write path, and the measured baseline)\n- **[docs/runbook.md](docs/runbook.md)**: production readiness checklist, golden-signal alerts, failure-recovery playbooks\n- **[docs/threat-model.md](docs/threat-model.md)**: what silo defends against, how, and the current security edges\n- **[docs/known-gaps.md](docs/known-gaps.md)**: what's not finished yet and why (deferred work, kernel-bound seams)\n- **[deploy/helm/silo-csi/README.md](deploy/helm/silo-csi/README.md)**: chart reference\n- **[.env.example](.env.example)**: annotated `silod` configuration\n\n---\n\n## Building from source\n\n```sh\nmake build              # ./bin/{silod,siloctl,silo-csi}\nmake images             # silo/silod and silo/silo-csi container images\nmake test               # unit tests\nmake test-integration   # end-to-end against a real silod (build-tag 'integration')\nmake test-nbd-kernel    # NBD attach/reconnect against a real kernel (privileged docker)\nmake check              # fmt + vet + lint + test\n```\n\nRequirements: Go 1.25+, and Docker for the local cluster and image builds.\n\n## License\n\nSee the repository for license details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperized%2Fsilo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhyperized%2Fsilo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperized%2Fsilo/lists"}