{"id":15172361,"url":"https://github.com/blockdaemon/solana-cluster","last_synced_at":"2025-10-26T02:31:08.835Z","repository":{"id":37920888,"uuid":"442803527","full_name":"Blockdaemon/solana-cluster","owner":"Blockdaemon","description":"Tooling to manage Solana snapshots","archived":false,"fork":false,"pushed_at":"2025-02-08T05:10:20.000Z","size":889,"stargazers_count":14,"open_issues_count":5,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-02-08T06:18:51.981Z","etag":null,"topics":["cluster","rpc","solana"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Blockdaemon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-29T14:50:12.000Z","updated_at":"2025-02-08T05:10:17.000Z","dependencies_parsed_at":"2024-01-19T19:30:11.134Z","dependency_job_id":"d1115635-cfeb-4925-a850-475ead8d074b","html_url":"https://github.com/Blockdaemon/solana-cluster","commit_stats":{"total_commits":255,"total_committers":7,"mean_commits":36.42857142857143,"dds":0.3411764705882353,"last_synced_commit":"b70319c94514f1a99811b78b82a0115889f4b52e"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockdaemon%2Fsolana-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockdaemon%2Fsolana-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockdaemon%2Fsolana-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blockdaemon%2Fsolana-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Blockdaemon","download_url":"https://codeload.github.com/Blockdaemon/solana-cluster/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238247978,"owners_count":19440879,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cluster","rpc","solana"],"created_at":"2024-09-27T10:00:43.594Z","updated_at":"2025-10-26T02:31:08.465Z","avatar_url":"https://github.com/Blockdaemon.png","language":"Go","readme":"# Solana Cluster Manager\n\nTooling to manage private clusters of Solana nodes.\n\n\u003ca href=\"https://pkg.go.dev/go.blockdaemon.com/solana/cluster-manager\"\u003e\u003cimg src=\"https://pkg.go.dev/badge/go.blockdaemon.com/solana/cluster-manager.svg\" alt=\"Go Reference\"\u003e\u003c/a\u003e\n\u003cimg src=\"https://github.com/Blockdaemon/solana-cluster/actions/workflows/build.yml/badge.svg?branch=main\"\u003e\n\u003cimg src=\"https://github.com/Blockdaemon/solana-cluster/actions/workflows/test.yml/badge.svg?branch=main\"\u003e\n\n[Issue Tracker](https://github.com/orgs/Blockdaemon/projects/1/views/1)\n\n\n## Deployment\n\n### Building from source\n\nRequirements: Go 1.18 \u0026 build essentials.\n\n```shell\ngo mod download\ngo build -o ./solana-cluster .\n```\n\n### Docker Image\n\nFind amd64 and arm64 Docker images for all releases on [GitHub Container Registry](https://github.com/Blockdaemon/solana-cluster/pkgs/container/solana-cluster-manager).\n\n```shell\ndocker pull ghcr.io/blockdaemon/solana-cluster-manager\ndocker run \\\n  --network-mode=host \\\n  -v /solana/ledger:/ledger:ro \\\n  ghcr.io/blockdaemon/solana-cluster-manager \\\n  sidecar --ledger /ledger\n```\n\n## Usage\n\n```\n$ solana-cluster sidecar --help\n\nRuns on a Solana node and serves available snapshot archives.\nDo not expose this API publicly.\n\nUsage:\n  solana-snapshots sidecar [flags]\n\nFlags:\n      --interface string    Only accept connections from this interface\n      --ledger string       Path to ledger dir\n      --port uint16         Listen port (default 13080)\n```\n\n```\n$ solana-cluster tracker --help\n\nConnects to sidecars on nodes and scrapes the available snapshot versions.\nProvides an API allowing fetch jobs to find the latest snapshots.\nDo not expose this API publicly.\n\nUsage:\n  solana-snapshots tracker [flags]\n\nFlags:\n      --config string            Path to config file\n      --internal-listen string   Internal listen URL (default \":8457\")\n      --listen string            Listen URL (default \":8458\")\n```\n\n```\n$ solana-cluster fetch --help\n\nFetches a snapshot from another node using the tracker API.\n\nUsage:\n  solana-snapshots fetch [flags]\n\nFlags:\n      --download-timeout duration   Max time to try downloading in total (default 10m0s)\n      --ledger string               Path to ledger dir\n      --max-slots uint              Refuse to download \u003cn\u003e slots older than the newest (default 10000)\n      --min-slots uint              Download only snapshots \u003cn\u003e slots newer than local (default 500)\n      --request-timeout duration    Max time to wait for headers (excluding download) (default 3s)\n      --tracker string              Download as instructed by given tracker URL\n```\n\n```\n$ solana-cluster mirror --help\n\nPeriodically mirrors snapshots from nodes to an S3-compatible data store.\nSpecify credentials via env $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY\n\nUsage:\n  solana-snapshots mirror [flags]\n\nFlags:\n      --refresh duration   Refresh interval to discover new snapshots (default 30s)\n      --s3-bucket string   Bucket name\n      --s3-prefix string   Prefix for S3 object names (optional)\n      --s3-region string   S3 region (optional)\n      --s3-secure          Use secure S3 transport (default true)\n      --s3-url string      URL to S3 API\n      --tracker string     URL to tracker API\n```\n\n## Architecture\n\n### Snapshot management\n\n**[Twitter 🧵](https://twitter.com/terorie_dev/status/1520289936611725312)**\n\nSnapshot management tooling enables efficient peer-to-peer transfers of accounts database archives.\n\n![Snapshot Fetch](./docs/snapshots.png)\n\n**Scraping** (Flow A)\n\nSnapshot metadata collection runs periodically similarly to Prometheus scraping.\n\nEach cluster-aware node runs a lightweight `solana-cluster sidecar` agent providing telemetry about its snapshots.\n\nThe `solana-cluster tracker` then connects to all sidecars to assemble a complete list of snapshot metadata.\nThe tracker is stateless so it can be replicated.\nService discovery is available through HTTP and JSON files. Consul SD support is planned.\n\nSide note: Snapshot sources are configurable in stock Solana software but only via static lists.\nThis does not scale well with large fleets because each cluster change requires updating the lists of all nodes.\n\n**Downloading** (Flow B)\n\nWhen a Solana node needs to fetch a snapshot remotely, the tracker helps it find the best snapshot source.\nNodes will download snapshots directly from the sidecars of other nodes.\n\n### TPU \u0026 TVU\n\nNot yet public. 🚜 Subscribe to releases! ✨\n\n## Motivation\n\nBlockdaemon manages one of the largest Solana validator and RPC infrastructure deployments to date, backed by a custom peer-to-peer backbone.\nThis repository shares our performance and sustainability optimizations.\n\nWhen Solana validators first start, they have to retrieve and validate hundreds of gigabytes of state data from a remote node.\nDuring normal operation, validators stream at least 500 Mbps of traffic in either direction.\n\nFor Solana infra operators that manage more than node (not to mention hundreds), this cost currently scales linearly as well.\nUnmodified Solana deployments treat their cluster peers the same as any other.\nThis can end in a large number of streams between globally dispersed validators.\n\nThis is obviously inefficient. 10 Gbps connectivity is cheap and abundant locally within data centers.\nIn contrast, major public clouds (who shall not be named) charge egregious premiums on Internet traffic.\n\nThe solution: Co-located Solana validators that are controlled by the same entity should also behave as one entity.\n\nLeveraging internal connectivity to distribute blockchain data can\nreduce public network _leeching_ and increase total cluster bandwidth.\n\nAuthenticated internal connectivity allows delegation of expensive calculations and re-use of results thereof.\nConcretely, the amount write-heavy snapshot creation \u0026 verification procedures per node can decrease as the cluster scales out.\n","funding_links":[],"categories":["Solana Validator Implementations"],"sub_categories":["Validator Tools and Resources"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockdaemon%2Fsolana-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblockdaemon%2Fsolana-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblockdaemon%2Fsolana-cluster/lists"}