{"id":28795051,"url":"https://github.com/clyso/chorus","last_synced_at":"2026-01-14T21:57:53.077Z","repository":{"id":218872276,"uuid":"730603979","full_name":"clyso/chorus","owner":"clyso","description":"s3 multi provider data lifecycle management","archived":false,"fork":false,"pushed_at":"2026-01-13T17:43:12.000Z","size":2546,"stargazers_count":114,"open_issues_count":34,"forks_count":15,"subscribers_count":7,"default_branch":"main","last_synced_at":"2026-01-13T19:25:49.255Z","etag":null,"topics":["backup","go","migration","rclone","replication","s3","s3-storage"],"latest_commit_sha":null,"homepage":"https://clyso.com/eu/en/products/chorus/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clyso.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-12-12T09:35:18.000Z","updated_at":"2026-01-08T13:41:45.000Z","dependencies_parsed_at":"2024-06-21T14:13:41.163Z","dependency_job_id":"09f95201-f74b-4e91-a8c2-40ed77c40584","html_url":"https://github.com/clyso/chorus","commit_stats":null,"previous_names":["clyso/chorus"],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/clyso/chorus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clyso%2Fchorus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clyso%2Fchorus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clyso%2Fchorus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clyso%2Fchorus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clyso","download_url":"https://codeload.github.com/clyso/chorus/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clyso%2Fchorus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28436255,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T21:32:52.117Z","status":"ssl_error","status_checked_at":"2026-01-14T21:32:33.442Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","go","migration","rclone","replication","s3","s3-storage"],"created_at":"2025-06-18T03:01:53.032Z","updated_at":"2026-01-14T21:57:53.070Z","avatar_url":"https://github.com/clyso.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"[![CI](https://github.com/clyso/chorus/actions/workflows/ci-go.yml/badge.svg)](https://github.com/clyso/chorus/actions/workflows/ci-go.yml)\n![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/clyso/chorus)\n[![GoDoc](https://godoc.org/github.com/clyso/chorus?status.svg)](https://pkg.go.dev/github.com/clyso/chorus?tab=doc)\n[![Go Report Card](https://goreportcard.com/badge/github.com/clyso/chorus)](https://goreportcard.com/report/github.com/clyso/chorus)\n\n# Chorus\n![chorus.png](./docs/media/banner.png)\n\nChorus is a distributed, vendor-agnostic, S3-compatible tool for backup, migration, and routing. It enables:\n - Faster data transfers between S3 storages using multiple machines.\n - Resumable transfers with checkpointing on failure.\n - Syncing of existing buckets, objects, and metadata (e.g., ACLs) from source to destination S3.\n - Real-time capture and propagation of bucket/object changes.\n - Routing of S3 requests to different storages based on user-defined rules.\n - Reduce downtime up to zero for switching to different S3 provider.\n\nListed features can be configured per S3 user and per bucket with [management CLI](./tools/chorctl), [REST](https://petstore.swagger.io/?url=https://raw.githubusercontent.com/clyso/chorus/refs/heads/main/proto/gen/openapi/chorus/chorus.swagger.json)/[gRPC](./proto/chorus/chorus.proto) API, or [WebUI](./ui/).\n\n## Components\n[Chorus S3 Proxy](./service/proxy) service responsible for routing S3 requests and capturing data change events. \n[Chorus Agent](./service/agent) can be used as an alternative solution for capturing events instead of Proxy.\n[Chorus Worker](./service/worker) service does actual data replication. Worker also hosts management API so it is a central and the only required component to start with Chorus.\nCommunication between Proxy/Agent and Worker is done over work queue. \n[Asynq](https://github.com/hibiken/asynq) with [Redis](https://github.com/redis/redis) is used as a work queue.\n\n![diagram.png](./docs/media/diagram.png)\n\nFor more details, see:\n- [Proxy](./service/proxy)\n- [Worker](./service/worker)\n- [Agent](./service/agent)\n- [Management CLI](./tools/chorctl)\n- [Web UI](./ui)\n- [Standalone](./service/standalone) - all-in-one binary. Local playground with zero dependencies.\n\n## Documentation\n\n- Documentation available at [docs.clyso.com](https://docs.clyso.com/docs/products/chorus/overview).\n- Project [Blog](https://docs.clyso.com/blog/tags/chorus/).\n\n## Quick start\n\nFor a hands-on introduction, use the [docker-compose](./docker-compose) example. It includes a step-by-step README and sample configurations—ideal if you’re familiar with Docker and S3.\n\nIf using containers is not an option, try to run [standalone](./service/standalone) binary. It bundles all docker-compose services into one executable, letting you follow the same steps without containers.\n\n## Installation\n\n### From source\n\nRequires [Go language](https://go.dev/doc/install). Clone and build with:\n```shell\ngit clone https://github.com/clyso/chorus.git \u0026\u0026 cd chorus\n\n# Run worker\ngo run ./cmd/worker\n\n# Build worker binary\ngo build ./cmd/worker\n./worker\n\n# Run with custom config\ngo run ./cmd/worker -config \u003cpath-to-worker.yaml\u003e\ngo run ./cmd/proxy -config \u003cpath-to-proxy.yaml\u003e\ngo run ./cmd/agent -config \u003cpath-to-agent.yaml\u003e\n```\n\nOr install globally:\n```shell\n# Ensure $GOPATH/bin is in $PATH\nexport PATH=$PATH:$GOPATH/bin\n\n# Install binaries\ngo install github.com/clyso/chorus/cmd/worker@latest\ngo install github.com/clyso/chorus/cmd/proxy@latest\ngo install github.com/clyso/chorus/cmd/agent@latest\ngo install github.com/clyso/chorus/cmd/chorus@latest\n\n# Run with config\nworker -config \u003cpath-to-worker.yaml\u003e\n```\n\n### Prebuilt Binaries\nDownload binaries for Linux, Windows, and macOS from [GitHub releases](https://github.com/clyso/chorus/releases) page.\n\n### Docker\n\nMulti-platform images are available at:\n- `harbor.clyso.com/chorus/proxy`\n- `harbor.clyso.com/chorus/worker`\n- `harbor.clyso.com/chorus/agent`\n- `harbor.clyso.com/chorus/web-ui`\n\nExample:\n```shell\ndocker run -v /path/to/worker.yaml:/bin/config/config.yaml harbor.clyso.com/chorus/worker\n```\n\nBuilt from [Dockerfile](./Dockerfile) and [ui/Dockerfile](./ui/Dockerfile).\n\nSee also [docker-compose](./docker-compose) example.\n\n### Kubernetes\nDeploy with Helm:\n```shell\nhelm install \u003crelease name\u003e oci://harbor.clyso.com/chorus/chorus\n```\nThe chart source is located in [deploy/chorus](./deploy/chorus).\n\n## Develop\n\nChorus is written in Go—no other dependencies are required to build, run, or test.\n\nWant to contribute? See our [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n### Test\n[Test](./test) package contains e2e tests for replications between S3 storages.\nIt starts:\n- 3 embedded [gofakes3](https://github.com/johannesboyne/gofakes3) S3 storages\n- embedded Redis [miniredis](https://github.com/alicebob/miniredis)\n- Chorus Worker and Proxy services\n\nAll listed tools are written in go so test can be run without external dependencies just by:\n```shell\ngo test ./test/...\n```\n\nTests emulate real-world scenarios like:\n- replication of existing buckets and objects with metadata\n- capturing and syncing live bucket/object changes during migration\n- switching between S3 endpoints with and without downtime\n- access S3 with Chorus Proxy\n- data migration integrity check\n\n### Add APIs\nChorus worker [implements gRPC server](./pkg/api/) defined in [proto/chorus/chorus.proto](./proto/chorus/chorus.proto). It also provides REST API which is automatically generated from the proto file with [grpc-gateway](https://github.com/grpc-ecosystem/grpc-gateway). GRPC\u003c-\u003eREST mappings are defined in [proto/http.yaml](./proto/http.yaml).\n\nSteps to add new API:\n1. Add new service or endpoint definition to [proto/chorus/chorus.proto](./proto/chorus/chorus.proto)\n2. Add corresponding mapping to [proto/http.yaml](./proto/http.yaml)\n3. Run `./proto/gen_proto.sh` to generate go client, server, and openapi definitions to [proto/gen](./proto/gen) directory.\n4. Implement new service or endpoint in [pkg/api](./pkg/api/)\n5. Support new API in [chorctl](./tools/chorctl) and [WebUI](./ui) clients.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclyso%2Fchorus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclyso%2Fchorus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclyso%2Fchorus/lists"}