{"id":21840529,"url":"https://github.com/rubiojr/charmedring","last_synced_at":"2025-04-14T10:52:15.633Z","repository":{"id":37747104,"uuid":"442258672","full_name":"rubiojr/charmedring","owner":"rubiojr","description":"A smart TCP proxy for CharmFS  files.","archived":false,"fork":false,"pushed_at":"2023-09-04T17:08:17.000Z","size":1389,"stargazers_count":4,"open_issues_count":3,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-28T00:05:54.341Z","etag":null,"topics":["charm","high-availability"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rubiojr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-27T19:50:27.000Z","updated_at":"2024-01-12T18:25:35.000Z","dependencies_parsed_at":"2024-06-19T22:53:20.402Z","dependency_job_id":"fd3c8faa-35e0-4cb1-a809-92dbe73b75d4","html_url":"https://github.com/rubiojr/charmedring","commit_stats":{"total_commits":39,"total_committers":3,"mean_commits":13.0,"dds":0.3846153846153846,"last_synced_commit":"d69e886d93377bc85b782baa6320dbc3fd51c9d6"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubiojr%2Fcharmedring","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubiojr%2Fcharmedring/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubiojr%2Fcharmedring/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubiojr%2Fcharmedring/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rubiojr","download_url":"https://codeload.github.com/rubiojr/charmedring/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248868838,"owners_count":21174754,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["charm","high-availability"],"created_at":"2024-11-27T21:26:37.384Z","updated_at":"2025-04-14T10:52:15.613Z","avatar_url":"https://github.com/rubiojr.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![](docs/images/charmed-ring.png)\n\n# Charmed 💍 Ring\n\nA smart TCP proxy to replicate and backup [Charm FS](https://charm.sh) files.\n\n## Overview\n\nCharmed Ring is an HTTP/SSH proxy that acts as a Charm server and replicates files uploaded with `charm fs` to a number (two or more) of backend Charm servers.\n\nThe proxy uses [consistent hashing](https://github.com/buraksezer/consistent) to replicate data among a number of configured servers.\n\n## Status\n\nCharmed Ring is currently a working prototype. Using the proxy to replicate important data is highly discouraged at the moment. Most of the work to measure performance, reliability and correctness still needs to happen.\nThe proxy should not interfere with other Charm functionality (KV, linking, etc), but it hasn't been tested extensively.\n\nFrom a security point of view, the proxy is stateless, and no JTW token manipulation happens when the HTTP requests traverse the proxy, so it should be relatively safe. SSH proxying happens at the TCP level, simply forwarding what the proxy receives from the client to the first configured Charm backend. Work to understand the security implications of having a proxy sitting between a Charm client and a server still needs to happen.\n\nBesides the fairly immature status of this proxy, the Charm ecosystem is young and quickly evolving, and this proxy may need to adapt introducing breaking changes. It's certainly possible upstream changes before the Charm ecosystem solidifies may render it useless eventually.\n\nTooling to quickly setup a development environment to evaluate the proxy functionality, [is provided](#testing).\n\n## Quick Start\n\nBoot a [local cluster](docs/sample-cluster.md) that includes a Charmed Ring proxy, an S3 server (minio) and 3 Charm servers.\n\n## Operational modes\n\nThe goal of the project is to provide automatic backups and/or high availability of your CharmFS data, so the proxy has three different operational modes: replication, backup and a mixed replication+backup mode.\n\n### Replication mode\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/images/replica-diag.png\" alt=\"replication\"/\u003e\n\u003c/p\u003e\n\nIn replication mode, the proxy tries to replicate every write operation (HTTP POST and DELETE) to **three** of the configured servers. A consistent hashing algorithm determines the backend servers that will receive the the client requests, based on the path of the file being uploaded. After **two successful replication attempts**, the server returns a successful response to the client. If less than 2 replication attempts succeeded, the proxy returns a 503 to the client, and the upload or delete attempt fails.\n\nThe first configured backend server is somewhat special, as it's the backend server chosen by the proxy to receive the SSH traffic. If the first backend server fails, the proxy will stop working correctly. The same happens for read (HTTP GET) requests (i.e. when retrieving files).\nThis will eventually change to provide high availability for reads also.\n\n### Backup mode\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/images/backup-diag.png\" alt=\"backup\" width=\"300px\"/\u003e\n\u003c/p\u003e\n\nIn backup mode, every write operation is replicated to an S3 bucket. If bucket versioning is enabled, you also get protection against accidental file change or removal, given that delete/update operations are also replicated.\n\n### Replication+backup mode\n\nCombines the first two modes. Files are sent to an S3 bucket and then replicated to the other Charm nodes.\n\n## Known limitations\n\n### Shared server state\n\nEvery Charm (backend) server part of the cluster needs to share server keys and users in the sqlite database.\n\nIt's currently not possible to replicate database and server keys to other charm servers automatically, which means that you should start with a single server, and once you have an account there, use a copy of the server data directory to spawn new charm servers.\n\nThe [script/dev](script/dev) is a naive attempt at setting up a cluster that can be used as a reference.\n\n### Single account server\n\nThe proxy currently proxies account creation and JTW token requests (via SSH) to the first configured backend. If that backend fails, the rest of the operations will fail.\n\n## Missing features\n\nThe following features will be available eventually, ordered by priority (high to low):\n\n* [X] Backup mode (🚧 WiP)\n* [X] Mixed replica+backup mode (🚧 WiP)\n* [ ] HTTP load balancing (🚧 WiP) for reads\n* [ ] Multiple SSH backends support ((🚧 WiP)\n* [ ] HTTP proxy speed optimizations\n* [ ] Tools to make Charm replica initialization easier\n* [ ] Auto healing (repairing missing replica blobs)\n* [ ] Proxy stats (backend usage, blob distribution, usage, errors, etc)\n* [ ] Prometheus exporter\n* [ ] Tools (Charm admin API?) to create and replicate charm accounts from CLI, when automatic server accounts has been disabled.\n* [ ] Optionally serve blobs from the remote S3 bucket if backend servers are not available (read-only mode)\n\n## Development\n\n### Building\n\nYou'll need Go installed.\n\n```\nmake\n```\n\n### Running the proxy\n\n```\ncring --host http://charm1:35354 --host http://charm2:35354 --host http://charm3:35354\n```\n\nThe `--host` argument needs to be repeated one for every backend server to be used (minimum of two).\n\n### Testing\n\n`script/integration-tests` can be used to run the tests.\n\n#### Setting up a development environment\n\nThe development environment sets up a Charmed Ring cluster with three Charm servers. Linux only.\n\nRequirements:\n\n* docker and docker-compose\n* [charm](https://github.com/charmbracelet/charm) CLI available in PATH, we'll use it to test the replication.\n\nRun the `script/dev` to setup the cluster:\n\n![](docs/images/dev-setup.png)\n\n\nThe cluster is now ready, let's copy this very same README to test it:\n\n```\nexport CHARM_HOST=localhost\ncharm fs cp README.md charm:README.md\n```\n\nYou can `docker-compose logs -f` to monitor cluster logs, it'll look something like this after a successful copy:\n\n![](docs/images/copy-logs.png)\n\nThe proxy needs at least 2 Charm servers to receive the file successfully to return a valid response to the client, it'll return a 503 to the charm client otherwise. We can use the cluster to test this behavior:\n\n```\nrubiojr@kraken:~/git/rubiojr/charmedring$ docker-compose stop charm3 charm2\nStopping charm2 ... done\nStopping charm3 ... done\nrubiojr@kraken:~/git/rubiojr/charmedring$ charm fs cp README.md charm:README.md\nError: server error: 503 Service Unavailable\nUsage:\n  charm fs cp [charm:]PATH [charm:]PATH [flags]\n\nFlags:\n  -h, --help        help for cp\n  -r, --recursive   copy directories recursively\n\nserver error: 503 Service Unavailable\n```\n\nTo destroy the dev cluster and associated data:\n\n```\nscript/clean-cluster\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frubiojr%2Fcharmedring","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frubiojr%2Fcharmedring","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frubiojr%2Fcharmedring/lists"}