{"id":15056908,"url":"https://github.com/instaclustr/icarus","last_synced_at":"2025-07-14T06:33:29.765Z","repository":{"id":56130123,"uuid":"255634369","full_name":"instaclustr/icarus","owner":"instaclustr","description":"Sidecar for Cassandra with integrated backup / restore","archived":false,"fork":false,"pushed_at":"2024-11-13T08:23:05.000Z","size":398,"stargazers_count":10,"open_issues_count":4,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-10T05:03:39.386Z","etag":null,"topics":["backup","car","cassandra","database","db","devops","java","jmx","kubernetes","netapp-public","operations","ops","remote","rest","restore","side","sidecar"],"latest_commit_sha":null,"homepage":"https://instaclustr.com","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/instaclustr.png","metadata":{"files":{"readme":"README.adoc","changelog":"CHANGES-icarus.txt","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-14T14:35:50.000Z","updated_at":"2024-11-20T06:33:52.000Z","dependencies_parsed_at":"2023-12-02T22:31:10.764Z","dependency_job_id":"f22d17a6-3d23-46b4-9d66-f070bd40ea28","html_url":"https://github.com/instaclustr/icarus","commit_stats":{"total_commits":139,"total_committers":3,"mean_commits":"46.333333333333336","dds":0.3453237410071942,"last_synced_commit":"1403dc9af3fc7a0a70e8914362dc8607d43f3c53"},"previous_names":["instaclustr/cassandra-sidecar","instaclustr/instaclustr-icarus"],"tags_count":29,"template":false,"template_full_name":null,"purl":"pkg:github/instaclustr/icarus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Ficarus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Ficarus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Ficarus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Ficarus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/instaclustr","download_url":"https://codeload.github.com/instaclustr/icarus/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Ficarus/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265252828,"owners_count":23735086,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","car","cassandra","database","db","devops","java","jmx","kubernetes","netapp-public","operations","ops","remote","rest","restore","side","sidecar"],"created_at":"2024-09-24T21:58:05.447Z","updated_at":"2025-07-14T06:33:29.276Z","avatar_url":"https://github.com/instaclustr.png","language":"Java","readme":"= Instaclustr Icarus\n\nimage:https://img.shields.io/maven-central/v/com.instaclustr/icarus.svg?label=Maven%20Central[link=https://search.maven.org/search?q=g:%22com.instaclustr%22%20AND%20a:%22icarus%22]\nimage:https://circleci.com/gh/instaclustr/icarus.svg?style=svg[\"Instaclustr\",link=\"https://circleci.com/gh/instaclustr/icarus\"]\n\n_Cloud-ready and Kubernetes-enabled Sidecar for Apache Cassandra_\n\nimage::Icarus.png[Icarus,width=50%]\n\n- Website: https://www.instaclustr.com/\n- OpenAPI Spec: https://instaclustr.github.io/instaclustr-icarus-go-client/\n- Documentation: https://www.instaclustr.com/support/documentation/\n\nThis repository is home of a sidecar for Apache Cassandra database.\nSidecar is meant to be run alongside of Cassandra instance\nand talks to Cassandra via JMX.\n\nSidecar JAR is also integrating https://github.com/instaclustr/esop[Esop] for backup and restore seamlessly\nso it might be used in two modes—either as a long-running server application which responds to clients requests, or as\na CLI application via which one can backup/restore a Cassandra node.\n\nNOTE: Icarus is used in https://github.com/instaclustr/cassandra-operator[Instaclustr Kubernetes operator] as well as in\nhttps://github.com/Orange-OpenSource/casskop[CassKop]—Kubernetes operator for Cassandra from Orange where it is used for backup and restore of clusters.\n\n== Overview\n\nIcarus is a server-like application which acts as a gateway to the Cassandra node it is logically coupled with. In theory,\nIcarus process can run wherever, but it needs to talk to a Cassandra node via JMX connection. If you do not want to expose\nJMX remotely, Icarus connects to `127.0.0.1:7199` by default. The other prerequisite for backup and restore operations\nis that Icarus has to be able to see the same directory structure as Cassandra does as it uploads/downloads SSTables and so on.\n\nIf you have a cluster of, let's say, 5 nodes, there should be one Icarus server running alongside each node.\n\nEach Icarus is independent and will execute operations only against the particular node it is configured to interact with.\nHowever, for backups and restores, we need to have a cluster-wide orchestration layer which deals with the complicated\nlogic of on-the-fly backup and restore for the whole cluster. This mode of operation is called \"global\". When a backup request\nis sent to one of sidecars, whichever Icarus does the job will act as a _coordinator_ of that operation.\n\nThe job of a coordinator Icarus is to disperse individual requests to each node in a cluster (and to itself as well), and\ncontinually pull for results and advance the execution once all nodes finish their _phases_. Backup and restore only advances to the other _phase_ when all nodes have finished the current phase\na coordinator orchestrates. After all phases are passed successfully, the coordinator's operation transitions to completed state.\n\nCurrently, the coordination/orchestration layer is done only for backup and restore operations as they require\nsuch behavior inherently. All other operations are, as of now, executed only against one particular sidecar and only that\nparticular Cassandra node it is configured to connect to is affected.\n\nNOTE: For global operation cases, it is important to realize that there is not any main Icarus; all Icaruses are equal in their role—\nonly if a global request is sent to one particular Icarus during backup or restore will it result in it being promoted to be a coordinator\nof a cluster-wide backup and restore operation.\n\nThe whole idea of how Icarus works in the case of backup/restore is depicted in the following image:\n\nimage::orchestration.svg[Icarus Workflow]\n\n* `c1`,`c2`,`c3` are Cassandra nodes, forming a ring\n* Client is a REST client, submitting operations (1) to Icarus, `s1`. In the image, it acts\nas a coordinator of the whole operation\n* Coordinator Icarus disperses modified requests (2) to other Icaruses, `s2` and `s3` (it sends a request to itself too)\n* each Icarus talks to Cassandra via JMX (3), executing its particular operation\n* `s1` knows that other nodes finished a phase because it is continually retrieving these operations and checks its statuses (4)\n* After orchestration is done and all phases are finished, client can retrieve its status (5). Icarus does not\ntry to send anything back to client on its own. We can only query it.\n\n== Build and Release\n\n[source,bash]\n----\n$ mvn clean install\n----\n\nIf you want to build rpm or deb package, you need to enable `rpm` and/or `deb` Maven profile.\n\nThere are built RPM and DEB packages for each GitHub release as well as Icarus JAR.\nWe are publishing each release to Maven central.\n\nIf you install RPM or DEB package, there is a bash script installed in `/usr/local/bin/icarus`\nvia which Icarus is interacted with. You can or course invoke Icarus via `java -jar icarus.jar`.\n\n== CLI Tool or Server?\n\nWhatever installation method you prefer, Icarus contains both an Esop and Icarus servers. The reason\nfor CLI being embedded into Icarus is purely practical—Icarus needs to be able\nto do backup/restores and this implementation logic is located in Esop, so Icarus\ndepends on this tool. Secondly, we do not need two separate JARs—this is particularly handy\nwhen Icarus is used in our Kubernetes operator for Cassandra for restoration purposes. It is\ninvoked like a CLI tool but for the sidecar container; it is invoked like a server application which\naccepts user's operation requests from the operator.\n\nNOTE: Icarus reuses Esop—that means that the configuration parameters for CLI of Esop have the same meaning for Icarus.\nInstead of feeding these flags via CLI in the case of Esop, we interact with Icarus via REST by sending a JSON body to the operations endpoint.\n\n[source,bash]\n----\n$ java -jar target/icarus.jar\nUsage: \u003cmain class\u003e [-hV] [COMMAND]\n  -h, --help      Show this help message and exit.\n  -V, --version   Print version information and exit.\nCommands:\n  esop    Application for backup and restore of a Cassandra node.\n  icarus  Sidecar management application for Cassandra.\n----\n\nTo execute Icarus:\n\n[source,bash]\n----\n$ java -jar target/icarus.jar icarus\n----\n\nTo execute help for Icarus:\n\n[source,bash]\n----\n$ java -jar target/icarus.jar icarus help\n----\n\n== Connecting to Cassandra\n\nBy default, Icarus uses `127.0.0.1:7199` for JMX connection to Cassandra. Please consult\nIcarus `help` command to learn how to secure it, or how to point it to another location.\n\nThe connection to Cassandra via JMX is done lazily; it is only initiated if it is needed and\nclosed afterwards, so each JMX interaction is a completely separate JMX connection.\n\n== REST Interface\n\nYou may consult (and try) all operations via our https://instaclustr.github.io/instaclustr-icarus-go-client/[Swagger spec].\n\nTo walk-though a reader via an operation lifecycle, let's explain the most \"complicated\" operations—backups and restores.\nAfter the explanation, a user will be able to do a backup and restore against a running Cassandra node (or Cassandra cluster)\nas well as being able to similarly submit and query any other operation.\n\nThe default address and port for REST is `127.0.0.1:4567`\n\n=== Backup Request Submission\n\nLet's say we have keyspaces `test1` and `test2` with tables `testtable1`,`testtable2`, and `testtable3` respectively\nwhich we want to make a backup of. In order to do that, we need to send a _backup operation_ to one of the sidecars.\n\nLet's say that we have three nodes, where hostname is `node1`,`node2`, and `node3` respectively and it resolves to IP.\n\nYou need to send the following JSON as body for `POST http://node1:4567/operations` with `application/json` `Content-Type` and `Accept` headers.\n\n[source,json]\n----\n{\n    \"type\": \"backup\",\n    \"storageLocation\": \"s3://my-s3-bucket\",\n    \"snapshotTag\": \"icarus-snapshot\",\n    \"entities\": \"system_schema,test1,test2\",\n    \"globalRequest\": true\n}\n----\n\nFirstly, we said that this operation is of type `backup`—each operation has to have this field with its own type.\n\nNext, we specify our bucket to backup the cluster to via `storageLocation`—we do not expose credentials in this request.\nFor example, the interaction with S3 has to be backed by some credentials.\nPlease refer to Esop documentation to learn how the credentials resolution for each supported cloud environment works.\n\nThirdly, we specify `snapshotTag` under which this backup will be taken and which we need to refer to upon restoration.\n\nAdditionally, we specify `entities` that tells us what keyspaces to actually backup. Icarus does not backup anything you\ndo not specify, so if you want to backup `system_schema` where table definitions are you have to do that yourself.\nIf you want to backup just one table (or only tables), you need to use `ks1.t1,ks2.t2` format.\n\nLast but not least, we say that this request is _global_ by setting `globalRequest: true`. Once this request\nis sent to `node1` Icarus, that instance will act as a coordinator of the cluster-wide backup. If we did not specify it,\nthere would be a backup of just that `node1` done.\n\nAfter submission of this operation, we may check what operations there are by calling `GET http://node1:4567/operations`\n(again JSON `Accept`).\n\n----\n[\n    {\n        \"type\": \"backup\",\n        \"id\": \"ce3014a7-6d5c-4bbd-a680-586f9be27435\",\n        \"creationTime\": \"2020-11-09T11:49:26.178Z\",\n        \"state\": \"COMPLETED\",\n        \"errors\": [],\n        \"progress\": 1.0,\n        \"startTime\": \"2020-11-09T11:49:26.224Z\",\n        \"storageLocation\": \"s3://my-s3-bucket/test-cluster/dc1/bf96d50b-bb7b-4493-9a2b-048f0fd354da\",\n        \"concurrentConnections\": 10,\n        \"metadataDirective\": \"COPY\",\n        \"cassandraDirectory\": \"/var/lib/cassandra\",\n        \"entities\": \"system_schema,test1,test2\",\n        \"snapshotTag\": \"icarus-snapshot-6f5e9841-4f97-3198-9398-161b445b3954-1604922565811\",\n        \"k8sNamespace\": \"default\",\n        \"k8sSecretName\": \"cloud-backup-secrets\",\n        \"globalRequest\": false,\n        \"timeout\": 5,\n        \"insecure\": false,\n        \"createMissingBucket\": false,\n        \"skipBucketVerification\": false,\n        \"schemaVersion\": \"6f5e9841-4f97-3198-9398-161b445b3954\",\n        \"uploadClusterTopology\": false,\n        \"completionTime\": \"2020-11-09T11:49:54.628Z\"\n    },\n    {\n        \"type\": \"backup\",\n        \"id\": \"d668d300-b28c-414d-a08e-4e41f6f0cdfc\",\n        \"creationTime\": \"2020-11-09T11:49:16.485Z\",\n        \"state\": \"COMPLETED\",\n        \"errors\": [],\n        \"progress\": 1.0,\n        \"startTime\": \"2020-11-09T11:49:16.491Z\",\n        \"storageLocation\": \"s3://my-s3-bucket\",\n        \"concurrentConnections\": 10,\n        \"metadataDirective\": \"COPY\",\n        \"cassandraDirectory\": \"/var/lib/cassandra\",\n        \"entities\": \"system_schema,test1,test2\",\n        \"snapshotTag\": \"icarus-snapshot-6f5e9841-4f97-3198-9398-161b445b3954-1604922565811\",\n        \"k8sNamespace\": \"default\",\n        \"k8sSecretName\": \"cloud-backup-secrets\",\n        \"globalRequest\": true,\n        \"timeout\": 5,\n        \"insecure\": false,\n        \"createMissingBucket\": false,\n        \"skipBucketVerification\": false,\n        \"schemaVersion\": \"6f5e9841-4f97-3198-9398-161b445b3954\",\n        \"uploadClusterTopology\": false,\n        \"completionTime\": \"2020-11-09T11:50:02.292Z\"\n    }\n]\n----\n\nWe see that our Icarus running on node `node1` (Cassandra runs there too, having same IP/hostname) is running two\nbackup operations. There are also other fields returned which the reader can consult by reading our Swagger spec.\nThe first fields returned are the same for every operation (`type`, `id`, `creationTime`, `state`, `errors`, `progress`, `startTime`).\n\nThe last operation is the global one (`globalRequest:true`). Internally, Icarus detected what the cluster topology is and\nit executed other, individual, backup request, to each node in a cluster. This fact is reflected in having two backup operations.\nIf you check closely, you see that `globalOperation` for the first operation is set to `false` and `storageLocation` is updated with\nname of a cluster, data center, and node id.\n\nThe similar response to the first operation would be found for the other two nodes too—`node2` and `node3`, (with specific\nfields updated to reflect each node).\n\nOnce an operation is finished or changes its state, we might see it from `state` field. Here, we see that such operation is `COMPLETED`.\nAn operation bumps over these states:\n\n* submitted—operation is submitted to be executed but it is not actively run\n* running—operation is executing its logic\n* completed—operation finished successfully\n* failed—operation finished with an error.\n\nProgress of an operation might be checked via `progress` field—it varies from `0.0` to `1.0`. If job has completed at\n80%, the progress would be `0.8`. It is up to each operation to update its progress.\n\nIf an operation fails, whatever exceptions are thrown, they are captured in `errors` field. `errors` is an array;\nthere might be 5 nodes and each might fail in its own way, so for each failure there would be a separate entry in `errors` array\nhaving different hostname—distinguishing the source of that error from each other.\n\n`snapshotTag` is updated as well—there is automatically appended host id as well as timestamp—timestamp is equal\nfor each node.\n\n=== Restore Request Submission\n\nAfter a cluster is backed up, we may want to restore it. The restoration logic is similar to the backup one when it comes to\na coordination/orchestration. If you want to restore a keyspace on a running Cassandra cluster, you just send one JSON\nto whatever node with this payload:\n\n[source,json]\n----\n{\n\t\"type\": \"restore\",\n\t\"storageLocation\": \"s3://my-s3-bucket/test-cluster/dc1/abcd\",\n\t\"snapshotTag\": \"icarus-snapshot\",\n\t\"entities\": \"test1\",\n\t\"globalRequest\": \"true\",\n\t\"restorationStrategyType\": \"HARDLINKS\",\n\t\"restorationPhase\": \"download\",\n\t\"import\": {\n\t\t\"type\": \"import\",\n\t\t\"sourceDir\": \"/var/lib/cassandra/downloadedsstables\"\n\t}\n}\n----\n\nAgain, `globalRequest` is `true`, `storageLocation` will be updated on node id, `test-cluster` and `dc1` needs to be there.\nOther fields are self-explanatory—you can consult Swagger spec to learn more about them.\n\nYou may similarly `GET` operations to see there are multiple individual  operations for each respective restoration _phase_.\n\nThere are restoration phases; these phases are passed _per node_, and each phase is initiated from coordinator only\nin case all nodes have passed that particular phase successfully.\n\n* `DOWNLOAD`—download SSTables from remote location locally\n* `IMPORT`—hardlinking (for Cassandra 3/4) or importing (for Cassandra 4 only) of SSTables from local destination of the respective node\n* `CLEANUP`—optionally (by default turned on) delete downloaded SSTables\n\nSSTables are truncated only after `DOWNLOAD` phase is over for all nodes.\n\n`import` field specifies where downloaded SSTables will be located before they will be hardlinked or imported.\n\nEach global restoration procedure has to start with `restorationPhase: download`. All other phases are handled automatically.\nThis is the advantage of Icarus. In the case of Esop, one would have to execute each phase individually. In Icarus' case, a coordinator\nwill take care of that transparently so in order to do a cluster-wide backup and restore on a running cluster, one just needs to\never send one JSON to whichever node and say it is a global request.\n\nNOTE: During the backup and restore operations, your whole cluster functions without any disruption.\nIt is even possible to backup and restore while your cluster is running, there is no downtime for both operations. Please\nconsult https://github.com/instaclustr/esop[Esop] readme to learn more about the nature of backups and restore\nand their configuration.\n\n== Anatomy of an Operation\n\nThe interaction with Icarus via REST is conceptually driven around _operations_.\n\nAn operation has these features:\n\n* it is created by calling `POST` on `/operations`\n* upon submit, it returns UUID which uniquely identifies it\n* such an operation may be checked for its _status_ and _progress_ via returned UUID on `GET /operations/{uuid}` endpoint\n* operation runs asychronously\n* in general, there might run multiple operations in parallel\n* you may query an operation via its UUID even after such operation has finished, regardless of sucess or error\n* finished operations from `GET /operations` endpoint expire, by default, after one hour\n\n=== What Operations and Endpoints Are Available?\n\nYou may consult (and try) all operations via our https://instaclustr.github.io/instaclustr-icarus-go-client/[Swagger spec].\n\n=== How to Create My Own Operation?\n\nTo implement an operation against a Cassandra node, you need to, in general, do the following steps. We\nwill guide a reader through _cleanup_ operation which is very easy to grasp. In this simple example,\nan operation consists of three classes:\n\n* `CleanupOperation`—the implementation of cleanup against Cassandra\n\n[source,java]\n----\n// each operation has to extend Operation class\n// each operation has its operation request\npublic class CleanupOperation extends Operation\u003cCleanupOperationRequest\u003e {\n\n    private final CassandraJMXService cassandraJMXService;\n\n    // Injection of necessary resources / instances\n    // for the interaction with Cassandra, you will very likely need JMX connection\n    // CassandraJMX service encapsulates this logic.\n    // request instance is injected there according to body in HTTP POST request upon\n    // operation submission\n    @Inject\n    public CleanupOperation(final CassandraJMXService cassandraJMXService,\n                            @Assisted final CleanupOperationRequest request) {\n        super(request);\n        this.cassandraJMXService = cassandraJMXService;\n    }\n\n    // for brevity, constructor for JSON deserialisation is not shown here\n    // please consult the source code to know more\n\n    @Override\n    protected void run0() throws Exception {\n        // implement your operation, you have access to request from this method\n        cassandraJMXService.doWithStorageServiceMBean(\n                new FunctionWithEx\u003cStorageServiceMBean, Integer\u003e() {\n                    @Override\n                    public Integer apply(StorageServiceMBean ssMbean) throws Exception {\n                        return ssMbean.forceKeyspaceCleanup(/* necessary arguments */);\n                    }\n                });\n    }\n}\n----\n\n* `CleanupOperationRequest`—this class represents your operation request sent to `POST /operations`. Feel free to\nmodel your operation request as you please, but `type` field has to be there. Each operation request\nis validated via Hibernate Validator so you may use `javax.validation` annotations on\nfields.\n\n[source,java]\n----\npublic class CleanupOperationRequest extends OperationRequest {\n\n    @NotEmpty\n    public final String keyspace;\n\n    public final Set\u003cString\u003e tables;\n\n    @Min(0)\n    public final int jobs;\n\n    @JsonCreator\n    public CleanupOperationRequest(\n            @JsonProperty(\"type\") String type,\n            @JsonProperty(\"keyspace\") String keyspace,\n            @JsonProperty(\"tables\") Set\u003cString\u003e tables,\n            @JsonProperty(\"jobs\") int jobs) {\n        this.jobs = jobs;\n        this.keyspace = keyspace;\n        this.tables = tables;\n        this.type = type;\n    }\n----\n\n* Finally, we need a Guice module which is installed upon Icarus start:\n\n[source,java]\n----\npublic class CleanupsModule extends AbstractModule {\n\n    @Override\n    protected void configure() {\n        installOperationBindings(binder(),\n                                 \"cleanup\",\n                                 CleanupOperationRequest.class,\n                                 CleanupOperation.class);\n    }\n}\n----\n\nPlease note:\n\n* this class extends Guice's `AbstractModule`\n* `installOperationBindings` static method is from `com.instaclustr.operations.OperationBindings`\n* `cleanup` is _operation type_. You have to specify `type` field in your JSON request. Each operation\nis uniquely known and refered to through this type.\n* Each operation has its implementation and its request—specified as other arguments to installation method.\n\nAfter we have our operation module, we need to _install it_. This happens\nvia `com.instaclustr.icarus.Icarus#operationModules` method.\n\n== Tests\n\nThere are _cloud tests_ which are primarily focused on backup and restore against a cloud destination—being\nGCP, Azure, or S3. Clouds tests are disabled by default. You have to enable them like this:\n\n[source,bash]\n----\n mvn clean install -PcloudTests \\\n  -Dawsaccesskeyid=_enter__ \\\n  -Dawssecretaccesskey=_enter_ \\\n  -Dgoogle.application.credentials=/path/to/gcp.json \\\n  -Dazurestorageaccount=_enter_ \\\n  -Dazurestoragekey=_enter_\n----\n\n== Usage\n\nPlease see https://www.instaclustr.com/support/documentation/announcements/instaclustr-open-source-project-status/ for Instaclustr support status of this project","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finstaclustr%2Ficarus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finstaclustr%2Ficarus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finstaclustr%2Ficarus/lists"}