{"id":22379140,"url":"https://github.com/sky-uk/cqlmigrate","last_synced_at":"2025-07-10T18:37:26.958Z","repository":{"id":45323576,"uuid":"43748284","full_name":"sky-uk/cqlmigrate","owner":"sky-uk","description":"Cassandra schema migration library","archived":false,"fork":false,"pushed_at":"2024-05-22T08:41:38.000Z","size":539,"stargazers_count":46,"open_issues_count":21,"forks_count":28,"subscribers_count":98,"default_branch":"master","last_synced_at":"2024-05-22T11:13:09.264Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sky-uk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-10-06T12:11:27.000Z","updated_at":"2024-05-27T12:47:26.201Z","dependencies_parsed_at":"2024-05-02T14:31:47.589Z","dependency_job_id":"53282a82-b760-43d8-af6c-50c41d77d1c8","html_url":"https://github.com/sky-uk/cqlmigrate","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/sky-uk/cqlmigrate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fcqlmigrate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fcqlmigrate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fcqlmigrate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fcqlmigrate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sky-uk","download_url":"https://codeload.github.com/sky-uk/cqlmigrate/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fcqlmigrate/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264631351,"owners_count":23640947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-04T23:09:05.865Z","updated_at":"2025-07-10T18:37:26.916Z","avatar_url":"https://github.com/sky-uk.png","language":"Java","funding_links":[],"categories":["Packages"],"sub_categories":["Tools"],"readme":"[![Build Status](https://travis-ci.com/sky-uk/cqlmigrate.svg?branch=master)](https://travis-ci.com/sky-uk/cqlmigrate)\n\n# Cassandra CQL migration tool\n\ncqlmigrate is a library for performing schema migrations on a cassandra cluster including:\n* [Apache Cassandra](https://cassandra.apache.org)\n* [DataStax Enterprise](https://www.datastax.com/products/datastax-enterprise) \n* [Astra DB](https://www.datastax.com/products/datastax-astra) \n* [AWS Keyspaces](https://aws.amazon.com/keyspaces/)\n\nIt is best used as an application dependency, but can also be used standalone.\n\n## Adding as a Gradle dependency\n\n```groovy\nrepositories {\n    mavenCentral()\n}\n\ncompile 'uk.sky:cqlmigrate:0.13.0'\n```\n\n## Adding as a Maven dependency\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003euk.sky\u003c/groupId\u003e\n  \u003cartifactId\u003ecqlmigrate\u003c/artifactId\u003e\n  \u003cversion\u003e0.13.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n## Cassandra Prerequisites\n\nThe locks keyspace and table needs to be created before running any migrations.\n\n    CREATE KEYSPACE IF NOT EXISTS cqlmigrate WITH replication = {'class': 'REPLICATION_CLASS', 'replication_factor': REPLICATION_FACTOR };\n    CREATE TABLE IF NOT EXISTS cqlmigrate.locks (name text PRIMARY KEY, client text);\n\n## Library usage\n\nTo apply all `.cql` files located in `/cql` in the classpath:\n\n```java\nimport com.datastax.oss.driver.api.core.*;\n\n// Configure locking for coordination of multiple nodes\nCassandraLockConfig lockConfig = CassandraLockConfig.builder()\n        .withTimeout(Duration.ofSeconds(3))\n        .withPollingInterval(Duration.ofMillis(500))\n        .withConsistencyLevel(ConsistencyLevel.ALL)\n        .withLockKeyspace(\"cqlmigrate\")\n        .build();\n\n// Configure cql migrator\nCqlMigratorConfig cqlMigratorConfig = CqlMigratorConfig.builder()\n        .withLockConfig(lockConfig)\n        .withReadConsistencyLevel(ConsistencyLevel.LOCAL_ONE)\n        .withWriteConsistencyLevel(ConsistencyLevel.ALL)\n        .withTableCheckerInitDelay(Duration.ofSeconds(5))\n        .withTableCheckerTimeout(Duration.ofMinutes(1))\n        .build()\n\n// Create a Cassandra session for cassandra driver 4.x\nCqlSession session = CqlSession.builder()\n        .addContactPoints(cassandraHosts)\n        .withLocalDatacenter(\"datacenter1\")\n        .withAuthProvider(new ProgrammaticPlainTextAuthProvider(\"username\", \"password\"))\n        .build();\n\n// Create a migrator and run it\nCqlMigrator migrator = CqlMigratorFactory.create(lockConfig);\nPath schemas = Paths.get(ClassLoader.getSystemResource(\"/cql\").toURI());\nmigrator.migrate(session, \"my_keyspace\", asList(schemas));\n```\n\nThe migrator will look for a `bootstrap.cql` file for setting up the keyspace.\n\n## Standalone usage\n\n```sh\n$ java -Dhosts=localhost,192.168.1.1 -Dport=9042 -DlocalDC=DC1 -Dkeyspace=my_keyspace -Ddirectories=cql-common,cql-local -jar cqlmigrate.jar\n```\n\nSpecify credentials, if required, using `-Dusername=\u003cusername\u003e` and `-Dpassword=\u003cpassword\u003e`.\n\nSpecify optional properties, if required, using\n* `-Dprecheck=\u003ctrue/false\u003e` default `false` \n* `-DtableCheckerTimeout=\u003cduration\u003e` default `PT1M` (This was introduced for AWS Keyspaces as when new table is created it might take time to allocate resources and if additional changes or data are inserted while table is in `CREATING` state, it will fail. If cluster is no AWS Keyspaces it will not wait.)\n* `-DtableCheckerInitDelay=\u003cduration\u003e` default `PT5S` (Supplement previous option as it might take time AWs Keyspaces update their system so we need to wait before we check first time.)\n* `-DreadCL` default `LOCAL_ONE`\n* `-DwriteCL` default `ALL`\n\n## What it does\n\n1. Checks all nodes are up and their schemas are in agreement.\n\n2. Tries to acquire a lock for the keyspace. If it can't initially be acquired it will continue to retry at a set polling time until the timeout is reached.\n\n3. Looks for a `bootstrap.cql` file and runs it first. This file should contain the keyspace definition:\n\n    ```\n    CREATE KEYSPACE my_keyspace\n     WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1 };\n    ```\n\n4. Applies `.cql` files one by one, sorted by filename in ascending order. It is suggested to prefix\n   the files with a datetime to order them:\n\n   ```\n   /cql/2015-05-02-19:20-create-airplanes-table.cql\n   /cql/2015-05-03-14:19-add-manufacturer-column.cql\n   ```\n\n   Any previously applied files will be skipped. For AWS Keyspaces, it will wait after each successfully applied file for tables to get into ACTIVE state for initial \u003ctableCheckerInitDelay\u003e and maximum \u003ctableCheckerTimeout\u003e duration. \n\n5. Releases the lock.\n\n### schema_updates table\n\nThis table is used to determine what has been previously applied.\n\n    SELECT * FROM schema_updates;\n\n     filename                         | applied_on               | checksum\n    ----------------------------------+--------------------------+------------------------------------------\n               0001-create-tables.cql | 2015-04-08 12:10:04+0100 | ec19dfac7ede62b2a40c0f39706b237cd5c30da6\n                     0002-dataset.cql | 2015-04-08 12:10:04+0100 | 4fa2d6c4fae9950f0c9140ae2eb57fe689192b4a\n                0003-initial-date.cql | 2015-04-08 12:10:04+0100 | 19d0c9522b6464a06b18192c6e04233f83e78a84\n\n    (3 rows)\n\nIt also maintains a checksum to ensure the script hasn't changed since it was last applied.\n\n### locks keyspace and table\n\nThe locks keyspace replication class and factor can be configured using the LocksConfig.\nThis table is used to keep track of what locks are currently in place, and relies on\nCassandra's [lightweight transactions](https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_ltwt_transaction_c.html).\n\n    SELECT * FROM locks;\n\n     name                                | client\n    -------------------------------------+--------------------------------------\n     airplanes_keyspace.schema_migration | 2a4ec2ae-d3d1-4b33-86a9-eb844e35eeeb\n\n    (1 rows)\n\nEach lock will be deleted by `cqlmigrate` once the migration is complete.\n\n## Supported Cassandra versions\n\nThis project has been tested against the following versions:\n* DSE 5.1.18 (3.11.3)\n* Apache Cassandra 3.11.5\n* AWS Keyspaces\n\n## Caveats\n\nCassandra is an eventually consistent, AP database, and so applying schema updates are not as simple\nas a traditional relational database.\n\n* Certain schema changes can cause data corruption on cassandra. Be very careful changing a schema for a\n  table that is being actively written to. Generally, adding columns is safe (but be careful with\n  collection types). Test the migration before rolling out to production.\n\n* AP properties of Cassandra also apply to schema updates - so it is possible for a cluster to have an\n  inconsistent schema across nodes in case of split brain or other situation. `cqlmigrate` tries to\n  alleviate this with appropriate consistency levels.\n\n## Cql File Comments\n\nThere are a number of ways to add comments to your `cql` files. \n\nFor inline comments prepend `--` to your comment, e.g:\n\n    -- Select Queries\n    SELECT * FROM schema_updates;\n\nFor multiline comments wrap them with `/*` and `*/`, e.g:\n\n    /*\n        Added by John Smith\n        19th September 2017\n    */\n    SELECT * FROM schema_updates;\n\n# Contributors\n\nOriginally developed by the Cirrus team at Sky.\n\n- Adam Dougal\n- James Booth\n- James Ravn\n- Adrian Ng\n- Malinda Rajapakse\n- Ashutosh Gawande\n- Dominic Mullings\n- Yoseph Sultan\n- David Sale\n- Supreeth Rao\n- Jose Taboada\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsky-uk%2Fcqlmigrate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsky-uk%2Fcqlmigrate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsky-uk%2Fcqlmigrate/lists"}