{"id":30576810,"url":"https://github.com/instaclustr/cassandra-parquet-transformer","last_synced_at":"2025-08-29T01:46:06.014Z","repository":{"id":309935462,"uuid":"999243786","full_name":"instaclustr/cassandra-parquet-transformer","owner":"instaclustr","description":"Transform SSTables from Apache Cassandra to Parquet or Avro files, locally or remotely via Apache Cassandra Sidecar","archived":false,"fork":false,"pushed_at":"2025-08-27T15:50:36.000Z","size":98,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-27T20:38:29.522Z","etag":null,"topics":["analytics","apache","apache-cassandra","avro","big","cassandra","data","parquet","spark","sstable","transformation"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/instaclustr.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-10T00:57:05.000Z","updated_at":"2025-08-27T15:50:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"7c46b441-22d6-4c3c-9c3f-0959d6325259","html_url":"https://github.com/instaclustr/cassandra-parquet-transformer","commit_stats":null,"previous_names":["instaclustr/cassandra-parquet-transformer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/instaclustr/cassandra-parquet-transformer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Fcassandra-parquet-transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Fcassandra-parquet-transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Fcassandra-parquet-transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Fcassandra-parquet-transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/instaclustr","download_url":"https://codeload.github.com/instaclustr/cassandra-parquet-transformer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Fcassandra-parquet-transformer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272604152,"owners_count":24963320,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","apache","apache-cassandra","avro","big","cassandra","data","parquet","spark","sstable","transformation"],"created_at":"2025-08-29T01:46:00.829Z","updated_at":"2025-08-29T01:46:05.982Z","avatar_url":"https://github.com/instaclustr.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"== SSTable Transformer\n\nThis tool transforms Cassandra's SSTable(s) to Parquet or Avro file(s). SSTables can be stored locally\nor it is possible to stream them remotely directly from a cluster. Spark integration is implemented as well.\n\n=== Build\n\nThe source code compiles to Java 11.\n\nThis will create \"bundled\" JAR artifact with all dependencies.\n\n----\n./mvnw clean install\n----\n\nThis will leave out all dependencies which are already in Spark so we do not need to bundle them.\n\n----\n./mvnw clean install -Pspark\n----\n\nThis will build RPM and Debian packages for bundled artifact.\n\n----\n./mvnw clean install -Pdefault,deb,rpm\n----\n\nWhen packages are installed, there will be `/usr/local/bin/sstable-transformer` script which\ninvokes the tool.\n\n=== Usage\n\nThe tool has two subcommands for now, called `transform` and `partitions`.\n\n----\n$ java -jar target/sstable-transformer-1.0.0-bundled.jar help\nUsage: transformer [-hV] [COMMAND]\nTransforms Cassandra SSTable to Parquet or Avro file\n  -h, --help      Show this help message and exit.\n  -V, --version   Print version information and exit.\nCommands:\n  transform   Transform SSTables to Parquet or Avro files.\n  partitions  Realize how many Spark partitions your Cassandra ring consists of.\n  help        Display help information about the specified command.\n----\n\nThe best explanation will be an example.\n\nImagine you have a directory with SSTables from Cassandra. Let's say that you have two SSTables.\n\nYou can look at the problem of \"transformation of SSTables\" in two ways:\n\n* You want to transform _each SSTable into one Parquet file_.\n* You want to transform _all SSTables into one Parquet file_.\n\nIf you transform _each SSTable into a Parquet file_, that means that you will end up with two Parquet files.\nOn the other hand, if you transform all SSTables into one Parquet file, two SSTables will be basically\nreduced into one Parquet file, obviously.\n\nThere are consequences for each transformation strategy.\n\nImagine we have a table like this:\n\n----\nCREATE TABLE spark_test.test3 (\n    id int PRIMARY KEY,\n    col1 int,\n    col2 int\n)\n----\n\nand you insert data in this fashion:\n\n----\ncqlsh\u003e INSERT INTO spark_test.test3(id , col1 ) VALUES ( 1, 10);\nshell\u003e nodetool flush\ncqlsh\u003e INSERT INTO spark_test.test3(id , col2 ) VALUES ( 1, 20);\nshell\u003e nodetool flush\n----\n\nBecause we flushed, we end up with 2 SSTables on disk. Due to the nature of the transformation (explained later on),\nif we transform these 2 SSTables into one Parquet file, the data in Parquet file will be effectively compacted,\nthey would be presented to you the same way as if you selected data in Cassandra:\n\n----\nid|col1|col2\n1 |  10|  20\n----\n\nOn the other hand, if you transformed each SSTable to a Parquet file, they will be just transformed and that's it.\nThere will be two Parquet files with this content:\n\nthe first Parquet file:\n\n----\nid|col1|col2\n 1|  10|null\n----\n\nthe second Parquet file:\n\n----\nid|col1|col2\n 1|null|  20\n----\n\nThe strategy of the transformation is specified via `--strategy` option, and it can have two values:\n\n* `ONE_FILE_ALL_SSTABLES`\n* `ONE_FILE_PER_SSTABLE`\n\nThe default is `ONE_FILE_PER_SSTABLE` when not specified,\nso you will have as many Parquet files as many input SSTables there are.\n\nFor compression of Parquet files, there is a support of `UNCOMPRESSED`, `SNAPPY` and `ZSTD`.\nThis option, `--compression` , is just pushed down to a Parquet writer and data are compressed accordingly.\n\nSame compression algorithms are available for Avro output format.\n\nExhaustive enumeration of all parameters is located at the end of this document.\n\n=== Remote and Local Transformation\n\nWe can process data which are co-located with this tool - locally on disk,\nor we can process data directly from Cassandra.\n\nFor the former case, we expect SSTables to be in a directory, or we can enumerate\nthem one by one. We expect that whole SSTables will be present even though\nwe are checking the presence of data files only (`-Data.db` component).\n\nSSTables or directories with them to process are specified with flag `--input`.\n\nFor transforming data remotely, we do that when `--input` is not present, and `--sidecar` option is specified\n(which we can repeat). By doing so,\nwe point this tool to https://github.com/apache/cassandra-sidecar[Apache Cassandra Sidecar] which might in practice\nrun on a different host. Sidecar instance(s) will read data from disk of a node Cassandra runs at, and it will\nstream data via HTTP to Spark Cassandra Analytics where we will iterate over them (over Spark's `InternalRow`)\nand we will transform them to Parquet data. This iteration occurs also in case of local processing as `SparkRowIterator`\nfrom Cassandra Analytics is agnostic to what `DataLayer` implementation is used with it.\n\nFor remote transformation, we expect Spark partitions to process to be specified. This is done via `--partitions` flag.\n\nBy default, all partitions are processed. In case of a need to process data in a more granular manner,\na user needs to specify particular partitions:\n\n----\n--partitions=0      - this will process only partition 0\n--partition=0,1,2,3 - this will process partitions 0, 1, 2 and 3\n--partitions=0..10  - this will process parititions from 0 to 10 (both included)\n----\n\nIt is important to realize that partitions are internally mapped to ranges. Each range is owned by a respective Cassandra node.\nHence, for successful transformation of remote data, we need sidecars which can read local data of each Cassandra node.\n\nFor example, if we have a cluster consisting of three nodes, each on a physically distinct machine while each node\nwould have 16 tokens (together 48), this would map to Spark 48 partitions. Each partition would be then mapped to\nspecific token range. A token range is owned by a specific Cassandra node (as a primary replica), so streaming\nwould occur from that Sidecar while backup replicas would be used in case of node's unavailability.\n\nIf there is a necessity to create Parquet files which contain at maximum certain number of rows, this can be\nspecified by `--max-rows-per-file` option.\n\n=== Internal details\n\nThis utility uses https://github.com/apache/cassandra-analytics[Apache Cassandra Analytics subproject]\nand its https://github.com/apache/cassandra-analytics/blob/trunk/cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/data/DataLayer.java[DataLayer] abstraction.\n\nFor the purposes of the local transformation, we are using `LocalDataLayer` which will be looking at local disk\nwhen processing SSTables. For remote transformation, we are using `CassandraDataLayer`.\n\nNext, we put either data layer implementation to Analytics' `SparkRowIterator` which will transparently\nread data from supplied SSTables over which we iterate, one row at a time.\n\nWe can use both transformation strategies while using `LocalDataLayer` because it has\ndirect access to the disk. However, we can use only `ONE_FILE_ALL_SSTABLES` for\n`CassandraDataLayer` (remote processing) because we can not remotely point Sidecar to\none SSTable only as this is all hidden behind `SparkRowIterator`. You can further specify maximum amount\nof rows in one file, even with `ONE_FILE_ALL_SSTABLES` option, so data are split into multiple files, each having\nmaximum number rows.\n\n`SparkRowIterator` is encapsulating all complexity when it comes to the compaction and reading from multiple SSTables\nso it seems like we are just getting one continuous stream of rows which are already compacted\ninternally.\n\nThen, we create a `ParquetWriter`. `ParquetWriter` needs an Avro schema. This schema is obtained by calling\nSpark's `SchemaConverters.toAvroType`, where its first argument, `StructType`, is internally constructed by `DataLayer` from `--create-table-statement` we supplied to the tool.\n\nFor now, only simple / primitive data types are supported.\n\n=== Examples\n\nNOTE: Please use the tool with the bellow JDK options for `java` command, they are not showed in the examples for brevity,\nyou can also see how it is used in `./run.sh` script.\n\n----\n-DSKIP_STARTUP_VALIDATIONS=true\n-Dfile.encoding=UTF-8\n-Djdk.attach.allowAttachSelf=true\n--add-exports java.base/jdk.internal.misc=ALL-UNNAMED\n--add-exports java.base/jdk.internal.ref=ALL-UNNAMED\n--add-exports java.base/sun.nio.ch=ALL-UNNAMED\n--add-exports java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED\n--add-exports java.rmi/sun.rmi.registry=ALL-UNNAMED\n--add-exports java.rmi/sun.rmi.server=ALL-UNNAMED\n--add-exports java.sql/java.sql=ALL-UNNAMED\n--add-opens java.base/java.lang.module=ALL-UNNAMED\n--add-opens java.base/jdk.internal.loader=ALL-UNNAMED\n--add-opens java.base/jdk.internal.ref=ALL-UNNAMED\n--add-opens java.base/jdk.internal.reflect=ALL-UNNAMED\n--add-opens java.base/jdk.internal.math=ALL-UNNAMED\n--add-opens java.base/jdk.internal.module=ALL-UNNAMED\n--add-opens java.base/jdk.internal.util.jar=ALL-UNNAMED\n--add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED\n----\n\n===== Examples of remote transformation via Sidecar\n\nSSTables remotely on Cassandra nodes, streamed by Sidecars.\n\n----\njava -jar target/sstable-transformer-1.0.0-bundled.jar transform \\\n  --keyspace=spark_test \\\n  --table=test3 \\\n  --output=\"/tmp/output-dir\" \\\n  --sidecar sidecar-node-1:9043 \\\n  --sidecar sidecar-node-2:9043 \\\n  --sidecar sidecar-node-3:9043\n----\n\n===== Examples of local transformation (SSTables present on local disk)\n\n----\njava -jar target/sstable-transformer.jar transform \\\n  --create-table-statement='CREATE TABLE spark_test.test3 (id int PRIMARY KEY, col1 int, col2 int)' \\\n  --strategy=ONE_FILE_PER_SSTABLE \\\n  --input=/tmp/input-dir-with-sstables \\\n  --output=/tmp/output-dir\n----\n\nSpecifying a directory for `--input`, each SSTable in it will be transformed to one Parquet file\nunder random name, e.g. `/tmp/output-dir/4882f391-ddfb-45db-94e2-4e9499212ace.parquet`\n\nWhen `--output=/tmp/my-transformation.parquet`, all SSTables will be transformed into one Parquet file\ncalled like that.\n\n----\njava -jar target/sstable-transformer.jar transform \\\n  --create-table-statement='CREATE TABLE spark_test.test3 (id int PRIMARY KEY, col1 int, col2 int)' \\\n  --strategy=ONE_FILE_ALL_SSTABLES \\\n  --input=/tmp/input-tables \\\n  --output=/tmp/my-transformation.parquet\n----\n\n=== Transformation of SSTables by Spark\n\nIt is possible to use this tool together with Spark. First, you need to build the tool with `spark` profile.\nNext we need to realize how many partitions there are in your Cassandra cluster. Yo do this either\nby following code:\n\n----\nval partitions = CassandraPartitionsResolver.partitions(\"dc1\", \"spark_test\", \"spark-master-1\", 9043).toSeq\n----\n\nor you can invoke this tool like this:\n\n----\n$ java -jar target/sstable-transformer-1.0.0-bundled.jar partitions \\\n    --dc=dc1 \\\n    --keyspace=spark_test \\\n    --sidecar=sidecar-1:9043\n----\n\nIf you know all your partitions, you can start to build transformer options. You can re-use this builder and\npass a partition to it in `map`. The end result is that Spark will parallelize our `partitions` and it will\nrun one _transformation_ on that particular partition. That will invoke Transformer on a Spark worker while\nprocessing a particular partition it was assigned to.\n\nThis will effectively invoke _remote transformation_ on a Spark worker by reading data from a cluster for given\npartition, and it will store transformed Parquet files into `/data/transformed` directory (where all Spark workers will\nbe storing their output Parquet files as well.)\n\n----\nval options = new PartitionResolverOptions\noptions.sidecar = \"sidecar-1:9043\"\noptions.dc = \"dc1\"\noptions.keyspace = \"ks\"\noptions.rf = 3\n\nval partitions = new CassandraPartitionsResolver(options).getPartitions.toSeq\n\nval builder = new TransformerOptions.Builder()\n    .keyspace(\"ks\")\n    .table(\"test\")\n    .maxRowsPerFile(100000)\n    .output(\"/data/transformed\")\n    .outputFormat(TransformerOptions.OutputFormat.PARQUET)\n    .sidecar(\"sidecar-1:9043\")\n    .sidecar(\"sidecar-2:9043\")\n    .sidecar(\"sidecar-3:9043\")\n\n// we parallelize partitions, each one will transform respective data to Parquet files\n// \"files\" will contain all Parquet files with transformed data\n\nval files = sc.parallelize(partitions, 6).map(p =\u003e {\n    new SSTableTransformer(builder.partition(p).build())\n        .runTransformation()\n        .asScala.toList.map(t =\u003e t.getPath)\n}).collect().flatten\n----\n\n=== Debian and RHEL packages\n\nYou can produce deb and rpm packages by `-Pdeb` and `-Prpm` profile respectively.\n\n==== Complete help\n\n----\n $ java -jar target/sstable-transformer-1.0.0-bundled.jar transform help\nUsage: transformer transform [--bloom-filter] [--keep-snapshot] [--sorted]\n                             [--compression=\u003ccompression\u003e]\n                             [--create-table-statement=\u003ccreateTableStmt\u003e]\n                             [--keyspace=\u003ckeyspace\u003e]\n                             [--max-rows-per-file=\u003cmaxRowsPerFile\u003e]\n                             --output=\u003coutput\u003e [--output-format=\u003coutputFormat\u003e]\n                             [--parallelism=\u003cparallelism\u003e]\n                             [--partitions=\u003cpartitions\u003e]\n                             [--strategy=\u003ctransformationStrategy\u003e]\n                             [--table=\u003ctable\u003e] [--input=\u003cinput\u003e...]...\n                             [--sidecar=\u003csidecar\u003e...]... [COMMAND]\nTransform SSTables to Parquet or Avro files.\n      --bloom-filter       Flag for telling whether bloom filter should be used\n                             upon writing of a Parquet file.\n      --compression=\u003ccompression\u003e\n                           Use compression for output files, it can be\n                             UNCOMPRESSED, SNAPPY, ZSTD.\n      --create-table-statement=\u003ccreateTableStmt\u003e\n                           CQL statement as for table creation. You do not need\n                             to specify it for remote data layer.\n      --input=\u003cinput\u003e...   List of directories or individual files to\n                             transform. Directories can be mixed with files.\n                             You do not need to specify it if you specify\n                             --sidecar\n      --keep-snapshot      Flag for telling whether we should keep snapshot\n                             used for remote transformation.\n      --keyspace=\u003ckeyspace\u003e\n                           Cassandra keyspace name. You do not need to specify\n                             it for local data layers.\n      --max-rows-per-file=\u003cmaxRowsPerFile\u003e\n                           Maximal number of rows per file. Has to be bigger\n                             than 0. Defaults to undefined which will put all\n                             rows to one file.\n      --output=\u003coutput\u003e    Output file or destination\n      --output-format=\u003coutputFormat\u003e\n                           Output format of data, either AVRO or PARQUET\n      --parallelism=\u003cparallelism\u003e\n                           Number of transformation tasks to run\n                             simultaneously. Defaults to number of processors.\n      --partitions=\u003cpartitions\u003e\n                           Spark partitions to process. Can be a number, a\n                             range (n..m), or enumeration (1,2,3...). Defaults\n                             to all partitions.\n      --sidecar=\u003csidecar\u003e...\n                           List of sidecar hostnames with ports.\n      --sorted             Flag for telling whether rows in each file should be\n                             sorted or not. Use with caution as sorting will\n                             happen in memory and all Spark rows will be held\n                             in memory until sorting is done. For large\n                             datasets, use this flag together with\n                             --max-rows-per-file so sorting will be limited to\n                             number of rows per that option only.\n      --strategy=\u003ctransformationStrategy\u003e\n                           Whether to convert all SSTables into one file or\n                             there will be one output file per SSTable. Can be\n                             one of ONE_FILE_PER_SSTABLE,\n                             ONE_FILE_ALL_SSTABLES. Defaults to\n                             ONE_FILE_PER_SSTABLE - can not be used when\n                             --sidecar is specified.\n      --table=\u003ctable\u003e      Cassandra table name. You do not need to specify it\n                             for local data layer.\nCommands:\n  help  Display help information about the specified command.\n----\n\n----\n java -jar target/sstable-transformer-1.0.0-bundled.jar help partitions\nUsage: transformer partitions --dc=\u003cdc\u003e --keyspace=\u003ckeyspace\u003e --rf=\u003crf\u003e\n                              --sidecar=\u003csidecar\u003e [COMMAND]\nRealize how many Spark partitions your Cassandra ring consists of.\n      --dc=\u003cdc\u003e\n      --keyspace=\u003ckeyspace\u003e\n      --rf=\u003crf\u003e\n      --sidecar=\u003csidecar\u003e\nCommands:\n  help  Display help information about the specified command.\n----","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finstaclustr%2Fcassandra-parquet-transformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finstaclustr%2Fcassandra-parquet-transformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finstaclustr%2Fcassandra-parquet-transformer/lists"}