{"id":15056736,"url":"https://github.com/tolbertam/sstable-tools","last_synced_at":"2025-04-10T04:34:18.744Z","repository":{"id":87998365,"uuid":"47861279","full_name":"tolbertam/sstable-tools","owner":"tolbertam","description":"Tools for parsing, creating and doing other fun stuff with sstables","archived":false,"fork":false,"pushed_at":"2017-07-26T02:02:28.000Z","size":230,"stargazers_count":163,"open_issues_count":11,"forks_count":31,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-02-02T19:51:28.963Z","etag":null,"topics":["cassandra","cqlsh","dse","sstable","sstables"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tolbertam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-12-12T02:39:40.000Z","updated_at":"2025-01-22T07:35:04.000Z","dependencies_parsed_at":"2023-05-22T08:00:33.797Z","dependency_job_id":null,"html_url":"https://github.com/tolbertam/sstable-tools","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolbertam%2Fsstable-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolbertam%2Fsstable-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolbertam%2Fsstable-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolbertam%2Fsstable-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tolbertam","download_url":"https://codeload.github.com/tolbertam/sstable-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239140104,"owners_count":19588338,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra","cqlsh","dse","sstable","sstables"],"created_at":"2024-09-24T21:55:57.075Z","updated_at":"2025-02-16T14:31:12.134Z","avatar_url":"https://github.com/tolbertam.png","language":"Java","funding_links":[],"categories":["Packages"],"sub_categories":["Tools"],"readme":"# SSTable Tools\n\n[![Build Status](https://travis-ci.org/tolbertam/sstable-tools.svg?branch=master)](https://travis-ci.org/tolbertam/sstable-tools) [![Github Releases (by Release)](https://img.shields.io/github/downloads/tolbertam/sstable-tools/v3.11.0-alpha11/total.svg)](https://github.com/tolbertam/sstable-tools/releases/tag/v3.11.0-alpha11)\n\nA toolkit for parsing, creating and doing other fun stuff with Cassandra 3.x SSTables. This project is under development and not yet stable. Mainly a proof of concept playground for wish items.\n\nPre compiled binary available:\n\n* [sstable-tools-3.11.0-alpha11.jar](https://github.com/tolbertam/sstable-tools/releases/download/v3.11.0-alpha11/sstable-tools-3.11.0-alpha11.jar) -  Supports 3.0 to 3.11 (ma, mb and mc sstable formats)\n\n**Note:** This tool formerly included a `tojson` command which dumped SSTable contents to JSON.  This functionality has\nsince been merged into Cassandra starting with versions 3.0.4 and 3.4 as the `sstabledump` command.\n\n**Features:**\n\n* [cqlsh](#cqlsh) - Drop into an interactive shell to make queries against SSTables.\n* [describe](#describe) - Describe SSTable data and metadata. [CASSANDRA-11483](https://issues.apache.org/jira/browse/CASSANDRA-11483)\n* [hints](#hints) - Dump hints from a hint file\n* [compact](#compact) - offline compaction\n\nExample usage:\n\n    java -jar sstable-tools.jar cqlsh\n    java -jar sstable-tools.jar compact ma-2-big-Data.db ma-3-big-Data.db /path/to/tabledir\n    java -jar sstable-tools.jar hints 1458779867606-1.hints\n    java -jar sstable-tools.jar describe ma-2-big-Data.db\n\nExample shell usage:\n\n    java -jar sstable-tools.jar cqlsh\n\nSelect one or more sstables (space delimited, or choose directory to include all)\n\n    cqlsh\u003e use ma-2-big-Data.db;\n    Using: /home/user/sstable-tools/ma-2-big-Data.db\n\nUse predefined schema file. Can view with 'schema'. This is optional but without it the partition key and clustering index names are unknown.\n\n    cqlsh\u003e schema schema.cql\n    Successfully imported schema from '/home/user/sstable-tools/schema.cql'.\n\nAlternatively, use 'CREATE TABLE' statement to enter a schema.\n\n    cqlsh\u003e CREATE TABLE users (\n    ...        user_name varchar PRIMARY KEY,\n    ...        password varchar,\n    ...        gender varchar,\n    ...        state varchar,\n    ...        birth_year bigint\n    ...    );\n\nDiscover the data in sstable(s) using CQL queries\n\n    cqlsh\u003e SELECT * FROM sstable WHERE age \u003e 1 LIMIT 1\n\n     ┌────────────┬─────────────┬─────────┬───────────┬────────┐\n     │user_name   │birth_year   │gender   │password   │state   │\n     ╞════════════╪═════════════╪═════════╪═══════════╪════════╡\n     │frodo       │1985         │male     │pass@      │CA      │\n     └────────────┴─────────────┴─────────┴───────────┴────────┘\n\nDisplay raw sstable data (useful to see tombstones and expired ttls) with optional where clause\n\n    cqlsh\u003e DUMP WHERE age \u003e 1 LIMIT 1\n\n    [frodo] Row[info=[ts=1455937221199050] ]:  | [birth_year=1985 ts=1455937221199050], [gender=male ts=1455937221199050], [password=pass@ ts=1455937221199050], [state=CA ts=1455937221199050]\n\nDescribe the sstable data and metadata\n\n    cqlsh\u003e describe sstables;\n\n    /Users/clohfink/git/sstable-tools/./src/test/resources/ma-2-big-Data.db\n    =======================================================================\n    Partitions: 1\n    Rows: 1\n    Tombstones: 0\n    Cells: 4\n    Widest Partitions:\n       [frodo] 1\n    Largest Partitions:\n       [frodo] 104 (104 B)\n    Tombstone Leaders:\n    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner\n    Bloom Filter FP chance: 0.010000\n    Size: 50 (50 B)\n    Compressor: org.apache.cassandra.io.compress.LZ4Compressor\n      Compression ratio: -1.0\n    Minimum timestamp: 1455937221199050 (02/19/2016 21:00:21)\n    Maximum timestamp: 1455937221199050 (02/19/2016 21:00:21)\n    SSTable min local deletion time: 2147483647 (01/18/2038 21:14:07)\n    SSTable max local deletion time: 2147483647 (01/18/2038 21:14:07)\n    TTL min: 0 (0 milliseconds)\n    ...[snip]...\n\nPaging is enabled by default and can be manipulated by using 'PAGING'\n\n    cqlsh\u003e PAGING 20;\n    Now Query paging is enabled\n    Page size: 20\n    cqlsh\u003e PAGING OFF;\n    Disabled Query paging.\n    cqlsh\u003e PAGING ON;\n    Now Query paging is enabled\n    Page size: 20\n\nUsed sstables, schema, and paging settings and persisted for future use. Use the 'PERSIST' command to view preferences and to enable/disable persistence.\n\n    cqlsh\u003e PERSIST;\n    Preferences are currently enabled:\n    pagingEnabled=true\n    pagingSize=20\n    preferencesEnabled=true\n    schema=\"\"\n    sstables=[\n        \"/home/user/sstable-tools/ma--big-Data.db\"\n    ]\n    cqlsh\u003e PERSIST OFF;\n    Disabled Preferences.\n\n**Note:** No environment configuration is necessary for this tool to work if all components of the sstable are available but the cql create statement allows for more details.\n\n\n## Building\n\nThis project uses [Apache Maven](https://maven.apache.org/) to build a\nself-contained executable jar.  To build the jar simply execute:\n\n```shell\nmvn package\n```\n\nThe executable jar will be present in the target directory.\n\n## cqlsh\ncql shell similiar and modeled after the C* cqlsh tool. Enables issuing cql queries against raw sstables and\nprovides additional diagnostic tools over them. Provides history (reverse searchable with ctrl-r) and autocomplete for ease of use.\n\n```text\nCommands:\n\nHELP               - prints this message\nEXIT               - leaves the shell\nCREATE TABLE ...   - A CREATE TABLE cql statement to use as metadata when reading sstables (HIGHLY RECOMMENDED!)\nDESCRIBE SCHEMA    - Show currently used schema (or serialized cfmetadata if generated)\nDESCRIBE SSTABLES  - Provide details and statistics on current sstable(s)\nPAGING [(ON|OFF)]  - Enables, disables, or shows current status of query paging.\nPAGING \u003cSIZE\u003e      - Enables paging and sets paging size.\nPERSIST [(ON|OFF)] - Enables, disables, or shows current status of persistence of settings state.\nSCHEMA [\u003cFILE\u003e]    - Imports a cql file as the active table schema or shows active user-defined schema.\nUSE                - update the sstable[s] used by default with select, dump, describe commands\n    USE /var/lib/cassandra/data/system/peers/ma-1-big-Data.db\n    or with multiple sstables separated with spaces. This can also be a directory which will add all sstables in it.\n    USE ma-1-big-Data.db ma-2-big-Data.db \"/home/path with space/db/data/sstables\"\n\nSELECT             - run a cql query against the current sstable (unless other specified)\n    SELECT * FROM sstables WHERE key \u003e 1 LIMIT 10\n    the keyword sstables will use the current sstable set with the USE command or set when running cqlsh. You can also\n    specify an sstable here\n    SELECT avg(someColumn) FROM /var/lib/cassandra/data/system/peers/ma-1-big-Data.db WHERE key \u003e 1\n\nDUMP               - dump the raw unfiltered partitions/rows. Useful for debuging TTLed/tombstoned data.\n    DUMP;\n    Can also specify a where clause for filtering the results.\n    DUMP WHERE partitionKey = 'OpsCenter';\n```\n\n![cql example](http://imgur.com/YXyjffj.gif)\n(note: slow analyzing artifact of ttygif, describing sstable in this scenario is sub second.)\n\n## describe\n\nProvides information about an sstable's data and its metadata. Can be used as argument or via cqlsh.\n\nExample Output:\n\n```\n/Users/clohfink/git/sstable-tools/ma-119-big-Data.db\n====================================================\nPartitions: 22515                                                               \nRows: 13579337\nTombstones: 0\nCells: 13579337\nWidest Partitions:\n   [12345] 999999\n   [99049] 62664\n   [99007] 60437\n   [99017] 59728\n   [99010] 59555\nLargest Partitions:\n   [12345] 189888705 (189.9 MB)\n   [99049] 2965017 (3.0 MB)\n   [99007] 2860391 (2.9 MB)\n   [99017] 2826094 (2.8 MB)\n   [99010] 2818038 (2.8 MB)\nTombstone Leaders:\nPartitioner: org.apache.cassandra.dht.Murmur3Partitioner\nBloom Filter FP chance: 0.010000\nSize: 540932262 (540.9 MB) \nCompressor: org.apache.cassandra.io.compress.LZ4Compressor\n  Compression ratio: 0.3068105022732033\nMinimum timestamp: 1456554952195298 (02/27/2016 00:35:52)\nMaximum timestamp: 1456594562846756 (02/27/2016 11:36:02)\nSSTable min local deletion time: 2147483647 (01/18/2038 21:14:07)\nSSTable max local deletion time: 2147483647 (01/18/2038 21:14:07)\nTTL min: 0 (0 milliseconds)\nTTL max: 0 (0 milliseconds)\nminClustringValues: [1]\nmaxClustringValues: [999999]\nEstimated droppable tombstones: 0.0\nSSTable Level: 0\nRepaired at: 0 (12/31/1969 18:00:00)\n  Lower bound: ReplayPosition(segmentId=-1, position=0)\n  Upper bound: ReplayPosition(segmentId=1456414025108, position=16709244)\ntotalColumnsSet: 13579337\ntotalRows: 13579337\nEstimated tombstone drop times:\n  Value                            | Count      %   Histogram \n  2147483647 (01/18/2038 21:14:07) | 27158674 (100) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ \nEstimated partition size:\n  Value     | Count   %   Histogram \n  258       |     6 (  0)  \n  310       |    20 (  0) ▏ \n  372       |    50 (  0) ▍ \n  446       |    45 (  0) ▍ \n  535       |    66 (  0) ▍ \n  642       |    62 (  0) ▍ \n  770       |   120 (  0) ▉ \n  924       |   116 (  0) ▉ \n  1109      |   144 (  0) ▉▏ \n  1331      |   175 (  0) ▉▍ \n  1597      |   210 (  0) ▉▊ \n  1916      |   248 (  1) ▉▉ \n  2299      |   324 (  1) ▉▉▋ \n  2759      |   356 (  1) ▉▉▉ \n  3311      |   460 (  2) ▉▉▉▉ \n  3973      |   534 (  2) ▉▉▉▉▍ \n  4768      |   569 (  2) ▉▉▉▉▊ \n  5722      |   667 (  2) ▉▉▉▉▉▋ \n  6866      |   786 (  3) ▉▉▉▉▉▉▋ \n  8239      |  1002 (  4) ▉▉▉▉▉▉▉▉▍ \n  9887      |  1266 (  5) ▉▉▉▉▉▉▉▉▉▉▋ \n  11864     |  1446 (  6) ▉▉▉▉▉▉▉▉▉▉▉▉▏ \n  14237     |  1779 (  7) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ \n  17084     |  2081 (  9) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▍ \n  20501     |  2598 ( 11) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ \n  24601     |  3089 ( 13) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ \n  29521     |  3555 ( 15) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ \n  35425     |   681 (  3) ▉▉▉▉▉▋ \n  186563160 |     1 (  0)  \nEstimated column count:\n  Value   | Count   %   Histogram \n  10      |    41 (  0) ▎ \n  12      |    39 (  0) ▎ \n  14      |    32 (  0) ▎ \n  17      |    63 (  0) ▍ \n  20      |    64 (  0) ▍ \n  24      |   102 (  0) ▉ \n  29      |   120 (  0) ▉ \n  35      |   123 (  0) ▉ \n  42      |   165 (  0) ▉▍ \n  50      |   189 (  0) ▉▋ \n  60      |   220 (  0) ▉▉ \n  72      |   313 (  1) ▉▉▋ \n  86      |   326 (  1) ▉▉▊ \n  103     |   429 (  1) ▉▉▉▋ \n  124     |   493 (  2) ▉▉▉▉▎ \n  149     |   566 (  2) ▉▉▉▉▉ \n  179     |   643 (  2) ▉▉▉▉▉▍ \n  215     |   765 (  3) ▉▉▉▉▉▉▋ \n  258     |   949 (  4) ▉▉▉▉▉▉▉▉▏ \n  310     |  1197 (  5) ▉▉▉▉▉▉▉▉▉▉▍ \n  372     |  1382 (  6) ▉▉▉▉▉▉▉▉▉▉▉▉ \n  446     |  1722 (  7) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ \n  535     |  1971 (  8) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ \n  642     |  2469 ( 10) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▍ \n  770     |  2908 ( 12) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▎ \n  924     |  3455 ( 15) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ \n  1109    |  1708 (  7) ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▊ \n  1131752 |     1 (  0)  \nEstimated cardinality: 22873\nEncodingStats minTTL: 0 (0 milliseconds)\nEncodingStats minLocalDeletionTime: 1442880000 (01/17/1970 10:48:00)\nEncodingStats minTimestamp: 1456554952195298 (02/27/2016 00:35:52)\nKeyType: org.apache.cassandra.db.marshal.UTF8Type\nClusteringTypes: [org.apache.cassandra.db.marshal.UTF8Type]\nStaticColumns: {}\nRegularColumns: {val:org.apache.cassandra.db.marshal.UTF8Type}\n```\n\n### Usage\n\n```\njava -jar sstable-tools.jar describe /path/to/file.db\n```\n\n## hints\n\nDeserialize the mutations in a hints file and print them to standard out. To have the information necessary do deserialize\nthe mutations this tool requires the schema of the file. This is currently handled by connecting to the cluster and querying\nthe metadata. You can specify the host and (cql) port via the `-h` and `-p` options.\n\nExample Output:\n\n```\njava -jar sstable-tools.jar hints 1458786695234-1.hints\nLoading schema from 127.0.0.1:9042\n/Users/clohfink/1458786695234-1.hints\n=====================================\n[test.t1] key=1 columns=[[] | [val]]\n    Row: EMPTY | val=1\n[[val=1 ts=1458786688691698]]\n[test.wide] key=myPartitionKey columns=[[] | [val]]\n    Row: key2=myClusteringKey | val=myValue\n[[val=myValue ts=1458790233298650]]\n[test.wide] key=myPartitionKey columns=[[] | [val]]\n    Row: key2=myClusteringKey2 | val=myValue2\n[[val=myValue2 ts=1458790238249336]]\n\n```\n\n### Usage\n\n```\njava -jar sstable-tools.jar hints\n\nusage: hints [-h \u003carg\u003e] [-p \u003carg\u003e] [-s] hintfile [hintfile ...]\n\nHint Dump for Apache Cassandra 3.x\nOptions:\n  -h \u003carg\u003e Host to extract schema frome.\n  -p \u003carg\u003e CQL native port.\n  -s       Only output mutations.\n  hintfile at least one file containing hints\n\n```\n\n## compact\n\nMerge multiple sstables into a single sstable. Arguments can either be sstables or a directory holding multiple sstables.\n\nExample Output:\n\n```\njava -jar sstable-tools.jar compact /Users/clohfink/node1table/mc-15-big-Data.db /Users/clohfink/node2table\nMerging 4 sstables to /Users/clohfink/git/sstable-tools/mc-0-big-Data.db\n```\n\n### Usage\n\n```\njava -jar sstable-tools.jar compact\n\nUsage: compact sstable [sstable ...]\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftolbertam%2Fsstable-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftolbertam%2Fsstable-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftolbertam%2Fsstable-tools/lists"}