{"id":38505181,"url":"https://github.com/messede-degod/sstable-migrator","last_synced_at":"2026-01-17T06:08:11.553Z","repository":{"id":209058223,"uuid":"718705944","full_name":"messede-degod/sstable-migrator","owner":"messede-degod","description":"Generate SStables From CSV. The Data Conversion Workhorse behind https://ip.thc.org","archived":false,"fork":false,"pushed_at":"2025-12-15T16:50:13.000Z","size":59715,"stargazers_count":2,"open_issues_count":6,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-12-18T22:57:50.824Z","etag":null,"topics":["cassandra-database","segfault","subdomain-discovery","subdomains-enumeration","thc"],"latest_commit_sha":null,"homepage":"https://ip.thc.org","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/messede-degod.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-11-14T16:25:39.000Z","updated_at":"2025-12-18T01:02:06.000Z","dependencies_parsed_at":"2023-12-02T09:26:04.088Z","dependency_job_id":"8ade56d2-2c63-4cd1-862b-77202d86bb30","html_url":"https://github.com/messede-degod/sstable-migrator","commit_stats":null,"previous_names":["messede-degod/sstable-migrator"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/messede-degod/sstable-migrator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/messede-degod%2Fsstable-migrator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/messede-degod%2Fsstable-migrator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/messede-degod%2Fsstable-migrator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/messede-degod%2Fsstable-migrator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/messede-degod","download_url":"https://codeload.github.com/messede-degod/sstable-migrator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/messede-degod%2Fsstable-migrator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28501813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T04:31:57.058Z","status":"ssl_error","status_checked_at":"2026-01-17T04:31:45.816Z","response_time":85,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra-database","segfault","subdomain-discovery","subdomains-enumeration","thc"],"created_at":"2026-01-17T06:08:09.981Z","updated_at":"2026-01-17T06:08:11.539Z","avatar_url":"https://github.com/messede-degod.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"### sstable-migrator\n\n# Building\n -  install and use **java 8**, check with `java -version`\n -  compile - `mvn compile`\n -  run - `MAVEN_OPTS=\"-Xmx7114M\" mvn exec:java -DargLine=\"-Xms6144m  -Xmx7144m\"` to convert `input/*` to sstables in `/output`\n\n# Setup Cassandra\n - Start Container - `sudo docker run -v ./output/:/ferret/dnsdata  -d --name cassandra --hostname cassandra --network cassandra cassandra` (Allow upto a minute for bootup)\n - Start a cqlsh shell - `sudo docker exec -it cassandra cqlsh`\n \n - Create Keyspace \n    ```\n    CREATE KEYSPACE ferret WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'};\n    ```\n \n - Create RDNS Table\n    ```\n    CREATE TABLE ferret.rdnsv4 (\n        ip8 INET,\n        ip16 INET,\n        ip24 INET,\n        ipAddress INET,\n        p1 VARCHAR,\n        p2 VARCHAR,\n        p3 VARCHAR,\n        p4 VARCHAR,\n        p5 VARCHAR,\n        p6 VARCHAR,\n        p7 VARCHAR,\n        country VARCHAR,\n        city VARCHAR,\n        asn INT,\n        as_name VARCHAR,\n        source VARCHAR,\n        sourceRecordType VARCHAR,\n        firstSeen timestamp,\n        lastSeen timestamp,\n        updatedAt timestamp,\n        PRIMARY KEY (ip8, ip16, ip24, ipAddress, p1, p2, p3, p4, p5, p6, p7)\n    );\n    ```\n\n - Create SubDomains table - \n    ```\n    CREATE TABLE ferret.subdomains (\n        p1 VARCHAR, \n        p2 VARCHAR, \n        p3 VARCHAR, \n        p4 VARCHAR, \n        p5 VARCHAR, \n        p6 VARCHAR, \n        p7 VARCHAR, \n        source VARCHAR,\n        sourceRecordType VARCHAR,\n        firstSeen timestamp,\n        lastSeen timestamp,\n        updatedAt timestamp,\n        PRIMARY KEY ((p1, p2, p3), p4, p5, p6, p7)\n    );\n    ```\n\n - Create CNAME table - \n    ```\n    CREATE TABLE ferret.cnames (\n        target VARCHAR, \n        apexDomain VARCHAR, \n        domain VARCHAR,\n        source VARCHAR,\n        firstSeen timestamp,\n        lastSeen timestamp,\n        updatedAt timestamp,\n        PRIMARY KEY (target, apexDomain, domain)\n    );\n    ```\n \n - Move Data - `sudo docker container exec -it cassandra  sstableloader -d 172.18.0.2 /ferret/dnsdata/`\n\n# Possible Improvements\n -  \u003cstrike\u003euse java FileChannel to read files (possible performance improvements)\u003c/strike\u003e (no improvements observed)\n -  use fastjson parser\n -  \u003cstrike\u003euse multithreaded writes to CQLSSTableWriter (https://issues.apache.org/jira/browse/CASSANDRA-7463)\u003c/strike\u003e (bad idea, write performance is far better when keys are in order, writes with out of order keys take up a lot of cpu, but yield no improvement in conversion time)\n\n\n# TLD Source\n - https://data.iana.org/TLD/tlds-alpha-by-domain.txt","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmessede-degod%2Fsstable-migrator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmessede-degod%2Fsstable-migrator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmessede-degod%2Fsstable-migrator/lists"}