{"id":32484393,"url":"https://github.com/bewaremypower/kafka-keys-cleaner","last_synced_at":"2026-07-14T09:32:17.956Z","repository":{"id":313937047,"uuid":"1053477794","full_name":"BewareMyPower/kafka-keys-cleaner","owner":"BewareMyPower","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-09T16:23:02.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-27T03:51:33.907Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BewareMyPower.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-09T13:59:55.000Z","updated_at":"2025-09-09T16:23:05.000Z","dependencies_parsed_at":"2025-09-09T17:43:11.303Z","dependency_job_id":"f5df8513-7cac-40c1-8242-6017f76e92af","html_url":"https://github.com/BewareMyPower/kafka-keys-cleaner","commit_stats":null,"previous_names":["bewaremypower/kafka-keys-cleaner"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BewareMyPower/kafka-keys-cleaner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BewareMyPower%2Fkafka-keys-cleaner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BewareMyPower%2Fkafka-keys-cleaner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BewareMyPower%2Fkafka-keys-cleaner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BewareMyPower%2Fkafka-keys-cleaner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BewareMyPower","download_url":"https://codeload.github.com/BewareMyPower/kafka-keys-cleaner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BewareMyPower%2Fkafka-keys-cleaner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35455901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-14T02:00:06.603Z","response_time":114,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-27T03:50:59.874Z","updated_at":"2026-07-14T09:32:17.949Z","avatar_url":"https://github.com/BewareMyPower.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafka-keys-cleaner\n\nA simple tool to clean up old keys in Kafka topics.\n\nP.S. With a standard Apache Kafka cluster, the same goal can be achieved by configuring `delete,compact` as the cleanup policy for a topic.\n\n## How to build\n\nBuild the project with JDK 17 or later:\n\n```bash\nmvn clean install -DskipTests\n```\n\n## Demo against a cluster on StreamNative Cloud\n\nAfter building the project, save the Kafka service URL to `./KAFKA-URL.txt` and token to `./TOKEN.txt`, then save them as environment variables:\n\n```bash\nexport URL=$(cat ./KAFKA-URL.txt)\nexport TOKEN=$(cat ./TOKEN.txt)\n```\n\nAfter that, you can leverage this demo project to show how to delete keys before a timestamp.\n\n### Step 1: Create a compacted topic\n\n```bash\njava -jar target/kafka-keys-cleaner.jar $URL test-topic --token $TOKEN compact --create\n```\n\n### Step 2: produce some messages\n\n```bash\njava -jar target/kafka-keys-cleaner.jar $URL test-topic --token $TOKEN produce -f DATA.txt\n```\n\nSee [DATA.txt](./DATA.txt) for the content.\n\nYou will see outputs like:\n\n```\nSent K0 =\u003e V0 to test-topic-0@0 timestamp: 1757433987842\nSent K0 =\u003e V1 to test-topic-0@1 timestamp: 1757433990328\nSent K1 =\u003e V0 to test-topic-0@2 timestamp: 1757433990603\nSent K2 =\u003e v0 to test-topic-0@3 timestamp: 1757433990932\nSent K1 =\u003e V1 to test-topic-0@4 timestamp: 1757433991207\nSent K0 =\u003e V2 to test-topic-0@5 timestamp: 1757433991491\n```\n\nYou can try consuming these messages with `my-group` as the group id:\n\n```bash\njava -jar target/kafka-keys-cleaner.jar $URL test-topic --token $TOKEN consume my-group\n```\n\n```\nReceived K0 =\u003e V0 from test-topic-0@0 timestamp: 1757433987842\nReceived K0 =\u003e V1 from test-topic-0@1 timestamp: 1757433990328\nReceived K1 =\u003e V0 from test-topic-0@2 timestamp: 1757433990603\nReceived K2 =\u003e v0 from test-topic-0@3 timestamp: 1757433990932\nReceived K1 =\u003e V1 from test-topic-0@4 timestamp: 1757433991207\nReceived K0 =\u003e V2 from test-topic-0@5 timestamp: 1757433991491\n```\n\n### Step 3: Write timestones for keys before a timestamp\n\nLet's use `1757433990933` as the timestamp:\n\n```bash\njava -jar target/kafka-keys-cleaner.jar $URL test-topic --token $TOKEN compact --set-key-ts=1757433990933\n```\n\nYou will see the following outputs:\n\n```\nDeleted key K2\n```\n\nIt's expected because the messages after this timestamp are:\n\n```\nK1 =\u003e V1\nK0 =\u003e V2\n```\n\nSo the outdated key `K2` will be deleted as well.\n\n### Step 4: Verify the compacted results\n\nYou can try consuming again and will see a tombstone message for key `K2` has been written:\n\n```bash\njava -jar target/kafka-keys-cleaner.jar $URL test-topic --token $TOKEN consume my-group\n```\n\nOutputs:\n\n```\nReceived K2 =\u003e null from test-topic-0@6 timestamp: 1757434254734\n```\n\nLet's use Pulsar CLI to manually trigger the compaction:\n\n```bash\n# Please note that the URL is a Pulsar admin service URL\nbin/pulsar-admin \\\n    --admin-url \u003cpulsar-admin-service-url\u003e \\\n    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \\\n    --auth-params token:$TOKEN \\\n    topics compact test-topic\n```\n\nWait for a while and consume again:\n\n```bash\njava -jar target/kafka-keys-cleaner.jar $URL test-topic --token $TOKEN consume my-group\n```\n\nNow, you will see the messages after compaction:\n\n```\nReceived K1 =\u003e V1 from test-topic-0@4 timestamp: 1757433991207\nReceived K0 =\u003e V2 from test-topic-0@5 timestamp: 1757433991491\n```\n\nAs a result, key `K2` has been deleted.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbewaremypower%2Fkafka-keys-cleaner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbewaremypower%2Fkafka-keys-cleaner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbewaremypower%2Fkafka-keys-cleaner/lists"}