{"id":13783632,"url":"https://github.com/itadventurer/kafka-backup","last_synced_at":"2025-04-04T18:05:44.211Z","repository":{"id":45398864,"uuid":"189896453","full_name":"itadventurer/kafka-backup","owner":"itadventurer","description":"Backup and Restore for Apache Kafka","archived":false,"fork":false,"pushed_at":"2024-12-04T10:04:02.000Z","size":2845,"stargazers_count":168,"open_issues_count":42,"forks_count":48,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-28T17:08:15.199Z","etag":null,"topics":["backup","kafka","kafka-connect"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/itadventurer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-02T21:21:59.000Z","updated_at":"2025-03-28T14:17:51.000Z","dependencies_parsed_at":"2025-01-18T21:52:52.015Z","dependency_job_id":null,"html_url":"https://github.com/itadventurer/kafka-backup","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itadventurer%2Fkafka-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itadventurer%2Fkafka-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itadventurer%2Fkafka-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itadventurer%2Fkafka-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/itadventurer","download_url":"https://codeload.github.com/itadventurer/kafka-backup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247226213,"owners_count":20904465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","kafka","kafka-connect"],"created_at":"2024-08-03T19:00:27.020Z","updated_at":"2025-04-04T18:05:44.190Z","avatar_url":"https://github.com/itadventurer.png","language":"Java","funding_links":[],"categories":["进程间通信","Operations"],"sub_categories":["Spring Cloud框架","Backup"],"readme":"# Kafka Backup\n\n\u003e **Update:** I am no longer maintaining the Kafka Backup project. As an alternative, I recommend [Kannika](https://kannika.io/?utm_source=github_anatoly), a commercial backup solution developed by my friends at [Cymo](https://cymo.eu/?utm_source=github_anatoly) (and don't forget to say hello from Anatoly 😊).\n\u003e [Disclosure: I am a business partner of Cymo and may receive compensation for referrals to Kannika]\n\u003e \n\u003e Please contact me if you want to continue maintaining this project.\n\nKafka Backup is a tool to back up and restore your Kafka data\nincluding all (configurable) topic data and especially also consumer\ngroup offsets. To the best of our knowledge, Kafka Backup is the only\nviable solution to take a cold backup of your Kafka data and restore\nit correctly.\n\nIt is designed as two connectors for Kafka\nConnect: A sink connector (backing data up) and a source connector\n(restoring data).\n\nCurrently `kafka-backup` supports backup and restore to/from the file\nsystem.\n\n## Features\n\n* Backup and restore topic data\n* Backup and restore consumer-group offsets\n* Currently supports only backup/restore to/from local file system\n* Released as a jar file or packaged as a Docker image\n\n# Getting Started\n\n**Option A) Download binary**\n\nDownload the latest release [from GitHub](https://github.com/itadventurer/kafka-backup/releases) and unzip it.\n\n**Option B) Use Docker image**\n\nPull the latest Docker image from [Docker Hub](https://hub.docker.com/repository/docker/itadventurer/kafka-backup/tags)\n\n**DO NOT USE THE `latest` STAGE IN PRODUCTION**. `latest` are automatic builds of the master branch. Be careful!\n\n**Option C) Build from source**\n\nJust run `./gradlew shadowJar` in the root directory of Kafka Backup. You will find the CLI tools in the `bin` directory.\n\n## Start Kafka Backup\n\n```sh\nbackup-standalone.sh --bootstrap-server localhost:9092 \\\n    --target-dir /path/to/backup/dir --topics 'topic1,topic2'\n```\n\nIn Docker:\n```sh\ndocker run -d -v /path/to/backup-dir/:/kafka-backup/ --rm \\\n    kafka-backup:[LATEST_TAG] \\\n    backup-standalone.sh --bootstrap-server kafka:9092 \\\n    --target-dir /kafka-backup/ --topics 'topic1,topic2'\n```\n\nYou can pass options via CLI arguments or using environment variables:\n\n| Parameter                                   | Type/required? | Description                                                                                                          |\n|---------------------------------------------|----------------|----------------------------------------------------------------------------------------------------------------------|\n| `--bootstrap-server`\u003cbr/\u003e`BOOTSTRAP_SERVER` | [REQUIRED]     | The Kafka server to connect to                                                                                       |\n| `--target-dir`\u003cbr/\u003e`TARGET_DIR`             | [REQUIRED]     | Directory where the backup files should be stored                                                                    |\n| `--topics`\u003cbr/\u003e`TOPICS`                     | \u003cT1,T2,…\u003e      | List of topics to be backed up. You must provide either `--topics` or `--topics-regex`. Not both                     |\n| `--topics-regex`\u003cbr/\u003e`TOPICS_REGEX`         |                | Regex of topics to be backed up. You must provide either `--topics` or `--topics-regex`. Not both                    |\n| `--max-segment-size`\u003cbr/\u003e`MAX_SEGMENT_SIZE` |                | Size of the backup segments in bytes DEFAULT: 1GiB                                                                   |\n| `--command-config`\u003cbr/\u003e`COMMAND_CONFIG`     | \u003cFILE\u003e         | Property file containing configs to be passed to Admin Client. Only useful if you have additional connection options |\n| `--debug`\u003cbr/\u003e`DEBUG=y`                     |                | Print Debug information                                                                                              |\n| `--help`                                    |                | Prints this message                                                                                                  |\n\n**Kafka Backup does not stop!** The Backup process is a continous background job that runs forever as Kafka models data as a stream without end. See [Issue 52: Support point-in-time snapshots](https://github.com/itadventurer/kafka-backup/issues/52) for more information.\n\n## Restore data\n\n```sh\nrestore-standalone.sh --bootstrap-server localhost:9092 \\\n    --target-dir /path/to/backup/dir --topics 'topic1,topic2'\n```\n\nIn Docker:\n```sh\ndocker run -v /path/to/backup/dir:/kafka-backup/ --rm \\\n    kafka-backup:[LATEST_TAG]\n    restore-standalone.sh --bootstrap-server kafka:9092 \\\n    --source-dir /kafka-backup/ --topics 'topic1,topic2'\n```\n\nYou can pass options via CLI arguments or using environment variables:\n\n\n| Parameter                                   | Type/required? | Description                                                                                                                                                                                                                |\n|---------------------------------------------|----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `--bootstrap-server`\u003cbr/\u003e`BOOTSTRAP_SERVER` | [REQUIRED]     | The Kafka server to connect to                                                                                                                                                                                             |\n| `--source-dir`\u003cbr/\u003e`SOURCE_DIR`             | [REQUIRED]     | Directory where the backup files are found                                                                                                                                                                                 |\n| `--topics`\u003cbr/\u003e`TOPICS`                     | [REQUIRED]     | List of topics to restore                                                                                                                                                                                                  |\n| `--batch-size`\u003cbr/\u003e`BATCH_SIZE`             |                | Batch size (Default: 1MiB)                                                                                                                                                                                                 |\n| `--offset-file`\u003cbr/\u003e`OFFSET_FILE`           |                | File where to store offsets. THIS FILE IS CRUCIAL FOR A CORRECT RESTORATION PROCESS IF YOU LOSE IT YOU NEED TO START THE BACKUP FROM SCRATCH. OTHERWISE YOU WILL HAVE DUPLICATE DATA Default: [source-dir]/restore.offsets |\n| `--command-config`\u003cbr/\u003e`COMMAND_CONFIG`     | \u003cFILE\u003e         | Property file containing configs to be passed to Admin Client. Only useful if you have additional connection options                                                                                                       |\n| `--help`\u003cbr/\u003e`HELP`                         |                | Prints this message                                                                                                                                                                                                        |\n| `--debug`\u003cbr/\u003e`DEBUG`                       |                | Print Debug information (if using the environment variable, set it to 'y')                                                                                                                                                 |\n\n## More Documentation\n\n* [FAQ](./docs/FAQ.md)\n* [High Level\n  Introduction](./docs/Blogposts/2019-06_Introducing_Kafka_Backup.md)\n* [Comparing Kafka Backup\n  Solutions](./docs/Comparing_Kafka_Backup_Solutions.md)\n* [Architecture](./docs/Kafka_Backup_Architecture.md)\n* [Tooling](./docs/Tooling.md)\n\n## License\n\nThis project is licensed under the Apache License Version 2.0 (see\n[LICENSE](./LICENSE)).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitadventurer%2Fkafka-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fitadventurer%2Fkafka-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitadventurer%2Fkafka-backup/lists"}