{"id":19421595,"url":"https://github.com/kochava/firehose","last_synced_at":"2025-02-25T04:15:49.842Z","repository":{"id":48233477,"uuid":"71408447","full_name":"Kochava/firehose","owner":"Kochava","description":"A Kafka topic transfer agent","archived":false,"fork":false,"pushed_at":"2021-08-04T15:31:51.000Z","size":84,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-01-07T18:32:09.891Z","etag":null,"topics":["tf"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Kochava.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-19T23:39:35.000Z","updated_at":"2021-08-04T15:31:53.000Z","dependencies_parsed_at":"2022-08-30T11:50:26.119Z","dependency_job_id":null,"html_url":"https://github.com/Kochava/firehose","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kochava%2Ffirehose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kochava%2Ffirehose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kochava%2Ffirehose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kochava%2Ffirehose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Kochava","download_url":"https://codeload.github.com/Kochava/firehose/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240599194,"owners_count":19826959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["tf"],"created_at":"2024-11-10T13:29:02.216Z","updated_at":"2025-02-25T04:15:49.681Z","avatar_url":"https://github.com/Kochava.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"Firehose [![Build Status](https://travis-ci.org/Kochava/firehose.svg?branch=master)](https://travis-ci.org/Kochava/firehose) [![Coverage Status](https://coveralls.io/repos/github/Kochava/firehose/badge.svg?branch=master)](https://coveralls.io/github/Kochava/firehose?branch=master)\n======\n\nFirehose is a Kafka transfer agent which can do real-time transferring of a Topic from one set of Brokers to another. This is useful when you want the two clusters to remain independent and not use the built in replication process of Kafka. NOTE: This is mostly meant for seeding one cluster with another and not for any process needing data loss guarantees, for that check out [uReplicator](https://github.com/uber/uReplicator)\n\n### Requirements\n* Docker (for local testing)\n\n## Install\n\nClone this repository then:\n\n```\n$ git clone --recursive https://github.com/Kochava/firehose.git\n$ cd firehose\n$ go build ./cmd/firehose\n```\n\n#### Update Dependancies\n```\n$ git submodule update --recursive --remote\n```\n\n## Usage\n\n```\nNAME:\n   Kochava Kafka Transfer Agent - An agent which consumes a topic from one set of brokers and publishes to another set of brokers\n\nUSAGE:\n   firehose [global options] command [command options] [arguments...]\n\nVERSION:\n   0.3.4\n\nGLOBAL OPTIONS:\n   --src-zookeepers value        Comma delimited list of zookeeper nodes to connect to for the source brokers [$FIREHOSE_SRC_ZOOKEEPERS]\n   --dst-zookeepers value        Comma delimited list of zookeeper nodes to connect to for the destination brokers [$FIREHOSE_DST_ZOOKEEPERS]\n   --topic value                 Topic to transfer (default: \"firehose\") [$FIREHOSE_TOPIC]\n   --cg-name-suffix value        Suffix to use for the consumer group name in the format \u003ctopic\u003e_\u003csuffix\u003e (default: \"firehose\") [$FIREHOSE_CG_NAME_SUFFIX]\n   --buffer-size value           The number of messages to hold in memory at once (default: 10000) [$FIREHOSE_BUFFER_SIZE]\n   --max-errors value            The maximum number of errors to allow the kafka to experience before quitting (default: 10) [$FIREHOSE_MAX_ERROR]\n   --max-retry value             The maximum number of times to retry sending a message to the destination cluster (default: 5) [$FIREHOSE_MAX_RETRY]\n   --batch-size value            The number of messages to batch together when sending to the destination cluster (default: 500) [$FIREHOSE_BATCH_SIZE]\n   --flush-interval value        The interval (in ms) to flush messages that haven't been sent in a batch yet (default: 10000) [$FIREHOSE_FLUSH_INTERVAL]\n   --reset-offset                Resets the offset in the consumer group to real-time before starting [$FIREHOSE_RESET_OFFSET]\n   --influx-address value        Influx address (default: \"http://localhost:8086\") [$FIREHOSE_INFLUX_ADDRESS]\n   --influx-user value           Influx user name (default: \"firehose\") [$FIREHOSE_INFLUX_USER]\n   --influx-pass value           Influx password (default: \"firehose\") [$FIREHOSE_INFLUX_PASS]\n   --influx-db value             Influx database name (default: \"firehose\") [$FIREHOSE_INFLUX_DB]\n   --consumer-concurrency value  Number of consumer threads to run (default: 4) [$FIREHOSE_CONSUMER_CONCURRENCY]\n   --producer-concurrency value  Number of producer threads to run (default: 4) [$FIREHOSE_PRODUCER_CONCURRENCY]\n   --log-file value              Main log file to save to (default: \"/var/log/firehose/firehose.log\") [$FIREHOSE_LOG_FILE]\n   --stdout-logging              Override logging settings to log to STDOUT [$FIREHOSE_STDOUT_LOGGING]\n   --help, -h                    show help\n   --version, -v                 print the version\n```\n\n### Getting Started\n\nOnce you have the repo downloaded the following is all that's needed to get started testing locally. First grab your docker machine IP address, then in `Docker/test-compose.yml` update the `KAFKA_ADVERTISED_HOST_NAME` EnvVar to use this.\n\n```\n$ go build ./cmd/firehose\n$ docker-compose -f Docker/test-compose.yml up -d\n$ ./firehose --src-zookeepers=\"\u003cdocker-machine-ip\u003e:2181\" --dst-zookeepers=\"\u003cdocker-machine-ip\u003e:2182\" --stdout-logging\n```\n\nIf you want to scale up the kafka clusters you can do so\n```\n$ docker-compose -f Docker/test-compose.yml scale kafka_src=3\n```\n\nOnce up and running you can access Grafana at http://localhost:3000 with the default username and password. Next just add a new datasource, choosing `InfluxDB` and method `proxy`. For the location use `http://172.18.0.2:8086` with the default Influx db of `firehose` and user:pass being `firehose:firehose`. Once that's setup you can import the simple dashboard preprepared under `dist/dashboard.json`.\n\n#### Cloud Building\n\nYou can leverage Google Cloud Building in order to build the binary and generate a docker image.\n\n```\n$ PROJECT_ID=your-project-id BUILD_VERSION=v1 ./cloudbuild.sh\n```\n\n\n## Contributing\n\n### Grab the source and make a branch\n\n1. Fork it\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Make your changes\n4. Add some tests\n5. Commit your changes (`git commit -am 'Add some feature'`)\n6. Push to the branch (`git push origin my-new-feature`)\n7. Create new Pull Request\n\n### TODO\n\nWith the new refactor a lot of work was put into performance, because of that the historical transfer aspect was essentially scrapped in this release. It will be added back in with the next release.\n\n#### Github\nThe Github version of this repo is a mirror of Master from our internal repo. This means that feature branches are not available here.\n\n\n## Default Branch\n\nAs of October 1, 2020, github.com uses the branch name ‘main’ when creating the initial default branch for all new repositories.  In order to minimize any customizations in our github usage and to support consistent naming conventions, we have made the decision to rename the ‘master’ branch to be called ‘main’ in all Kochava’s github repos.\n\nFor local copies of the repo, the following steps will update to the new default branch:\n\n```\ngit branch -m master main\ngit fetch origin\ngit branch -u origin/main main\ngit remote set-head origin -a\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkochava%2Ffirehose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkochava%2Ffirehose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkochava%2Ffirehose/lists"}