{"id":21762116,"url":"https://github.com/segment-integrations/connect-kafka","last_synced_at":"2025-04-13T13:32:51.006Z","repository":{"id":37647538,"uuid":"60487117","full_name":"segment-integrations/connect-kafka","owner":"segment-integrations","description":"Reference Architecture for Accepting Segment Webhooks and Publishing to Kafka","archived":false,"fork":false,"pushed_at":"2017-10-28T03:59:55.000Z","size":12,"stargazers_count":37,"open_issues_count":1,"forks_count":12,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-03-27T04:33:32.850Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://segment.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/segment-integrations.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-06-06T00:08:17.000Z","updated_at":"2024-04-26T19:39:18.000Z","dependencies_parsed_at":"2022-09-04T20:11:23.929Z","dependency_job_id":null,"html_url":"https://github.com/segment-integrations/connect-kafka","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segment-integrations%2Fconnect-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segment-integrations%2Fconnect-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segment-integrations%2Fconnect-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segment-integrations%2Fconnect-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/segment-integrations","download_url":"https://codeload.github.com/segment-integrations/connect-kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248721272,"owners_count":21151074,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-26T12:11:01.749Z","updated_at":"2025-04-13T13:32:50.976Z","avatar_url":"https://github.com/segment-integrations.png","language":"Go","funding_links":[],"categories":["CRM \u0026 Marketing"],"sub_categories":["Segment"],"readme":"# connect-kafka\n\nThis program is an example implementation of a [Segment](https://segment.com/) [Webhook](https://segment.com/docs/integrations/webhooks) consumer that publishes events to [Kafka](http://kafka.apache.org/).\n\nThis is not an officially supported Segment product, but is meant to demonstrate a simple server that you can fork or emulate to route Segment data to your internal systems. It may even suit your needs as is! \n\n\u003cimg src=\"http://hortonworks.com/wp-content/uploads/2016/03/kafka-logo-wide.png\" data-canonical-src=\"http://hortonworks.com/wp-content/uploads/2016/03/kafka-logo-wide.png\" width=\"200\" height=\"105\" /\u003e\n\n## Features\n`connect-kafka` is a simple server that you deploy in your infrastructure and expose to the internet. It listens for Segment events and forwards them to the Kafka topic of your choice.\n\n- Easily forward web, mobile, server analytics events to your Kafka instance\n- Deploys in your infrastructure\n- Supports any Kafka cluster\n- Built with [Heroku Kafka](https://www.heroku.com/kafka) support in mind (with public/private space support)\n- Supports SSL (or not) connections to your cluster\n- Supports all Segment standard methods (`identify`, `track`, `page`, `screen`, `group`)\n\n## Quickstart\n\n1. *Connect to Kafka* - connect the `connect-kafka` to your Kafka instance.\n2. *Setup Webbook* - Enter connect-kafka's listen address into your Segment webhook menu.\n\n## FAQ\n\n#### Does this support shared secret authentication? \n\nNot yet, though we'd love a contribution that adds it! \n\n#### How do Segment Webhooks behave if my server goes down?\n\nWe will retry the requests to the server 5 times over an hour if your server becomes unavailable.\n\n#### Will the events arrive in order?\n\nBecause we're dealing with unbounded streaming data, we can't guarantee that your events arrive in the absolute order that they were collected in your client devices. As such, we recommend using the `timestamp` fields on each message with event-time windowing approaches in your destinations and streaming data applications.\n\n### Connect to Kafka\n\nDownload `connect-kafka` using curl:\n\n```bash\ncurl -s http://connect.segment.com/install-connect-kafka.sh | sh\n```\n\nIf you just want the binary and install it yourself:\n\n```bash\nhttp://connect.segment.com/connect-kafka-darwin-amd64\n```\n\nYou can also use Docker:\n\n```bash\nmake docker\ndocker run segment/connect-kafka [...]\n```\n\nYou can connect to any internal Kafka deployment.\n\n```\n$ connect-kafka -h\n\nUsage:\n  connect-kafka\n    [--debug]\n    --topic=\u003ctopic\u003e\n    --broker=\u003curl\u003e...\n    [--listen=\u003caddr\u003e]\n    [--trusted-cert=\u003cpath\u003e --client-cert=\u003cpath\u003e --client-cert-key=\u003cpath\u003e]\n  connect-kafka -h | --help\n  connect-kafka --version\n\nOptions:\n  -h --help                   Show this screen\n  --version                   Show version\n  --topic=\u003ctopic\u003e             Kafka topic name\n  --listen=\u003caddr\u003e             Address to listen on [default: localhost:3000]\n  --broker=\u003curl\u003e              Kafka broker URL\n```\n\n#### Heroku Kafka\n\nBelow is an example to connect to a Heroku Kafka in a public space (via SSL):\n\n```bash\ngo get -u github.com/segment-integrations/connect-kafka\nheroku config:get KAFKA_URL -a kafka-integration-demo  # copy the kafka broker urls into command below\nheroku config:get KAFKA_TRUSTED_CERT -a kafka-integration-demo \u003e kafka_trusted_cert.cer\nheroku config:get KAFKA_CLIENT_CERT -a kafka-integration-demo \u003e kafka_client_cert.cer\nheroku config:get KAFKA_CLIENT_CERT_KEY -a kafka-integration-demo \u003e kafka_client_key_cert.cer\nconnect-kafka \\\n --debug \\\n --topic=segment \\\n --broker=kafka+ssl://ec2-51-16-10-109.compute-1.amazonaws.com:9096 \\\n --broker=kafka+ssl://ec2-62-7-61-181.compute-1.amazonaws.com:9096 \\\n --broker=kafka+ssl://ec2-33-20-240-35.compute-1.amazonaws.com:9096 \\\n --trusted-cert=kafka_trusted_cert.cer \\\n --client-cert=kafka_client_cert.cer \\\n --client-cert-key=kafka_client_key_cert.cer\n ```\n\n### Setup Webhook\n\n1. Go to the Segment.com and select the source you want to connect to Kafka\n2. Add your `connect-kafka` server's address to the webhook integration's settings.\n\n![](http://g.recordit.co/XcyIz2fqJv.gif)\n\n\n## Testing\n\n### via localtunnel\n\nYou can open up a localtunnel on your local machine while you're testing:\n\n```\nnpm install -g localtunnel\nlt --port 3000\n```\n\nEnter the resulting localtunnel url as the Segment webhook with `/listen` appended, like: `https://aqjujyhnck.localtunnel.me/listen`\n\n## License\n\n MIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsegment-integrations%2Fconnect-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsegment-integrations%2Fconnect-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsegment-integrations%2Fconnect-kafka/lists"}