https://github.com/segment-integrations/connect-kafka
Reference Architecture for Accepting Segment Webhooks and Publishing to Kafka
https://github.com/segment-integrations/connect-kafka
Last synced: about 1 year ago
JSON representation
Reference Architecture for Accepting Segment Webhooks and Publishing to Kafka
- Host: GitHub
- URL: https://github.com/segment-integrations/connect-kafka
- Owner: segment-integrations
- Created: 2016-06-06T00:08:17.000Z (about 10 years ago)
- Default Branch: master
- Last Pushed: 2017-10-28T03:59:55.000Z (over 8 years ago)
- Last Synced: 2025-03-27T04:33:32.850Z (about 1 year ago)
- Language: Go
- Homepage: https://segment.com
- Size: 11.7 KB
- Stars: 37
- Watchers: 28
- Forks: 12
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-kafka-connect - segment-integrations/connect-kafka - Segment destination for sending events to Kafka (CRM & Marketing / Segment)
README
# connect-kafka
This program is an example implementation of a [Segment](https://segment.com/) [Webhook](https://segment.com/docs/integrations/webhooks) consumer that publishes events to [Kafka](http://kafka.apache.org/).
This is not an officially supported Segment product, but is meant to demonstrate a simple server that you can fork or emulate to route Segment data to your internal systems. It may even suit your needs as is!
## Features
`connect-kafka` is a simple server that you deploy in your infrastructure and expose to the internet. It listens for Segment events and forwards them to the Kafka topic of your choice.
- Easily forward web, mobile, server analytics events to your Kafka instance
- Deploys in your infrastructure
- Supports any Kafka cluster
- Built with [Heroku Kafka](https://www.heroku.com/kafka) support in mind (with public/private space support)
- Supports SSL (or not) connections to your cluster
- Supports all Segment standard methods (`identify`, `track`, `page`, `screen`, `group`)
## Quickstart
1. *Connect to Kafka* - connect the `connect-kafka` to your Kafka instance.
2. *Setup Webbook* - Enter connect-kafka's listen address into your Segment webhook menu.
## FAQ
#### Does this support shared secret authentication?
Not yet, though we'd love a contribution that adds it!
#### How do Segment Webhooks behave if my server goes down?
We will retry the requests to the server 5 times over an hour if your server becomes unavailable.
#### Will the events arrive in order?
Because we're dealing with unbounded streaming data, we can't guarantee that your events arrive in the absolute order that they were collected in your client devices. As such, we recommend using the `timestamp` fields on each message with event-time windowing approaches in your destinations and streaming data applications.
### Connect to Kafka
Download `connect-kafka` using curl:
```bash
curl -s http://connect.segment.com/install-connect-kafka.sh | sh
```
If you just want the binary and install it yourself:
```bash
http://connect.segment.com/connect-kafka-darwin-amd64
```
You can also use Docker:
```bash
make docker
docker run segment/connect-kafka [...]
```
You can connect to any internal Kafka deployment.
```
$ connect-kafka -h
Usage:
connect-kafka
[--debug]
--topic=
--broker=...
[--listen=]
[--trusted-cert= --client-cert= --client-cert-key=]
connect-kafka -h | --help
connect-kafka --version
Options:
-h --help Show this screen
--version Show version
--topic= Kafka topic name
--listen= Address to listen on [default: localhost:3000]
--broker= Kafka broker URL
```
#### Heroku Kafka
Below is an example to connect to a Heroku Kafka in a public space (via SSL):
```bash
go get -u github.com/segment-integrations/connect-kafka
heroku config:get KAFKA_URL -a kafka-integration-demo # copy the kafka broker urls into command below
heroku config:get KAFKA_TRUSTED_CERT -a kafka-integration-demo > kafka_trusted_cert.cer
heroku config:get KAFKA_CLIENT_CERT -a kafka-integration-demo > kafka_client_cert.cer
heroku config:get KAFKA_CLIENT_CERT_KEY -a kafka-integration-demo > kafka_client_key_cert.cer
connect-kafka \
--debug \
--topic=segment \
--broker=kafka+ssl://ec2-51-16-10-109.compute-1.amazonaws.com:9096 \
--broker=kafka+ssl://ec2-62-7-61-181.compute-1.amazonaws.com:9096 \
--broker=kafka+ssl://ec2-33-20-240-35.compute-1.amazonaws.com:9096 \
--trusted-cert=kafka_trusted_cert.cer \
--client-cert=kafka_client_cert.cer \
--client-cert-key=kafka_client_key_cert.cer
```
### Setup Webhook
1. Go to the Segment.com and select the source you want to connect to Kafka
2. Add your `connect-kafka` server's address to the webhook integration's settings.
## Testing
### via localtunnel
You can open up a localtunnel on your local machine while you're testing:
```
npm install -g localtunnel
lt --port 3000
```
Enter the resulting localtunnel url as the Segment webhook with `/listen` appended, like: `https://aqjujyhnck.localtunnel.me/listen`
## License
MIT