https://github.com/redborder/k2http
Forwards messages from kafka to an HTTP endpoint.
https://github.com/redborder/k2http
autodelivery http kafka redborder-ng
Last synced: 21 days ago
JSON representation
Forwards messages from kafka to an HTTP endpoint.
- Host: GitHub
- URL: https://github.com/redborder/k2http
- Owner: redBorder
- License: agpl-3.0
- Created: 2016-06-08T11:19:40.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2025-04-14T14:21:03.000Z (10 months ago)
- Last Synced: 2025-04-14T15:35:07.747Z (10 months ago)
- Topics: autodelivery, http, kafka, redborder-ng
- Language: Go
- Homepage:
- Size: 94.7 KB
- Stars: 4
- Watchers: 5
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://travis-ci.org/redBorder/k2http)
[](https://goreportcard.com/report/github.com/redBorder/k2http)
# K2HTTP2
k2http is a application that forwards messages from kafka 0.9+ to an HTTP
endpoint using [rbforwarder](https://github.com/redBorder/rbforwarder)
package.
## Installing
To install this application ensure you have the `GOPATH` environment variable
set and **[glide](https://glide.sh/)** installed.
```bash
curl https://glide.sh/get | sh
```
And then:
1. Clone this repo and cd to the project
```bash
git clone https://github.com/redBorder/k2http.git && cd k2http
```
2. Install dependencies and compile
```bash
make
```
3. Install on desired directory
```bash
prefix=/opt/rb make install
```
## Usage
```
Usage of k2http:
--config string
Config file
--debug
Show debug info
--version
Print version info
```
To run `k2http` just execute the following command:
```bash
k2http --config path/to/config/file
```
## Example config file
```yaml
pipeline:
queue: 1000 # Max internal queue size
backoff: 10 # Time to wait between retries in seconds
retries: 3 # Number of retries on fail (-1 not limited)
limiter:
max_messages: 5000 # Max messages per second
# max_bytes: 5242880 # Max bytes per second
kafka:
broker: "localhost:9092" # Kafka brokers
consumergroup: "k2http" # Consumer group ID
topics: # Kafka topics to listen
- rb_nmsp
- rb_radius
- rb_flow
- rb_loc
- rb_monitor
- rb_state
- rb_social
batch:
workers: 1 # Number of workers
size: 1000 # Max messages per batch
timeoutMillis: 100 # Max time to wait for send a batch
deflate: false # Use deflate to compress batches
http:
workers: 1
url: "http://localhost:8888" # Number of workers, one connection per worker
insecure: false # Skip SSSL verification
```
## Using with Docker
You can use the application with Docker. First you need to compile as usual and then generate the docker image:
```bash
make
docker build -t k2http-docker .
```
You can then use the app inside a docker container:
```bash
docker run --rm k2http-docker --version
```