https://github.com/geniusrabbit/eventstream

🚀 Eventstream pipeline to preprocess and resend from some source to some storage
https://github.com/geniusrabbit/eventstream

clickhouse eventstream go golang kafka nats postgresql prometheus redis redis-pubsub vertica

Last synced: 5 months ago
JSON representation

🚀 Eventstream pipeline to preprocess and resend from some source to some storage

• Host: GitHub
• URL: https://github.com/geniusrabbit/eventstream
• Owner: geniusrabbit
• License: apache-2.0
• Created: 2017-05-20T15:42:52.000Z (about 9 years ago)
• Default Branch: master
• Last Pushed: 2024-11-22T12:05:00.000Z (over 1 year ago)
• Last Synced: 2024-11-22T12:34:26.582Z (over 1 year ago)
• Topics: clickhouse, eventstream, go, golang, kafka, nats, postgresql, prometheus, redis, redis-pubsub, vertica
• Language: Go
• Homepage: https://hub.docker.com/r/geniusrabbit/eventstream
• Size: 426 KB
• Stars: 16
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Eventstream message pipeline service

![License](https://img.shields.io/github/license/geniusrabbit/eventstream)

[![Docker Pulls](https://img.shields.io/docker/pulls/geniusrabbit/eventstream.svg?maxAge=604800)](https://hub.docker.com/r/geniusrabbit/eventstream)

[![Go Report Card](https://goreportcard.com/badge/github.com/geniusrabbit/eventstream)](https://goreportcard.com/report/github.com/geniusrabbit/eventstream)

[![Coverage Status](https://coveralls.io/repos/github/geniusrabbit/eventstream/badge.svg?branch=master)](https://coveralls.io/github/geniusrabbit/eventstream?branch=master)

[![Testing Status](https://github.com/geniusrabbit/eventstream/workflows/Tests/badge.svg)](https://github.com/geniusrabbit/eventstream/actions?workflow=Tests)

[![Publish Docker Status](https://github.com/geniusrabbit/eventstream/workflows/Publish/badge.svg)](https://github.com/geniusrabbit/eventstream/actions?workflow=Publish)

Eventstream pipeline for storing and re-sending events inside the system.

```sh

go get -v -u github.com/geniusrabbit/eventstream/cmd/eventstream

```

## Run eventstream service in docker

```sh

docker run -d -it --rm -v ./custom.config.hcl:/config.hcl \

  geniusrabbit/eventstream

```

## Source list

- **kafka**

- **NATS** & **NATS stream**

- **Redis** stream

## Storage list

- **Clickhouse**

- **Vertica**

- **kafka**

- **NATS**

- **Redis** stream

## Config example

Supports two file formats YAML & HCL

```js

stores {

  clickhouse_1 {

    connect = "{{@env:CLICKHOUSE_STORE_CONNECT}}"

    buffer = 1000

    init_query = [<)

- [X] Ack message only if success

- [X] Buffering all data until be stored

- [X] Add support HCL config