{"id":36864026,"url":"https://github.com/geniusrabbit/eventstream","last_synced_at":"2026-01-12T14:52:22.731Z","repository":{"id":21197055,"uuid":"91899072","full_name":"geniusrabbit/eventstream","owner":"geniusrabbit","description":"🚀 Eventstream pipeline to preprocess and resend from some source to some storage","archived":false,"fork":false,"pushed_at":"2024-11-22T12:05:00.000Z","size":436,"stargazers_count":16,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-22T12:34:26.582Z","etag":null,"topics":["clickhouse","eventstream","go","golang","kafka","nats","postgresql","prometheus","redis","redis-pubsub","vertica"],"latest_commit_sha":null,"homepage":"https://hub.docker.com/r/geniusrabbit/eventstream","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/geniusrabbit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-20T15:42:52.000Z","updated_at":"2024-11-22T12:04:28.000Z","dependencies_parsed_at":"2024-06-20T06:41:14.172Z","dependency_job_id":"6aca30fa-b7b5-4668-935b-df8797060d58","html_url":"https://github.com/geniusrabbit/eventstream","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"purl":"pkg:github/geniusrabbit/eventstream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geniusrabbit%2Feventstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geniusrabbit%2Feventstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geniusrabbit%2Feventstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geniusrabbit%2Feventstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/geniusrabbit","download_url":"https://codeload.github.com/geniusrabbit/eventstream/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geniusrabbit%2Feventstream/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28340411,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T12:22:26.515Z","status":"ssl_error","status_checked_at":"2026-01-12T12:22:10.856Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","eventstream","go","golang","kafka","nats","postgresql","prometheus","redis","redis-pubsub","vertica"],"created_at":"2026-01-12T14:52:22.164Z","updated_at":"2026-01-12T14:52:22.719Z","avatar_url":"https://github.com/geniusrabbit.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Eventstream message pipeline service\n\n![License](https://img.shields.io/github/license/geniusrabbit/eventstream)\n[![Docker Pulls](https://img.shields.io/docker/pulls/geniusrabbit/eventstream.svg?maxAge=604800)](https://hub.docker.com/r/geniusrabbit/eventstream)\n[![Go Report Card](https://goreportcard.com/badge/github.com/geniusrabbit/eventstream)](https://goreportcard.com/report/github.com/geniusrabbit/eventstream)\n[![Coverage Status](https://coveralls.io/repos/github/geniusrabbit/eventstream/badge.svg?branch=master)](https://coveralls.io/github/geniusrabbit/eventstream?branch=master)\n[![Testing Status](https://github.com/geniusrabbit/eventstream/workflows/Tests/badge.svg)](https://github.com/geniusrabbit/eventstream/actions?workflow=Tests)\n[![Publish Docker Status](https://github.com/geniusrabbit/eventstream/workflows/Publish/badge.svg)](https://github.com/geniusrabbit/eventstream/actions?workflow=Publish)\n\nEventstream pipeline for storing and re-sending events inside the system.\n\n```sh\ngo get -v -u github.com/geniusrabbit/eventstream/cmd/eventstream\n```\n\n## Run eventstream service in docker\n\n```sh\ndocker run -d -it --rm -v ./custom.config.hcl:/config.hcl \\\n  geniusrabbit/eventstream\n```\n\n## Source list\n\n- **kafka**\n- **NATS** \u0026 **NATS stream**\n- **Redis** stream\n\n## Storage list\n\n- **Clickhouse**\n- **Vertica**\n- **kafka**\n- **NATS**\n- **Redis** stream\n\n## Config example\n\nSupports two file formats YAML \u0026 HCL\n\n```js\nstores {\n  clickhouse_1 {\n    connect = \"{{@env:CLICKHOUSE_STORE_CONNECT}}\"\n    buffer = 1000\n    init_query = [\u003c\u003cQ\n      CREATE TABLE IF NOT EXISTS stat.testlog (\n         timestamp        DateTime\n       , datemark         Date default toDate(timestamp)\n       , service          String\n       , msg              String\n       , error            String\n       , created_at       DateTime default now()\n      ) Engine=Memory COMMENT 'The test table';\n    Q]\n  }\n  kafka_1 {\n    connect = \"{{@env:KAFKA_EVENTS_CONNECT}}\"\n  }\n}\n\n// Source could be any supported stream service like kafka, nats, etc...\nsources {\n  nats_1 {\n    connect = \"{{@env:NATS_SOURCE_CONNECT}}\"\n    format  = \"json\"\n  }\n}\n\n// Streams it's pipelines which have source and destination store\nstreams {\n  log_1 {\n    store  = \"clickhouse_1\"\n    source = \"nats_1\"\n    target = \"testlog\"\n    // Optional if fields in log and in message the same\n    // Transforms into:\n    //   INSERT INTO testlog (service, msg, error, timestamp) VALUES($srv, $msg, $err, @toDateTime($timestamp))\n    fields = \"service=srv,msg,error=err,timestamp=@toDateTime({{timestamp:date}})\"\n    where  = \"srv == \\\"main\\\"\"\n    metrics = [\n      {\n        name = \"log.counter\"\n        type = \"counter\"\n        tags {\n          server  = \"{{srv}}\"\n        }\n      }\n    ]\n  }\n  kafka_retranslate {\n    store  = \"kafka_1\"\n    source = \"nats_1\"\n    targets = [\n      {\n        fields = {\n          server = \"{{srv}}\"\n          timestamp = \"{{timestamp}}\"\n        }\n        where = \"type = \\\"statistic\\\"\"\n      }\n    ]\n    where = \"srv = \\\"events\\\"\"\n  }\n}\n```\n\n## Metrics\n\nMetrics helps analyze some events during processing and monitor streams state.\nEvery stream can process metrics with the keyword `metrics`.\n\nExample:\n\n```js\nmetrics = [\n  {\n    name = \"log.counter\"\n    type = \"counter\"\n    tags { server = \"{{srv}}\" }\n  },\n  {\n    name = \"actions.counter\"\n    type = \"counter\"\n    tags { action = \"{{action}}\" }\n  },\n  {...}\n]\n```\n\nAll metrics available by URL `/metrics` with prometheus protocol.\nTo activate metrics need to define profile connection port.\n\n```env\nSERVER_PROFILE_MODE=net\nSERVER_PROFILE_LISTEN=:6060\n```\n\n## Health check\n\n```sh\ncurl \"http://hostname:port/health-check\"\n```\n\n```json\n{\"status\":\"OK\"}\n```\n\n## TODO\n\n- [ ] Add processing custom error metrics\n- [ ] Add MySQL database storage\n- [ ] Add PostgreSQL database storage\n- [ ] Add MongoDB database storage\n- [X] Add HTTP/Ping driver storage\n- [X] Add Redis database storage\n- [X] Prepare evetstream as Framework extension\n- [X] Add Kafka stream writer support\n- [X] Add NATS stream writer support\n- [X] Add Redis stream source/storage support\n- [ ] Add RabbitMQ stream source/storage support\n- [X] Add health check API\n- [X] Add customizable prometheus metrics\n- [x] Add 'where' stream condition (\u003chttp://github.com/Knetic/govaluate\u003e)\n- [X] Ack message only if success\n- [X] Buffering all data until be stored\n- [X] Add support HCL config\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeniusrabbit%2Feventstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeniusrabbit%2Feventstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeniusrabbit%2Feventstream/lists"}