https://github.com/dnutiu/speedy

A simple Kafka to Loki bridge written in Golang.
https://github.com/dnutiu/speedy

bridge golang-application kafka loki

Last synced: 9 months ago
JSON representation

A simple Kafka to Loki bridge written in Golang.

• Host: GitHub
• URL: https://github.com/dnutiu/speedy
• Owner: dnutiu
• License: apache-2.0
• Created: 2021-10-04T10:46:37.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2023-03-07T07:32:13.000Z (over 3 years ago)
• Last Synced: 2025-06-07T03:39:18.833Z (about 1 year ago)
• Topics: bridge, golang-application, kafka, loki
• Language: Go
• Homepage:
• Size: 69.3 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: Readme.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Speedy

Speedy is a Go streaming program that consumes data from Kafka topics based on a pattern and writes it to Loki.

### Configuration

Configuration is done via configuration file `config.json`, additionally you may override values using ENVIRONMENT variables.

Environment variables are prefixed with `SG_` and are directly mapped to the configuration file, for example:

`SG_KAFKA_GROUP_ID=override` will override `kafka_group_id` from `config.json`.

#### Example configuration:

```json

{

  "logging_level": "info",

  "kafka_bootstrap_servers": "kafkaboostrap:9092",

  "sentry_dsn": "",

  "kafka_group_id": "speedy",

  "kafka_offset_reset": "latest",

  "subscribe_topics": ["^topic\\.pattern.+"],

  "buffer_max_batch_size": 1000,

  "kafka_polling_goroutines": 10,

  "kafka_polling_timeout_ms": 30000,

  "loki_push_url": "http://loki-distributor.loki:3100/loki/api/v1/push",

  "loki_push_mode": "proto"

}

```

## Custom librdkafka build

To add support for regex negative lookahead expression a custom libdrdkafka build was necessary. 

Please see the `Dockerfile.librdkafka` file.

### Example deployment on Kubernetes

**deploy.yaml**

```yaml

  apiVersion: apps/v1

  kind: Deployment

  metadata:

    name: speedy-consumer-all

    labels:

      app: speedy

  spec:

    replicas: 1

    selector:

      matchLabels:

        app: speedy

    template:

      metadata:

        labels:

          app: speedy

      spec:

        containers:

          - name: speedy

            imagePullPolicy: Always

            image: xxx.dkr.ecr.eu-central-1.amazonaws.com/speedy:latest

            resources:

              requests:

                memory: "512Mi"

                cpu: "1000m"

              limits:

                memory: "2096Mi"

                cpu: "1000m"

            volumeMounts:

              - name: speedy-config-all

                mountPath: /root/.speedy

        volumes:

          - name: speedy-config-all

            configMap:

              # All configs.

              name: speedy-config-all

```

**configmap.yaml**

```yaml

apiVersion: v1

kind: ConfigMap

metadata:

  name: speedy-config-all

data:

  config.json: |-

    {

      "kafka_bootstrap_servers": "xxx.kafka.eu-central-1.amazonaws.com:9092",

      "sentry_dsn": "",

      "kafka_group_id": "speedy",

      "subscribe_topics": ["^.+"],

      "loki_push_url": "http://loki-distributor.loki:3100/loki/api/v1/push",

      "buffer_max_batch_size": 10000,

      "loki_push_mode": "proto"

    }

```