{"id":19473291,"url":"https://github.com/tillkuhn/rubin","last_synced_at":"2026-01-05T17:17:21.785Z","repository":{"id":203750063,"uuid":"710325163","full_name":"tillkuhn/rubin","owner":"tillkuhn","description":"💿 rubin - a simple record producer for Kafka topics  written in Go that acts as a thin wrapper around Confluent's REST Proxy API (v3)","archived":false,"fork":false,"pushed_at":"2025-01-01T23:46:13.000Z","size":348,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-08T06:05:58.733Z","etag":null,"topics":["cli","cloudevents","confluent","go","golang","golang-library","http-client","kafka","kafka-client","kafka-go","producer","rest-api"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/tillkuhn/rubin","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tillkuhn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-26T13:23:34.000Z","updated_at":"2024-12-02T07:40:35.000Z","dependencies_parsed_at":"2024-09-17T03:56:04.837Z","dependency_job_id":"442fb678-7ae0-4839-a322-b64cd6526018","html_url":"https://github.com/tillkuhn/rubin","commit_stats":null,"previous_names":["tillkuhn/rubin"],"tags_count":45,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tillkuhn%2Frubin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tillkuhn%2Frubin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tillkuhn%2Frubin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tillkuhn%2Frubin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tillkuhn","download_url":"https://codeload.github.com/tillkuhn/rubin/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240693091,"owners_count":19842396,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","cloudevents","confluent","go","golang","golang-library","http-client","kafka","kafka-client","kafka-go","producer","rest-api"],"created_at":"2024-11-10T19:18:00.477Z","updated_at":"2026-01-05T17:17:16.742Z","avatar_url":"https://github.com/tillkuhn.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 💿 rubin - a simple record producer for kafka topics\n\n[![GoTemplate](https://img.shields.io/badge/go/template-black?logo=go)](https://github.com/SchwarzIT/go-template)\n![ci-build](https://github.com/tillkuhn/rubin/actions/workflows/main.yml/badge.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/tillkuhn/rubin)](https://goreportcard.com/report/github.com/tillkuhn/rubin)\n[![Go Reference](https://pkg.go.dev/badge/github.com/tillkuhn/rubin.svg)](https://pkg.go.dev/github.com/tillkuhn/rubin)\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=tillkuhn_rubin\u0026metric=alert_status)](https://sonarcloud.io/summary/new_code?id=tillkuhn_rubin)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=tillkuhn_rubin\u0026metric=coverage)](https://sonarcloud.io/component_measures?id=tillkuhn_rubin\u0026metric=coverage\u0026view=list)\n[![Latest Release](https://img.shields.io/github/v/release/tillkuhn/rubin?include_prereleases)](https://github.com/tillkuhn/rubin/releases)\n\n## Introduction\n\n*rubin* is a thin wrapper around the [Confluent  REST Proxy API (v3)](https://docs.confluent.io/platform/current/kafka-rest/api.html#records-v3) that makes it easy to produce \nstructured event records into an existing Kafka Topic. \n\nIt's written in Go, can be used as a CLI standalone binary or as a library, and uses just plain http communication.\n\nBesides support for arbitrary JSON payloads and strings, it can also wrap events in the so called [CloudEvents](https://cloudevents.io/) format, which is \"a specification for describing event data in a common way\". \n\n## Installation and usage\n\n### Use as standalone CLI\n\nGrap the most recent release from the [releases page](https://github.com/tillkuhn/rubin/releases) and make sure your environment is configured properly (see *help* section below for additional optional args)\n\n```\n$ printenv | grep -e ^KAFKA\n\nKAFKA_REST_ENDPOINT=https://localhost:443\nKAFKA_PRODUCER_API_KEY=1234567890\nKAFKA_CLUSTER_ID=abc-r2d2\nKAFKA_PRODUCER_API_SECRET=********\n```\n\nRun the standalone binary (simple example with string message)\n\n```\n$ rubin -topic public.hello -record \"Dragonfly out in the sun you know what I mean\"\n\n9:49PM\tINFO\trubin/main.go:60\tWelcome to rubin  {\"version\": \"v0.0.5\", \"built\": \"now\", \"commit\": \"7759eb6\"}\n9:49PM\tINFO\trubin/client.go:27\tKafka REST Proxy Client configured  {\"endpoint\": \"https://localhost:443\", \"useSecret\": true}\n9:49PM\tINFO\trubin/client.go:53\tPush record  {\"url\": \"https://localhost.cloud:443/kafka/v3/clusters/abc-r2d2/topics/public.hello/records\"}\n9:49PM\tINFO\trubin/client.go:84\tRecord committed  {\"status\": \"topic\", \"public.hello\": 200, \"offset\": 43, \"partition\": 0}\n```\n\nExample with JSON Payload, custom message key and multiple message headers\n\n```\n$ rubin -topic public.hello -record '{\"msg\":\"hello\"}' -key \"123\" -header \"source=ci\" -header \"mood=good\"\n```\n\nGet Help on CLI arguments and environment configuration\n\n```\n$ rubin -help\n\nWelcome to rubin v0.1.5 built now by go (b368bf8)\n\nThis application is configured via the environment. The following environment\nvariables can be used:\n\nKEY                          TYPE             DEFAULT    REQUIRED    DESCRIPTION\nKAFKA_REST_ENDPOINT          String                      false       Kafka REST Proxy Endpoint\nKAFKA_CLUSTER_ID             String                      false       Kafka Cluster ID\nKAFKA_PRODUCER_API_KEY       String                      false       Kafka API Key with Producer Privileges\nKAFKA_PRODUCER_API_SECRET    String                      false       Kafka API Secret with Producer Privileges\nKAFKA_HTTP_TIMEOUT           Duration         10s        false       Timeout for HTTP Client\nKAFKA_DUMP_MESSAGES          True or False    false      false       Print http request/response to stdout\nKAFKA_LOG_LEVEL              String           info       false       Min LogLevel debug,info,warn,error\n\n```\n```\nIn addition, the following CLI arguments are supported\n  -ce\n    \tCloudEvents format for event payload (default: STRING or JSON)\n  -header value\n    \tHeader formatted as key=value, can be used multiple times\n  -help\n    \tDisplay this help\n  -key string\n    \tKafka Message Key (optional, default is generated uuid)\n  -record string\n    \tRequest payload to send into the Kafka Topic\n  -source string\n    \tCloudEventy: The context in which an event happened (default \"rubin/cli\")\n  -subject string\n    \tCloudEventy: The subject of the event in the context of the event producer\n  -topic string\n    \tName of target Kafka Topic\n  -type string\n    \tCloudEvents: Type of event related to the originating occurrence (default \"event.Event\")\n  -v string\n    \tVerbosity, one of 'debug', 'info', 'warn', 'error' (default \"info\")\n```\n\n### Use as library in an external Go app\n\n```\ngo get github.com/tillkuhn/rubin\n```\n```\nclient := rubin.New(\u0026rubin.Options{\n\tRestEndpoint:      \"https://localhost:443\",\n\tClusterID:         \"abc-r2d2\",\n\tProducerAPIKey:    \"1234567890\",\n\tProducerAPISecret: \"**********\",\n})\nresp, err := client.Produce(context.Background(), Request{\n\tTopic:   \"public.hello\",\n\tData:    \"Dragonfly out in the sun you know what I mean\",\n\tKey:     \"134-5678\",\n\tHeaders: map[string]string{\"heading\": \"for tomorrow\"},\n})\nfmt.Printf(\"Record successfully commited, offset=%d partition=%d\\n\", resp.Offset, resp.PartitionId)\n```\n\n### 🐳 Use as docker image\n\nReleased vaultpal versions are build for multiple architectures and pushed to the public GitHub Container Registry (https://ghcr.io).\n\n```\ndocker run --rm ghcr.io/tillkuhn/rubin:v0.1.2 -help\n```\n\n## ☁️ Support for *CloudEvents*\n\nIn Addition to simple string records and json payloads, *rubin* also supports [CloudEvents](https://cloudevents.io/), a \"a specification for describing event data in common formats to provide interoperability across services, platforms and systems\", hosted by the [Cloud Native Computing Foundation](https://www.cncf.io/). See the `-ce` switch and the [JSON Spec for CloudEvents](https://github.com/cloudevents/spec/blob/main/cloudevents/formats/json-format.md) for more details.\n\nCloudEvents with CLI\n```\n$ rubin -topic public.hello -record '{\"action\":\"push\"}' \\\n    -ce -type \"events.published\" -source \"/ci/build/123\" -subject \"artifact.zip\"\n```\nCloudEvents with library\n```\npayload := map[string]string{\"user\": \"james.bond\", \"id\": \"007\"}\nevent, err = rubin.NewCloudEvent(\"//hr/manager\", \"user.created\", payload)\nresp, err := client.Produce(ctx, Request{ \"app.user\",  event})\n```\nResulting event structure (CLI example code)\n```\n{\n  \"specversion\": \"1.0\",\n  \"id\": \"5b4eefde-bf4a-4d48-8f47-e2978df8d139\",\n  \"source\": \"/ci/build/123\",\n  \"type\": \"events.published\",\n  \"subject\": \"artifact.zip\",\n  \"datacontenttype\": \"application/json\",\n  \"time\": \"2023-11-04T14:11:33Z\",\n  \"data\": {\n\t\"action\": \"push\"\n  }\n }\n```\n\n## 🎸 Why the funky name?\n\nInitially I thought of technical names like `kafka-record-prodcer` or `topic-pusher`, but all of them turned out to be pretty boring. [Rick Rubin](https://en.wikipedia.org/wiki/Rick_Rubin) was simply the first name that showed up when I googled for \"famous record producers\", so I named the tool in his honour, and also in honour of the great Albums he produced in the past decades.\n\n\u003ca href=\"https://de.wikipedia.org/wiki/Rick_Rubin\"\u003e\n \u003cimg alt='rick-rubin' src=\"https://upload.wikimedia.org/wikipedia/commons/archive/4/43/20210617192624%21RickRubinSept09.jpg\" \u003e\n\u003c/a\u003e\n  \n*\"Rick Rubin in September 2006\" by jasontheexploder is licensed under [CC BY 3.0](https://creativecommons.org/licenses/by/3.0/)*\n\n\n## Development\n\nThe project uses `make` to make your life easier. If you're not familiar with Makefiles you can take a look at [this quickstart guide](https://makefiletutorial.com).\n\nWhenever you need help regarding the available actions, just use the following command.\n\n```\n$ make help\n\nUsage: make \u003cOPTIONS\u003e ... \u003cTARGETS\u003e\n\nAvailable targets are:\n\nall                  Initializes all tools\nbuild                Builds all binaries\nci                   Executes lint and test and generates reports\nclean                Cleans up everything\ncoverage             Displays coverage per func on cli\ndocker               Builds docker image\ndownload             Downloads the dependencies\nfmt                  Formats all code with go fmt\nhelp                 Shows the help\nhtml-coverage        Displays the coverage results in the browser\nlint                 Lints all code with golangci-lint\nrun                  Run the app\nrun-help             Run the app and display app helm\ntest                 Runs all tests  (with colorized output support if gotest is installed)\ntest-build           Tests whether the code compiles\ntest-int             Run integration test with tag //go:build integration\ntidy                 Cleans up go.mod and go.sum\n```\n\n## API stability\n\nThe package API for rubin is still version zero and therefore not yet considered stable as described in [gopkg.in](https://gopkg.in)\n\n## Credits\n\nThis project was bootstrapped with [go/template](https://github.com/SchwarzIT/go-template), an efficient tool to \"provides a blueprint for production-ready Go project layouts.\"\n\n## Contribution\nIf you want to contribute to *rubin* please have a look at the [contribution guidelines](./CONTRIBUTING.md).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftillkuhn%2Frubin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftillkuhn%2Frubin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftillkuhn%2Frubin/lists"}