{"id":13614478,"url":"https://github.com/etf1/kafka-transformer","last_synced_at":"2025-12-15T16:04:37.383Z","repository":{"id":55466253,"uuid":"247823559","full_name":"etf1/kafka-transformer","owner":"etf1","description":"kafka-transformer is a library for simplyfing message transformation tasks when using kafka","archived":false,"fork":false,"pushed_at":"2024-07-03T07:53:11.000Z","size":139,"stargazers_count":17,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"develop","last_synced_at":"2025-08-15T14:59:16.177Z","etag":null,"topics":["go","golang","kafka"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/etf1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-16T21:52:41.000Z","updated_at":"2025-05-22T08:23:33.000Z","dependencies_parsed_at":"2024-11-07T22:34:14.950Z","dependency_job_id":"48885172-1313-43ac-984a-db124622380d","html_url":"https://github.com/etf1/kafka-transformer","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/etf1/kafka-transformer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/etf1%2Fkafka-transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/etf1%2Fkafka-transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/etf1%2Fkafka-transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/etf1%2Fkafka-transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/etf1","download_url":"https://codeload.github.com/etf1/kafka-transformer/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/etf1%2Fkafka-transformer/sbom","scorecard":{"id":383741,"data":{"date":"2025-08-11","repo":{"name":"github.com/etf1/kafka-transformer","commit":"ca5bf454f936da3176f310562e3d5238f923e8e6"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.8,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":3,"reason":"Found 9/23 approved changesets -- score normalized to 3","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/go.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/go.yml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/etf1/kafka-transformer/go.yml/develop?enable=pin","Warn: containerImage not pinned by hash: Dockerfile.test:3","Warn: goCommand not pinned by hash: Dockerfile.test:7","Info:   0 out of   1 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned","Info:   0 out of   1 goCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENCE:0","Info: FSF or OSI recognized license: MIT License: LICENCE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during GetBranch(master): error during branchesHandler.query: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":7,"reason":"3 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0322 / GHSA-cg3q-j54f-5p7p","Warn: Project is vulnerable to: GO-2024-2611 / GHSA-8r3f-844c-mc37","Warn: Project is vulnerable to: GO-2022-0603 / GHSA-hp87-p4gw-j4gq"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 16 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-18T16:06:09.557Z","repository_id":55466253,"created_at":"2025-08-18T16:06:09.557Z","updated_at":"2025-08-18T16:06:09.557Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27753700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-15T02:00:09.782Z","response_time":96,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","kafka"],"created_at":"2024-08-01T20:01:01.944Z","updated_at":"2025-12-15T16:04:37.353Z","avatar_url":"https://github.com/etf1.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"kafka-transformer is a library for simplyfing message transformation tasks when using kafka (written proudly in GO :) ).\n\n![Go](https://github.com/etf1/kafka-transformer/workflows/Go/badge.svg)\n\n# Overview\n\nThe main goal of this project is to be able to consume messages from a topic, perform a customizable transformation and project the result to another topic via a kafka producer or to an external system with a custom projector.\n\nThe user of this library provides a transformation by providing an implementation of the interface `Transformer` as described below:\n\n```golang\ntype Transformer interface {\n\tTransform(src *kafka.Message) []*kafka.Message\n}\n```\n\n\n```\n                            +-------------------+\n+------------------+        |                   |         +------------------------------+\n|                  |        |                   |         |                              |\n|       topic      +-------\u003e+ kafka-transformer +--------\u003e+     topic or external        |\n|                  |        |                   |         |                              |\n+------------------+        |                   |         +------------------------------+\n                            +-------------------+\n\n```\n\n# Design pattern\n\n- Passthrough: 1 topic -\u003e 1 topic\n- Splitter:    1 topic -\u003e n topics (transformer returns n messages with different topics)\n- Projector:   1 topic -\u003e n external systems\n\n# Architecture\n\nkafka-transformer is based on the confluent kafka client: [confluent-kafka-go](https://github.com/confluentinc/confluent-kafka-go).\n\nImplementation details: \n\n- kafka-transformer is based confluent-kafka-go which is a wrapper around the C library librdkafka (for more [details](https://github.com/confluentinc/confluent-kafka-go)).\n- kafka-transformer is using a double buffering mechanism between the three main components: \n    * consumer\n    * transformer with workers (pool of goroutines for parallel transformation)\n    * projector (kafka producer of custom projector)\n- Each of this components are communicated through channels. For more details check the source code.\n\n# Example\n\nYou can find examples in [examples](examples/).\n\nBasically you can run your kafka-transformer as shown below:\n\n```golang\npackage main\n\nimport \"github.com/etf1/kafka-transformer/pkg/kafka\"\n\nfunc main(){\n\n    transformer, err := kafka.NewKafkaTransformer(config)\n\tif err != nil {\n\t\tlog.Fatalf(\"failed to create transformer: %v\", err)\n\t}\n    // blocking call\n    transformer.Run()\n}\n```\n\nYou have to provide a configuration `Config`: \n\n```golang\ntype Config struct {\n\tSourceTopic    string\n\tConsumerConfig *confluent.ConfigMap\n\tProducerConfig *confluent.ConfigMap\n\tTransformer    transformer.Transformer\n\tLog            logger.Log\n}\n```\n\n- SourceTopic: source topic where the messages will be read from.\n- ConsumerConfig: configuration for the consumer (see [documentation](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md))\n- ProducerConfig: configuration for the kafka producer. (see [documentation](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)). If you want custom projection provide a Projector (see below). \n- Transformer: implementation of the `Transformer` interface.\n- Log: the implementation of the `Log` interface, if you want to use your own logger.\n- Projector : implementation of the `Projector` interface for custom projection to an external system (eg redis). This option is incompatible with `ProducerConfig` (see [custom_projector](examples/custom_projector))\n\nBelow an example of a configuration:\n\n```golang\nbroker := \"localhost:9092\"\nconfig := kafka.Config{\n    SourceTopic: \"source-topic\",\n    ConsumerConfig: \u0026confluent.ConfigMap{\n        \"bootstrap.servers\":     broker,\n        \"broker.address.family\": \"v4\",\n        \"group.id\":              \"custom-transformer\",\n        \"session.timeout.ms\":    6000,\n        \"auto.offset.reset\":     \"earliest\",\n    },\n    ProducerConfig: \u0026confluent.ConfigMap{\n        \"bootstrap.servers\": broker,\n    },\n    Transformer: customTransformer{},\n}\n\n```\n\n# Custom transformer\n\nYou can provide your own transformation by providing an implementation of the Transformer interface:\n\n```golang\ntype Transformer interface {\n\tTransform(src *kafka.Message) []*kafka.Message\n}\n```\n\nHere is a usage example:\n\n```golang\ntype customTransformer struct{}\n\nfunc (ct customTransformer) Transform(src *kafka.Message) []*kafka.Message {\n\ttopic := \"custom-transformer\"\n\tdst := \u0026kafka.Message{\n\t\tTopicPartition: kafka.TopicPartition{\n\t\t\tTopic:     \u0026topic,\n\t\t\tPartition: kafka.PartitionAny,\n\t\t},\n\t\tValue: append([]byte(\"New value:\"), src.Value...),\n\t}\n\n\treturn []*kafka.Message{dst}\n}\n\n```\n\nIf you want to create multiple messages from one source message, you can achieve it in the transform function. If the messages has different topics, each message will be publish to its specified topic\n\n\n# Custom projector\n\nYou can provide your own projection, if you don't want to project your message to kafka but to an external system like a database or a cache (redis). You need to implement the Projector interface. \n\n```golang\ntype Projector interface {\n\tProject(message *kafka.Message)\n}\n```\nHere is an example which stores the messages in a slice (in memory):\n\n```golang\ntype sliceProjector struct {\n\tmsgs\t[]*kafka.Message\n}\n\nfunc (sp *sliceProjector) Project(msg *kafka.Message) {\n\tsp.msgs = append(sp.msgs, msg)\n}\n\n```\n\n# Instrumentation\n\nIf you need to instrument each main actions of the kafka transformer, you can provide an implementation of the Collector interface:\n\n```golang\ntype Collector interface {\n\tBefore(message *confluent.Message, action Action, start time.Time)\n\tAfter(message *confluent.Message, action Action, err error, start time.Time)\n}\n```\n\nAvailable actions for a Collector are:\n\t- KafkaConsumerConsume: when a message is consumed\n\t- KafkaProducerProduce: when a message is produced\n\t- TransformerTransform: when a message is transformed\n\t- ProjectorProject: when a message is projected (and not produced to kafka)\n\nYou can find examples in [examples](examples/).\n\n# Requirements\n\n* Go 1.13 (minimum)\n* install [librdkafka](https://github.com/confluentinc/confluent-kafka-go#installing-librdkafka)\n\nfor development:\n\n* docker\n* docker-compose\n\n# Local development\n\n* git clone git@github.com:etf1/kafka-transformer.git\n* to start local development environment (mainly kafka), run `make dev.up`, your kafka broker will be available at `localhost:9092`\n* to run tests `make tests`\n* to stop local environment, run `make dev.down`\n\n# OpenTelemetry support\n\nAn [OpenTelemetry](https://opentelemetry.io/) instrumentation is available under `instrumentation/otel/` directory.\n\nIt is a separated Go module you can import in your project:\n\n```bash\n$ go get github.com/etf1/kafka-transformer/instrumentation/otel\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fetf1%2Fkafka-transformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fetf1%2Fkafka-transformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fetf1%2Fkafka-transformer/lists"}