{"id":42358292,"url":"https://github.com/fluffy-bunny/benthos-cloudevents-fluffycore","last_synced_at":"2026-01-27T16:37:35.816Z","repository":{"id":206594756,"uuid":"711520008","full_name":"fluffy-bunny/benthos-cloudevents-fluffycore","owner":"fluffy-bunny","description":"Benthos plugin that processes cloudevents and sends them downstream to a gprc handler.  ","archived":false,"fork":false,"pushed_at":"2024-04-13T18:17:00.000Z","size":336,"stargazers_count":0,"open_issues_count":10,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2024-06-21T21:06:19.145Z","etag":null,"topics":["benthos","benthos-plugin","dependency-injection","docker","golang","kafka"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fluffy-bunny.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-29T14:19:52.000Z","updated_at":"2023-11-11T01:43:34.000Z","dependencies_parsed_at":"2024-05-11T17:15:18.589Z","dependency_job_id":null,"html_url":"https://github.com/fluffy-bunny/benthos-cloudevents-fluffycore","commit_stats":null,"previous_names":["fluffy-bunny/benthos-cloudevents-fluffycore"],"tags_count":58,"template":false,"template_full_name":null,"purl":"pkg:github/fluffy-bunny/benthos-cloudevents-fluffycore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluffy-bunny%2Fbenthos-cloudevents-fluffycore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluffy-bunny%2Fbenthos-cloudevents-fluffycore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluffy-bunny%2Fbenthos-cloudevents-fluffycore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluffy-bunny%2Fbenthos-cloudevents-fluffycore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fluffy-bunny","download_url":"https://codeload.github.com/fluffy-bunny/benthos-cloudevents-fluffycore/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluffy-bunny%2Fbenthos-cloudevents-fluffycore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28816563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-27T12:25:15.069Z","status":"ssl_error","status_checked_at":"2026-01-27T12:25:05.297Z","response_time":168,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benthos","benthos-plugin","dependency-injection","docker","golang","kafka"],"created_at":"2026-01-27T16:37:34.995Z","updated_at":"2026-01-27T16:37:35.802Z","avatar_url":"https://github.com/fluffy-bunny.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# benthos-cloudevents-fluffycore\n\nA [Benthos](https://www.benthos.dev/) plugin that processes cloudevents and sends them downstream to a gprc handler.  \n\n## CloudEvents\n\n[CloudEvents Specification](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md)  \n\n## Pipeline\n\n```mermaid\ngraph TD;\n    cloud_event_producer--\u003eKafka;\n    Kafka--\u003eBenthos.Input.kafka;\n    Benthos.Input.kafka--\u003eBenthos.Ouput.cloudevents;\n    Benthos.Ouput.cloudevents--\u003eDownstream.GRPC.Service;\n```\n\n### CloudEvent\n\n```json\n{\n    \"id\": \"1234\",\n    \"spec_version\": \"1.0\",\n    \"type\": \"my.type\",\n    \"source\": \"//my/source\",\n    \"text_data\": \"{\\\"a\\\":\\\"b\\\"}\",\n    \"attributes\": [\n        {\n            \"value\": {\n                \"ce_string\": \"ORG1234abcd\"\n            },\n            \"key\": \"orgid\"\n        },\n        {\n            \"value\": {\n                \"ce_string\": \"ORG1234abcd\"\n            },\n            \"key\": \"partition-key\"\n        }\n    ]\n}\n```\n### CloudEvents in Kafka\n\n| Kafka Part | data         |\n| ---        | ---          |\n| Key        | ORG1234abcd  |\n| Headers    | \u003cpre\u003e{\u003cbr\u003e  \"ce_type\":\"my.type\",\u003cbr\u003e  \"ce_orgid\":\"ORG1234abcd\",\u003cbr\u003e  \"ce_source\":\"//my/source\",\u003cbr\u003e  \"ce_specversion\":\"1.0\",\"ce_time\":\"2023-11-11T14:57:14.594525315Z\",\u003cbr\u003e  \"ce_id\":\"1234\",\u003cbr\u003e  \"content-type\":\"application/json\"\u003cbr\u003e}\u003c/pre\u003e|\n| Value   | \u003cpre\u003e{\u003cbr\u003e  \"a\":\"b\"\u003cbr\u003e}\u003c/pre\u003e|\n\n\n### Benthos kafka batching\n\nEach batch item must retain its headers and value in one json object.  I achieve this by doing a ```processors``` mapping which then gets packaged up as a binary array. ```[][]bytes```  \n\n```yaml\nhttp:\n  enabled: false\ninput:\n  kafka:\n    addresses: [\"${INPUT_KAFKA_BROKERS}\"]\n    topics: [\"cloudevents-core\"]\n    consumer_group: \"$Default2\"\n    multi_header: true\n    batching:\n      count: 3\n      period: 60s\n      processors:\n        - json_schema:\n            schema: '{\"$schema\":\"http://json-schema.org/draft-04/schema#\",\"type\":\"object\",\"properties\":{\"id\":{\"type\":\"string\"},\"source\":{\"type\":\"string\"},\"specVersion\":{\"type\":\"string\",\"enum\":[\"1.0\"]},\"type\":{\"type\":\"string\",\"enum\":[\"requestunits.v1\"]},\"attributes\":{\"type\":\"object\",\"properties\":{\"orgid\":{\"type\":\"object\",\"properties\":{\"ceString\":{\"type\":\"string\"}},\"required\":[\"ceString\"]},\"time\":{\"type\":\"object\",\"properties\":{\"ceTimestamp\":{\"type\":\"string\"}},\"required\":[\"ceTimestamp\"]}},\"required\":[\"orgid\",\"partition-key\",\"time\"]},\"textData\":{\"type\":\"string\"}},\"required\":[\"id\",\"source\",\"specVersion\",\"type\",\"attributes\",\"textData\"]}'\n        - switch:\n            - check: errored()\n              processors:\n                - for_each:\n                    - while:\n                        at_least_once: true\n                        max_loops: 0\n                        check: errored()\n                        processors:\n                          - catch: [] # Wipe any previous error\n                          - mapping: \"errorlogit(@,content())\"\n                - mapping: |\n                    deleted()\n\n        - archive:\n            format: binary\n\npipeline:\n  threads: 1\n  processors:\n    - sleep:\n        duration: 1s\n\noutput:\n  cloudevents_grpc:\n    grpc_url: \"${OUTPUT_CLOUDEVENTOUTPUT_GRPC_URL}\"\n    max_in_flight: 64\n    channel: \"mychannel\"\n\n    # auth[optional] one of: [oauth2,basic,apikey](ordered by priority)\n    #--------------------------------------------------------------------\n    #auth:\n    #  basic:\n    #    user_name: \"admin\"\n    #    password: \"password\"\n    #  oauth2:\n    #    client_id: \"my_client_id\"\n    #    client_secret: \"secret\"\n    #    token_endpoint: \"https://example.com/oauth2/token\"\n    #    scopes: [\"scope1\", \"scope2\"]\n    #  apikey:\n    #    name: \"x-api-key\"\n    #    value: \"secret\"\nlogger:\n  level: ${LOG_LEVEL}\n  format: json\n  add_timestamp: true\n  static_fields:\n    \"@service\": benthos.kafka\n```\n\nThe custom output gets messages via benthos in the following json format\n\n```json\n[\n    {\n        \"headers\": {\n            \"ce_id\": \"1234\",\n            \"ce_orgid\": \"ORG1234abcd\",\n            \"ce_source\": \"//my/source\",\n            \"ce_specversion\": \"1.0\",\n            \"ce_time\": \"2023-11-11T14:57:14.594525315Z\",\n            \"ce_type\": \"my.type\",\n            \"content-type\": \"application/json\",\n            \"kafka_key\": \"ORG1234abcd\",\n            \"kafka_lag\": 1,\n            \"kafka_offset\": 0,\n            \"kafka_partition\": 0,\n            \"kafka_timestamp_unix\": 1699714634,\n            \"kafka_tombstone_message\": false,\n            \"kafka_topic\": \"cloudevents-core\"\n        },\n        \"value\": {\n            \"a\": \"b\"\n        }\n    },\n    {\n        \"headers\": {\n            \"ce_id\": \"1234\",\n            \"ce_orgid\": \"ORG1234abcd\",\n            \"ce_source\": \"//my/source\",\n            \"ce_specversion\": \"1.0\",\n            \"ce_time\": \"2023-11-11T15:34:48.621943865Z\",\n            \"ce_type\": \"my.type\",\n            \"content-type\": \"application/json\",\n            \"kafka_key\": \"ORG1234abcd\",\n            \"kafka_lag\": 0,\n            \"kafka_offset\": 1,\n            \"kafka_partition\": 0,\n            \"kafka_timestamp_unix\": 1699716888,\n            \"kafka_tombstone_message\": false,\n            \"kafka_topic\": \"cloudevents-core\"\n        },\n        \"value\": {\n            \"a\": \"b\"\n        }\n    }\n]\n```\n## Downstream GRPC Handler\n\n[cloudeventprocessor proto](./pkg/proto/cloudeventprocessor/cloudeventprocessor.proto)  \n\nThe ```cloudevents``` plugin processes and groups the messages into Good and Bad buckets and sends them both downstream.  It is an all of nothing acknowledgement from the downstream processor if the messages where handled.  Returning an error will result in the messages being sent again.  It is the responsiblity of the downstream processor to deal with bad data.  So putting bad data on a dead-letter queue is not something we will do here in benthos.\n\nYour downstream processor can have 4 types of authentication.  ```None, OAuth2, ApiKey or basic auth.```\n\n```yaml\noutput:\n  cloudevents: \n    grpc_url: \"localhost:9050\"\n    max_in_flight: 64\n\n    # auth[optional] one of: [oauth2,basic,apikey](ordered by priority)\n    #--------------------------------------------------------------------\n    #auth:\n    #  basic:\n    #    user_name: \"admin\"\n    #    password: \"password\"\n    #  oauth2:\n    #    client_id: \"my_client_id\"\n    #    client_secret: \"secret\"\n    #    token_endpoint: \"https://example.com/oauth2/token\"\n    #    scopes: [\"scope1\", \"scope2\"]\n    #  apikey:\n    #    name: \"x-api-key\"\n    #    value: \"secret\"\n```\n\n## Build the proto\n\n[grpc.io](https://grpc.io/docs/languages/go/basics/)  \n[Transcode Ref](https://grpc-ecosystem.github.io/grpc-gateway/docs/tutorials/introduction/)  \n[custom protoc plugin](https://rotemtam.com/2021/03/22/creating-a-protoc-plugin-to-gen-go-code/)  \n\n```bash\ngo install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@latest\ngo install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2@latest\ngo install google.golang.org/protobuf/cmd/protoc-gen-go@latest\ngo install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest\ngo install github.com/fluffy-bunny/fluffycore/protoc-gen-go-fluffycore-di/cmd/protoc-gen-go-fluffycore-di@latest\n```\n\n```bash\nprotoc --go_out=. --go_opt=paths=source_relative ./pkg/proto/cloudevents/cloudevents.proto  \n\nprotoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative --go-fluffycore-di_out=.  --go-fluffycore-di_opt=paths=source_relative ./pkg/proto/cloudeventprocessor/cloudeventprocessor.proto \n\nprotoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative --go-fluffycore-di_out=.  --go-fluffycore-di_opt=paths=source_relative ./pkg/proto/kafkacloudevent/kafkacloudevent.proto \n\n```\n\n## Grpc References\n\n[auth examples](https://github.com/johanbrandhorst/grpc-auth-example)  \n\n\n## Docker\n\n```bash\n docker build --file .\\build\\Dockerfile.processor . --tag fluffybunny.benthos.processor\n\n docker build --file .\\build\\Dockerfile.benthos . --tag fluffybunny.benthos.benthos\n\n docker-compose up\n```\n\n## Services\n\n### Kafka UI\n\n[Kafka-ui](http://localhost:9099/)  \n\n### KafkaCloudEventService - grpc\n\n```bash\ngrpc://localhost:9050\n```\n\nThis is service to submit a cloud-event to kafka.  \n\n#### Request\n\n```json\n{\n    \"batch\": {\n        \"events\": [\n            {\n                \"attributes\": [\n                    {\n                        \"value\": {\n                            \"ce_string\": \"ORG1234abcd\"\n                        },\n                        \"key\": \"orgid\"\n                    },\n                    {\n                        \"value\": {\n                            \"ce_string\": \"ORG1234abcd\"\n                        },\n                        \"key\": \"partition-key\"\n                    }\n                ],\n                \"spec_version\": \"1.0\",\n                \"text_data\": \"{\\\"a\\\":\\\"b\\\"}\",\n                \"id\": \"1234\",\n                \"type\": \"my.type\",\n                \"source\": \"//my/source\"\n            }\n        ]\n    }\n}\n```\n\n#### Response \n\n```json\n{}\n```\n\n### CloudEventProcessor - grpc\n\n```bash\ngrpc://localhost:9050\n```\n\nThis service receives cloud-events as batches via out custom benthos output handler.\n\n**IMPORTANT**: This is an all or nothing process.  It sends the messages in batches.  There are good and bad and the processor needs to decide what it wants to do with the bad messages.  In some cases the entire batch is bad at the \n\n### Viewer\n\nI use docker desktop and look at the logs.  My downstream processor just logs what it got and returns a nil.  \n\n## Benthos Notes\n\nValidating Data in the input.\nI want to write my custom output pluging to expect only perfect data.\n\n```\ninput:\n  kafka:\n    addresses: [\"kafka:9092\"]\n    topics: [\"cloudevents-core\"]\n    consumer_group: \"$Default\"\n    batching:\n      count: 2\n      period: 20s\n      processors:\n        - mapping: |\n            root.value = this\n            root.headers = @   \n        - archive:\n            format: json_array\n```\nI expect a json, and to make this resilient I put not-json into a kafka message.\nI got the following errors from benthos(good)\n``` \nlevel=error msg=\"failed assignment (line 1): unable to reference message as structured (with 'this'): message is empty\" @service=benthos label=\"\" path=root.input.batching.processors.0\nlevel=error msg=\"Failed to create archive: failed to parse message as JSON: target message part does not exist\" @service=benthos label=\"\" path=root.input.batching.processors.1\n```\nI imagine that I could do something in the input.processors to validate an individual message.  \nUpon NOT getting the headers and value I expect, I either eat it as handled or wrap it into a json that my output can now deal with and deadletter.   I think wrapping would be great.  If  root.value = this is bad, then turn it into {\"crap\":\"encoded crap\"}\n\nThat way I can still get an  json array, but can evaluate each item looking for crap.\n\nOn a side note I REALLY LIKE how y'all are handling errors.\nSo my batch setting is 3.\nin kafka I have [crap]-[good]-[good]-[good]-[good]-[good]\nWhen benthos sees this it eats the [crap] and then sends the next 2 [good] one at a time.  \nI just need to code up that I may not get an array sometimes but may get the object instead.\n\nI then start getting arrays of 3.  \nNOICE!\n\neats the [crap]  =\u003e as a result of \n```\nlevel=error msg=\"failed assignment (line 2): unable to reference message as structured (with 'this'): parse as json: invalid character 'a' looking for beginning of value\" @service=benthos label=\"\" path=root.input.batching.processors.0\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluffy-bunny%2Fbenthos-cloudevents-fluffycore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffluffy-bunny%2Fbenthos-cloudevents-fluffycore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluffy-bunny%2Fbenthos-cloudevents-fluffycore/lists"}