{"id":25176697,"url":"https://github.com/conduitio/conduit-processor-sdk","last_synced_at":"2025-06-14T16:01:59.399Z","repository":{"id":211443415,"uuid":"729124241","full_name":"ConduitIO/conduit-processor-sdk","owner":"ConduitIO","description":"SDK for Conduit processors written in Go","archived":false,"fork":false,"pushed_at":"2025-06-02T14:13:04.000Z","size":506,"stargazers_count":2,"open_issues_count":3,"forks_count":1,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-06-03T02:58:44.974Z","etag":null,"topics":["conduit","go","golang","processor"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ConduitIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-12-08T13:22:26.000Z","updated_at":"2025-05-19T22:49:48.000Z","dependencies_parsed_at":"2024-06-10T16:23:05.565Z","dependency_job_id":"4ee28bbf-7af0-46a3-8b3e-16ee6a30ca5a","html_url":"https://github.com/ConduitIO/conduit-processor-sdk","commit_stats":null,"previous_names":["conduitio/conduit-processor-sdk"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/ConduitIO/conduit-processor-sdk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-processor-sdk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-processor-sdk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-processor-sdk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-processor-sdk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ConduitIO","download_url":"https://codeload.github.com/ConduitIO/conduit-processor-sdk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-processor-sdk/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259843302,"owners_count":22920302,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["conduit","go","golang","processor"],"created_at":"2025-02-09T13:17:55.510Z","updated_at":"2025-06-14T16:01:59.332Z","avatar_url":"https://github.com/ConduitIO.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Conduit Processor SDK\n\n[![License](https://img.shields.io/badge/license-Apache%202-blue)](https://github.com/ConduitIO/conduit-processor-sdk/blob/main/LICENSE.md)\n[![Test](https://github.com/ConduitIO/conduit-processor-sdk/actions/workflows/test.yml/badge.svg)](https://github.com/ConduitIO/conduit-processor-sdk/actions/workflows/test.yml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/conduitio/conduit-processor-sdk)](https://goreportcard.com/report/github.com/conduitio/conduit-processor-sdk)\n[![Go Reference](https://pkg.go.dev/badge/github.com/conduitio/conduit-processor-sdk.svg)](https://pkg.go.dev/github.com/conduitio/conduit-processor-sdk)\n\nThis repository contains the Go software development kit for implementing a\nprocessor for [Conduit](https://github.com/conduitio/conduit).\n\nNote: if you'd like to use another language for writing processors, feel free to\n[open an issue](https://github.com/ConduitIO/conduit/issues/new?assignees=\u0026labels=feature%2Ctriage\u0026projects=\u0026template=1-feature-request.yml\u0026title=Feature%3A+%3Ctitle%3E) and request a processor SDK for a specific language.\n\n## Quick Start\n\nCreate a new folder and initialize a fresh go module:\n\n```sh\ngo mod init example.com/conduit-processor-example\n```\n\nAdd the processor SDK dependency:\n\n```sh\ngo get github.com/conduitio/conduit-processor-sdk\n```\n\nYou can now create a new processor by implementing the\n[`Processor`](https://pkg.go.dev/github.com/conduitio/conduit-processor-sdk#Processor)\ninterface. For more details about that, check our documentation for \n[Building Standalone Processors](https://conduit.io/docs/processors/standalone/building).\n\nOn the other hand, if the processor is very simple and can be reduced to a single function (e.g. \nno configuration needed), then we can use `sdk.NewProcessorFunc()`, as below:\n\n```go\n//go:build wasm\n\npackage main\n\nimport (\n    sdk \"github.com/conduitio/conduit-processor-sdk\"\n)\n\nfunc main() {\n    sdk.Run(sdk.NewProcessorFunc(\n        sdk.Specification{Name: \"example-processor\"},\n        func(ctx context.Context, rec opencdc.Record) (opencdc.Record, error) {\n            // do something with the record\n            return rec, nil\n        },\n    ))\n}\n```\n\nWith this, you are set to build your processor. Note that we are building the\nprocessor as a WebAssembly module, so you need to set `GOARCH` and `GOOS`:\n\n```sh\nGOARCH=wasm GOOS=wasip1 go build -o example-processor.wasm main.go\n```\n\nThe produced `example-processor.wasm` file can be used as a processor in\nConduit. Copy it to the `processors` directory of your Conduit instance and\nconfigure it in the `processors` section of the pipeline configuration file.\n\n## FAQ\n\n### Why do I need to specify `GOARCH` and `GOOS`?\n\nConduit uses [WebAssembly](https://webassembly.org) to run standalone processors.\nThis means that you need to build your processor as a WebAssembly module. You can\ndo this by setting the environment variables `GOARCH=wasm` and `GOOS=wasip1` when\nrunning `go build`. This will produce a WebAssembly module that can be used as a\nprocessor in Conduit.\n\n### How do I use a processor?\n\nTo use a standalone WASM processor in Conduit, the following two steps need to be\ndone:\n\n1. Copying the WebAssembly module to the processors directory of your Conduit\n   instance. By default, that's a directory called `processors` that is in the same\n   directory as Conduit. The directory can be changed with the `-processors.path` flag.\n\n   An example directory structure would be:\n   ```shell\n   .\n   ├── conduit\n   └── processors\n       └── example-processor.wasm\n   ```\n2. Use the processor in the `processors` section of the pipeline configuration file.\n   using the name the processor defines in its specifications. For example:\n   ```yaml\n   processors:\n     - id: my-example-processor\n       plugin: example-processor\n       settings:\n         field: 'foo'\n         value: 'bar'\n   ```\n\n### How do I log in a processor?\n\nYou can get a [`zerolog.Logger`](https://pkg.go.dev/github.com/rs/zerolog) instance\nfrom the context using the\n[`sdk.Logger`](https://pkg.go.dev/github.com/conduitio/conduit-processor-sdk#Logger)\nfunction. This logger is pre-configured to append logs in the format expected by\nConduit.\n\nKeep in mind that logging in the hot path (i.e. in the `Process` method) can have\na significant impact on the performance of your processor, therefore we recommend\nto use the `Trace` level for logs that are not essential for the operation of the\nprocessor.\n\nExample:\n\n```go\nfunc (p *ExampleProcessor) Process(ctx context.Context, records []opencdc.Record) []sdk.ProcessedRecord {\n    logger := sdk.Logger(ctx)\n    logger.Trace().Msg(\"Processing records\")\n    // ...\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduitio%2Fconduit-processor-sdk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconduitio%2Fconduit-processor-sdk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduitio%2Fconduit-processor-sdk/lists"}