{"id":25176696,"url":"https://github.com/conduitio/conduit-connector-sdk","last_synced_at":"2025-05-06T23:21:19.164Z","repository":{"id":37083668,"uuid":"459546102","full_name":"ConduitIO/conduit-connector-sdk","owner":"ConduitIO","description":"SDK for Conduit connectors written in Go","archived":false,"fork":false,"pushed_at":"2025-05-06T15:14:51.000Z","size":805,"stargazers_count":10,"open_issues_count":13,"forks_count":5,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-05-06T16:44:54.779Z","etag":null,"topics":["conduit","go","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ConduitIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-02-15T11:17:12.000Z","updated_at":"2025-05-06T15:14:55.000Z","dependencies_parsed_at":"2023-11-14T15:27:59.824Z","dependency_job_id":"43a871ee-8e76-4912-ab1b-f5a2c0c49a94","html_url":"https://github.com/ConduitIO/conduit-connector-sdk","commit_stats":{"total_commits":108,"total_committers":7,"mean_commits":"15.428571428571429","dds":0.5462962962962963,"last_synced_commit":"b8fbf8dadd013a7d060fded7731f72f695333e94"},"previous_names":["conduitio/connector-plugin-sdk","conduitio/conduit-plugin-sdk"],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-connector-sdk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-connector-sdk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-connector-sdk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-connector-sdk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ConduitIO","download_url":"https://codeload.github.com/ConduitIO/conduit-connector-sdk/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252783881,"owners_count":21803571,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["conduit","go","golang"],"created_at":"2025-02-09T13:17:54.946Z","updated_at":"2025-05-06T23:21:19.147Z","avatar_url":"https://github.com/ConduitIO.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Conduit Connector SDK\n\n[![License](https://img.shields.io/badge/license-Apache%202-blue)](https://github.com/ConduitIO/conduit-connector-sdk/blob/main/LICENSE.md)\n[![Test](https://github.com/ConduitIO/conduit-connector-sdk/actions/workflows/test.yml/badge.svg)](https://github.com/ConduitIO/conduit-connector-sdk/actions/workflows/test.yml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/conduitio/conduit-connector-sdk)](https://goreportcard.com/report/github.com/conduitio/conduit-connector-sdk)\n[![Go Reference](https://pkg.go.dev/badge/github.com/conduitio/conduit-connector-sdk.svg)](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk)\n\nThis repository contains the Go software development kit for implementing a connector for\n[Conduit](https://github.com/conduitio/conduit). If you want to implement a connector in another language please\nhave a look at the [connector protocol](https://github.com/conduitio/conduit-connector-protocol).\n\n## Quickstart\n\nCreate a new folder and initialize a fresh go module:\n```\ngo mod init example.com/conduit-connector-demo\n```\n\nAdd the connector SDK dependency:\n\n```\ngo get github.com/conduitio/conduit-connector-sdk\n```\n\nWith this you can start implementing the connector. To implement a source (a connector that reads from a 3rd party\nresource and sends data to Conduit) create a struct that implements\n[`sdk.Source`](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Source). To implement a destination (a\nconnector that receives data from Conduit and writes it to a 3rd party resource) create a struct that implements\n[`sdk.Destination`](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Destination). You can implement both to\nmake a connector that can be used both as a source or a destination.\n\nApart from the source and/or destination you should create a global variable of type\n[`sdk.Connector`](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Connector) that contains references to\nconstructors for [`sdk.Source`](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Source),\n[`sdk.Destination`](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Destination) and\n[`sdk.Specification`](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Specification).\n\nThe last part is the entrypoint, it needs to call\n[`sdk.Serve`](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Serve) and pass in the connector\nmentioned above.\n\n```go\npackage main\n\nimport (\n\tdemo \"example.com/conduit-connector-demo\"\n\tsdk \"github.com/conduitio/conduit-connector-sdk\"\n)\n\nfunc main() {\n\tsdk.Serve(demo.Connector)\n}\n```\n\nNow you can build the standalone connector:\n\n```\ngo build path/to/main.go\n```\n\nYou will get a compiled binary which Conduit can use as a connector. To run your connector as part of a Conduit pipeline you\ncan create it using the connectors API and specify the path to the compiled connector binary in the field `plugin`.\n\nHere is an example request to `POST /v1/connectors` (find more about the [Conduit API](https://github.com/conduitio/conduit#api)):\n\n```json\n{\n  \"type\": \"TYPE_SOURCE\",\n  \"plugin\": \"/path/to/compiled/connector/binary\",\n  \"pipelineId\": \"...\",\n  \"config\": {\n    \"name\": \"my-connector\",\n    \"settings\": {\n      \"my-key\": \"my-value\"\n    }\n  }\n}\n```\n\nFind out more information on building a connector in the [Go doc reference](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk).\n\n## FAQ\n\n**Q: How to identify the source from which a record originated?**\n\nA connector can use whatever means available to associate a record with the\noriginating source. However, to promote compatibility between connectors, we \n**highly recommend** that a record's metadata is used to indicate from which\n_collection_[^1] a record originated.\n\nThe metadata key to be used is `opencdc.collection`, which can be accessed\nthrough the `sdk.MetadataCollection` constant.\n\nFor example, if a record was read from a database table called `employees`, it\nshould have the following in its metadata:\n```json5\n{\n  \"opencdc.collection\": \"employees\",\n  // other metadata\n}\n```\n\nAdditionally, Conduit automatically adds the following metadata to each record:\n\n* `conduit.source.plugin.name`: the source plugin that created a record\n* `conduit.source.plugin.version`: version of the source plugin that created\n  this record\n\nMore information about metadata in OpenCDC records can be found [here](https://conduit.io/docs/features/opencdc-record/#opencdc).\n\n**Q: If a destination connector is able to write to multiple tables (topics,\ncollections, indexes, etc.), how should a record be routed to the correct\ndestination?**\n\nSimilarly to above, we recommend that the metadata key `\"opencdc.collection\"` is\nused.\n\nFor example, if a record has the metadata field `\"opencdc.collection\"` set\nto `employees`, then the PostgreSQL destination connector will write it to\nthe `employees` table.\n\n**Q: Is there a standard format for errors?**\n\nConduit doesn't expect any specific error format. We still encourage developers to follow the conventional [error message\nformatting](https://github.com/golang/go/wiki/CodeReviewComments#error-strings) and include enough contextual information to make\ndebugging as easy as possible (e.g. stack trace, information about the value that caused the error, internal state).\n\n**Q: Is there a standard format for logging?**\n\nDevelopers should use [`sdk.Logger`](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Logger) to retrieve a\n[`*zerolog.Logger`](https://pkg.go.dev/github.com/rs/zerolog#Logger) instance. It can be used to emit structured and leveled\nlog messages that will be included in Conduit logs.\n\nKeep in mind that logging in the hot path (e.g. reading or writing a record) can have a negative impact on performance and should\nbe avoided. If you _really_ want to add a log message in the hot path please use the \"trace\" level.\n\n**Q: How do I enable logging in my tests?**\n\nBy default, logging calls made using the `sdk.Logger` in your tests will not produce any output. To enable logging while running your\nconnector tests or debugging, you need to pass a custom context with a [zerolog](https://github.com/rs/zerolog) logger attached:\n\n```go\nfunc TestFoo(t *testing.T) {\n\tlogger := zerolog.New(zerolog.NewTestWriter(t))\n\tctx := logger.WithContext(context.Background())\n\n\t// pass ctx to connector functions ...\n}\n```\n\n**Q: Do I need to worry about ordering?**\n\nIn case of the destination connector you do not have to worry about ordering. Conduit will supply records one by one in the order\nthey were produced in the source.\n\nOn the other hand, the source connector is in charge of producing records and thus dictates the order. That said, you do not have\nto worry about concurrent reads, the SDK will call [`Source.Read`](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Source)\nrepeatedly and only in one goroutine, all you have to do is return one record at a time.\n\n## Examples\n\nFor examples of simple connectors you can look at existing connectors like\n[conduit-connector-generator](https://github.com/ConduitIO/conduit-connector-generator) or\n[conduit-connector-file](https://github.com/ConduitIO/conduit-connector-file).\n\n[^1]: Collection is a generic term used in Conduit to describe an entity in a\n3rd party system from which records are read from or to which records they are\nwritten to. Examples are: topics (in Kafka), tables (in a database), indexes (in\na search engine), and collections (in NoSQL databases).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduitio%2Fconduit-connector-sdk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconduitio%2Fconduit-connector-sdk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduitio%2Fconduit-connector-sdk/lists"}