{"id":29179694,"url":"https://github.com/trendyol/go-dcp","last_synced_at":"2026-04-20T08:08:12.360Z","repository":{"id":64297448,"uuid":"543568014","full_name":"Trendyol/go-dcp","owner":"Trendyol","description":"The Go implementation of the Couchbase DCP with various features","archived":false,"fork":false,"pushed_at":"2025-06-03T18:56:44.000Z","size":2071,"stargazers_count":95,"open_issues_count":15,"forks_count":17,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-06-27T10:17:44.366Z","etag":null,"topics":["couchbase","dcp","go","golang","kubernetes"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Trendyol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-30T11:28:09.000Z","updated_at":"2025-06-25T09:27:33.000Z","dependencies_parsed_at":"2023-07-28T13:00:02.426Z","dependency_job_id":"de7d6672-9c0a-4f2e-9616-9b1e495cf2a1","html_url":"https://github.com/Trendyol/go-dcp","commit_stats":null,"previous_names":["trendyol/go-dcp","trendyol/go-dcp-client"],"tags_count":150,"template":false,"template_full_name":null,"purl":"pkg:github/Trendyol/go-dcp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trendyol%2Fgo-dcp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trendyol%2Fgo-dcp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trendyol%2Fgo-dcp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trendyol%2Fgo-dcp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Trendyol","download_url":"https://codeload.github.com/Trendyol/go-dcp/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trendyol%2Fgo-dcp/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263021821,"owners_count":23401148,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["couchbase","dcp","go","golang","kubernetes"],"created_at":"2025-07-01T19:06:05.172Z","updated_at":"2026-04-20T08:08:12.353Z","avatar_url":"https://github.com/Trendyol.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Go Dcp [![Go Reference](https://pkg.go.dev/badge/github.com/Trendyol/go-dcp.svg)](https://pkg.go.dev/github.com/Trendyol/go-dcp) [![Go Report Card](https://goreportcard.com/badge/github.com/Trendyol/go-dcp)](https://goreportcard.com/report/github.com/Trendyol/go-dcp) [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/Trendyol/go-dcp/badge)](https://scorecard.dev/viewer/?uri=github.com/Trendyol/go-dcp)\n\nThis repository contains go implementation of a Couchbase Database Change Protocol (DCP) client. You can find more information\nby looking at our [docs](docs/index.md).\n\n### Contents\n\n* [Why?](#why)\n* [Usage](#usage)\n* [Configuration](#configuration)\n* [Examples](#examples)\n\n### Why?\n\n+ Our main goal is to build a dcp client for faster and stateful systems. We're already using this repository in below\n implementations:\n + [Elastic Connector](https://github.com/Trendyol/go-dcp-elasticsearch)\n + [Kafka Connector](https://github.com/Trendyol/go-dcp-kafka)\n + [Couchbase Connector](https://github.com/Trendyol/go-dcp-couchbase)\n + [SQL Connector](https://github.com/Trendyol/go-dcp-sql)\n + [MongoDB Connector](https://github.com/Trendyol/go-dcp-mongodb)\n + [Cassandra Connector](https://github.com/Trendyol/go-dcp-cassandra)\n + [Redis Connector](https://github.com/Trendyol/go-dcp-redis)\n\n### Example\n\n```go\npackage main\n\nimport (\n \"github.com/Trendyol/go-dcp\"\n \"github.com/Trendyol/go-dcp/logger\"\n \"github.com/Trendyol/go-dcp/models\"\n)\n\nfunc listener(ctx *models.ListenerContext) {\n switch event := ctx.Event.(type) {\n case models.DcpMutation:\n logger.Log.Info(\n \"mutated(vb=%v,eventTime=%v) | id: %v, value: %v | isCreated: %v\",\n event.VbID, event.EventTime, string(event.Key), string(event.Value), event.IsCreated(),\n )\n case models.DcpDeletion:\n logger.Log.Info(\n \"deleted(vb=%v,eventTime=%v) | id: %v\",\n event.VbID, event.EventTime, string(event.Key),\n )\n case models.DcpExpiration:\n logger.Log.Info(\n \"expired(vb=%v,eventTime=%v) | id: %v\",\n event.VbID, event.EventTime, string(event.Key),\n )\n }\n\n ctx.Ack()\n}\n\nfunc main() {\n connector, err := dcp.NewDcp(\"config.yml\", listener)\n if err != nil {\n panic(err)\n }\n\n defer connector.Close()\n\n connector.Start()\n}\n```\n\n### Usage\n\n```\n$ go get github.com/Trendyol/go-dcp\n\n```\n\n### Configuration\n\n| Variable | Type | Required | Default | Description |\n|------------------------------------------|:-----------------:|:--------:|:----------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `hosts` | []string | yes | - | Couchbase host like `localhost:8091`. |\n| `username` | string | yes | - | Couchbase username. |\n| `password` | string | yes | - | Couchbase password. |\n| `bucketName` | string | yes | - | Couchbase DCP bucket. |\n| `dcp.group.name` | string | yes | | DCP group name for vbuckets. |\n| `scopeName` | string | no | _default | Couchbase scope name. |\n| `collectionNames` | []string | no | _default | Couchbase collection names. |\n| `connectionBufferSize` | uint, string | no | 20mb | Source Bucket tcp connection buffer size (x Node Count). Check this if you get OOM Killed. |\n| `maxQueueSize` | int | no | 2048 | The maximum number of requests that can be queued waiting to be sent to a node. Check this if you get queue overflowed or queue full. |\n| `connectionTimeout` | time.Duration | no | 1m | Couchbase connection timeout. |\n| `secureConnection` | bool | no | false | Enable TLS connection of Couchbase. |\n| `rootCAPath` | string | no | *not set | if `secureConnection` set `true` this field is required. |\n| `debug` | bool | no | false | For debugging purpose. |\n| `dcp.bufferSize` | int | no | 16mb | DCP internal queue buffer size (x Node Count). Check this if you get OOM Killed. |\n| `dcp.mode` | string | no | infinite | Set DCP mode `finite` If you want to listen to DCP events until now. Set DCP mode `infinite` If you want to listen to DCP events infinitely. |\n| `dcp.connectionBufferSize` | uint, string | no | 20mb | DCP tcp connection buffer size (x Node Count). Check this if you get OOM Killed. |\n| `dcp.connectionTimeout` | time.Duration | no | 1m | DCP connection timeout. |\n| `dcp.maxQueueSize` | int | no | 2048 | The maximum number of requests that can be queued waiting to be sent to a node. Check this if you get queue overflowed or queue full. |\n| `dcp.listener.skipUntil` | time.Time | no | | Set this if you want to skip events until certain time. |\n| `dcp.group.membership.type` | string | no | | DCP membership types. `couchbase`, `kubernetesHa`, `kubernetesStatefulSet`, `static`, `dynamic` or `lilCouchbase`. Check examples for details. |\n| `dcp.group.membership.memberNumber` | int | no | 1 | Set this if membership is `static`. Other methods will ignore this field. |\n| `dcp.group.membership.totalMembers` | int | no | 1 | Set this if membership is `static` or `kubernetesStatefulSet`. Other methods will ignore this field. |\n| `dcp.group.membership.rebalanceDelay` | time.Duration | no | 30s | Works for autonomous mode. If membership is `dynamic`, it is ignored and set to `0s`. |\n| `dcp.group.membership.config` | map[string]string | no | *not set | Set key-values of config. `expirySeconds`,`heartbeatInterval`,`heartbeatToleranceDuration`,`monitorInterval`,`timeout` for `couchbase` type |\n| `dcp.config.disableChangeStreams` | bool | no | false | Set this to true if you did not want to get [older versions of changes](https://docs.couchbase.com/server/current/learn/data/change-history.html) for Couchbase Server 7.2.0+ using Magma storage buckets |\n| `dcp.config.priority` | string | no | low | DCP stream priority requested from the Couchbase cluster. Allowed values: `low`, `medium`, `high`. Maps to gocbcore's `DcpAgentPriority` (`set_priority` DCP control). |\n| `leaderElection.enabled` | bool | no | false | Set this true for memberships `kubernetesHa`. |\n| `leaderElection.type` | string | no | kubernetes | Leader Election types. `kubernetes` |\n| `leaderElection.config` | map[string]string | no | *not set | Set key-values of config. `leaseLockName`,`leaseLockNamespace`, `leaseDuration`, `renewDeadline`, `retryPeriod` for `kubernetes` type. |\n| `leaderElection.rpc.port` | int | no | 8081 | This field is usable for `kubernetesStatefulSet` membership. |\n| `checkpoint.type` | string | no | auto | Set checkpoint type `auto` or `manual`. |\n| `checkpoint.autoReset` | string | no | earliest | Set checkpoint start point to `earliest` or `latest`. |\n| `checkpoint.interval` | time.Duration | no | 1m | Checkpoint checking interval. |\n| `checkpoint.timeout` | time.Duration | no | 1m | Checkpoint checking timeout. |\n| `healthCheck.disabled` | bool | no | false | Disable Couchbase connection health check. |\n| `healthCheck.interval` | time.Duration | no | 1m | Couchbase connection health checking interval duration. |\n| `healthCheck.timeout` | time.Duration | no | 1m | Couchbase connection health checking timeout duration. |\n| `rollbackMitigation.disabled` | bool | no | false | Disable reprocessing for roll-backed Vbucket offsets. |\n| `rollbackMitigation.interval` | time.Duration | no | 1s | Persisted sequence numbers polling interval. |\n| `rollbackMitigation.configWatchInterval` | time.Duration | no | 10s | Cluster config changes listener interval. |\n| `metadata.type` | string | no | couchbase | Metadata storing types. `file`, `couchbase`, `noop`. |\n| `metadata.readOnly` | bool | no | false | Set this for debugging state purposes. |\n| `metadata.config` | map[string]string | no | *not set | Set key-values of config. `hosts`, `username`, `password`, `bucket`,`scope`,`collection`,`maxQueueSize`,`connectionBufferSize` 5mb is default (x Node Count),`connectionTimeout`, `secureConnection`, `rootCAPath` for `couchbase` type |\n| `api.disabled` | bool | no | false | Disable metric endpoints |\n| `api.port` | int | no | 8080 | Set API port |\n| `metric.path` | string | no | /metrics | Set metric endpoint path. |\n| `logging.level` | string | no | info | Set logging level. |\n\n### Environment Variables\n\nThese environment variables will **overwrite** the corresponding configs.\n\n| Variable | Type | Corresponding Config | Description |\n|---------------------------------------------|:----:|:---------------------------------:|:------------------------------------------------------------:|\n| `GO_DCP__DCP_GROUP_MEMBERSHIP_MEMBERNUMBER` | int | dcp.group.membership.memberNumber | To be able to prevent making deployment to scale up or down. |\n| `GO_DCP__DCP_GROUP_MEMBERSHIP_TOTALMEMBERS` | int | dcp.group.membership.totalMembers | To be able to prevent making deployment to scale up or down. |\n\n### Monitoring\n\nThe client offers an API that handles different endpoints and expose several metrics.\n\n### API\n\n| Endpoint | Description | Debug Mode | Body |\n|-------------------------|------------------------------------------------------------------------------------------|------------|-------------------------------------------------|\n| `GET /status` | Returns a 200 OK status if the client is able to ping the couchbase server successfully. | | |\n| `GET /rebalance` | Triggers a rebalance operation for the vBuckets. | | |\n| `GET /states/offset` | Returns the current offsets for each vBucket. | x | |\n| `GET /states/followers` | Returns the list of follower clients if service discovery enabled | x | |\n| `GET /debug/pprof/*` | [Fiber Pprof](https://docs.gofiber.io/api/middleware/pprof/) | x | |\n| `PUT /membership/info` | Updates membership info and applies rebalance. | | ```{\"memberNumber\": 1,\"totalMembers\": 3 }``` | \n\nThe Client collects relevant metrics and makes them available at /metrics endpoint.\nIn case you haven't configured a metric.path, the metrics will be exposed at the /metrics.\n\n### Exposed metrics\n\n| Metric Name | Description | Labels | Value Type |\n|--------------------------------------|---------------------------------------------------------|------------------------------------------|------------|\n| cbgo_mutation_total | The total number of mutations on a specific vBucket | vbId: ID of the vBucket | Counter |\n| cbgo_deletion_total | The total number of deletions on a specific vBucket | vbId: ID of the vBucket | Counter |\n| cbgo_expiration_total | The total number of expirations on a specific vBucket | vbId: ID of the vBucket | Counter |\n| cbgo_agent_queue_current | The current number of agent queue | address: Couchbase, is dcp: Is Dcp Agent | Gauge |\n| cbgo_agent_queue_max | The max number of agent queue | address: Couchbase, is dcp: Is Dcp Agent | Gauge |\n| cbgo_seq_no_current | The current sequence number on a specific vBucket | vbId: ID of the vBucket | Gauge |\n| cbgo_start_seq_no_current | The starting sequence number on a specific vBucket | vbId: ID of the vBucket | Gauge |\n| cbgo_end_seq_no_current | The ending sequence number on a specific vBucket | vbId: ID of the vBucket | Gauge |\n| cbgo_persist_seq_no_current | The persist sequence number on a specific vBucket | vbId: ID of the vBucket | Gauge |\n| cbgo_lag_current | The current lag on a specific vBucket | vbId: ID of the vBucket | Gauge |\n| cbgo_total_lag_current | The current total lag | N/A | Gauge |\n| cbgo_process_latency_ms_current | The latest process latency in milliseconds | N/A | Gauge |\n| cbgo_dcp_latency_ms_current | The latest consumed dcp message latency in milliseconds | N/A | Gauge |\n| cbgo_rebalance_current | The number of total rebalance | N/A | Counter |\n| cbgo_active_stream_current | The number of total active stream | N/A | Gauge |\n| cbgo_total_members_current | The total number of members in the cluster | N/A | Gauge |\n| cbgo_member_number_current | The number of the current member | N/A | Gauge |\n| cbgo_membership_type_current | The type of membership of the current member | Membership type | Gauge |\n| cbgo_offset_write_current | The latest number of the offset write | N/A | Gauge |\n| cbgo_offset_write_latency_ms_current | The latest offset write latency in milliseconds | N/A | Gauge |\n\n### Compatibility\n\n| Go DCP Version | Minimum Couchbase Server Version |\n|----------------|----------------------------------|\n| x\u003c1.1.16 | 6.5.x |\n| 1.1.16\u003e=x | 5.x.x |\n\n## Breaking Changes\n\n| Date taking effect | Version | Change | How to check |\n|--------------------|---------|----------------------------------------------------------------------------------------|---------------------| \n| December 14, 2023 | v1.1.19 | dcp.config.[DisableExpiryOpcode,DisableStreamEndByClient, EnableChangeStreams] removed | Review your configs |\n\n### Examples\n\n- [example with couchbase membership](example/main.go)\n- [couchbase membership config](example/config.yml) - thanks to [@onursak](https://github.com/onursak)\n- [kubernetesStatefulSet membership config](example/config_k8s_stateful_set.yml)\n- [kubernetesHa membership config](example/config_k8s_leader_election.yml)\n- [static membership config](example/config_static.yml)\n- [dynamic membership config](example/config_dynamic.yml)\n- [lilCouchbase membership config](example/config_lil_couchbase.yml)\n\n## Grafana Metric Dashboard\n\n[Grafana \u0026 Prometheus Example](example/grafana)\n\n## Distributed Tracing\n\n[otel-go-dcp](https://github.com/Trendyol/otel-go-dcp)\n\n## Before Contribution\n\n```\n# Make sure to use .githooks as hooks path of your git\ngit config core.hooksPath .githooks\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrendyol%2Fgo-dcp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrendyol%2Fgo-dcp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrendyol%2Fgo-dcp/lists"}