{"id":20415876,"url":"https://github.com/obsidiandynamics/goharvest","last_synced_at":"2025-08-21T01:32:25.292Z","repository":{"id":38132958,"uuid":"258715729","full_name":"obsidiandynamics/goharvest","owner":"obsidiandynamics","description":"Transactional outbox harvester for Postgres → Kafka, written in Go","archived":false,"fork":false,"pushed_at":"2022-11-10T14:08:23.000Z","size":116,"stargazers_count":193,"open_issues_count":3,"forks_count":18,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-08T01:42:37.065Z","etag":null,"topics":["distributed-systems","distributed-transactions","golang","kafka","microservices","outbox","postgres"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/obsidiandynamics.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-25T07:44:40.000Z","updated_at":"2025-03-28T06:52:19.000Z","dependencies_parsed_at":"2022-07-09T13:16:03.889Z","dependency_job_id":null,"html_url":"https://github.com/obsidiandynamics/goharvest","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/obsidiandynamics/goharvest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obsidiandynamics%2Fgoharvest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obsidiandynamics%2Fgoharvest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obsidiandynamics%2Fgoharvest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obsidiandynamics%2Fgoharvest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/obsidiandynamics","download_url":"https://codeload.github.com/obsidiandynamics/goharvest/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obsidiandynamics%2Fgoharvest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271415043,"owners_count":24755629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","distributed-transactions","golang","kafka","microservices","outbox","postgres"],"created_at":"2024-11-15T06:17:53.357Z","updated_at":"2025-08-21T01:32:25.001Z","avatar_url":"https://github.com/obsidiandynamics.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\u003cimg src=\"https://raw.githubusercontent.com/wiki/obsidiandynamics/goharvest/images/goharvest-logo-wide.png\" width=\"400px\" alt=\"logo\"/\u003e\u0026nbsp;\n===\n![Go version](https://img.shields.io/github/go-mod/go-version/obsidiandynamics/goharvest)\n[![Build](https://travis-ci.org/obsidiandynamics/goharvest.svg?branch=master) ](https://travis-ci.org/obsidiandynamics/goharvest#)\n![Release](https://img.shields.io/github/v/release/obsidiandynamics/goharvest?color=ff69b4)\n[![Codecov](https://codecov.io/gh/obsidiandynamics/goharvest/branch/master/graph/badge.svg)](https://codecov.io/gh/obsidiandynamics/goharvest)\n[![Go Report Card](https://goreportcard.com/badge/github.com/obsidiandynamics/goharvest)](https://goreportcard.com/report/github.com/obsidiandynamics/goharvest)\n[![Total alerts](https://img.shields.io/lgtm/alerts/g/obsidiandynamics/goharvest.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/obsidiandynamics/goharvest/alerts/)\n[![GoDoc Reference](https://img.shields.io/badge/docs-GoDoc-blue.svg)](https://pkg.go.dev/github.com/obsidiandynamics/goharvest?tab=doc)\n\n`goharvest` is a Go implementation of the [Transactional Outbox](https://microservices.io/patterns/data/transactional-outbox.html) pattern for Postgres and Kafka.\n\n\u003cimg src=\"https://raw.githubusercontent.com/wiki/obsidiandynamics/goharvest/images/figure-outbox.png\" width=\"100%\" alt=\"Transactional Outbox\"/\u003e\n\nWhile `goharvest` is a complex beast, the end result is dead simple: to publish Kafka messages reliably and atomically, simply write a record to a dedicated **outbox table** in a transaction, alongside any other database changes. (Outbox schema provided below.) `goharvest` scrapes the outbox table in the background and publishes records to a Kafka topic of the application's choosing, using the key, value and headers specified in the outbox record. `goharvest` currently works with Postgres. It maintains causal order of messages and does not require CDC to be enabled on the database, making for a zero-hassle setup. It handles thousands of records/second on commodity hardware.\n\n# Getting started\n## 1. Create an outbox table for your application\n```sql\nCREATE TABLE IF NOT EXISTS outbox (\n  id                  BIGSERIAL PRIMARY KEY,\n  create_time         TIMESTAMP WITH TIME ZONE NOT NULL,\n  kafka_topic         VARCHAR(249) NOT NULL,\n  kafka_key           VARCHAR(100) NOT NULL,  -- pick your own maximum key size\n  kafka_value         VARCHAR(10000),         -- pick your own maximum value size\n  kafka_header_keys   TEXT[] NOT NULL,\n  kafka_header_values TEXT[] NOT NULL,\n  leader_id           UUID\n)\n```\n\n## 2. Run `goharvest`\n### Standalone mode\nThis runs `goharvest` within a separate process called `reaper`, which will work alongside **any** application that writes to a standard outbox. (Not just applications written in Go.)\n\n#### Install `reaper`\n```sh\ngo get -u github.com/obsidiandynamics/goharvest/cmd/reaper\n```\n\n#### Create `reaper.yaml` configuration\n```yaml\nharvest:\n  baseKafkaConfig: \n    bootstrap.servers: localhost:9092\n  producerKafkaConfig:\n    compression.type: lz4\n    delivery.timeout.ms: 10000\n  leaderTopic: my-app-name\n  leaderGroupID: my-app-name\n  dataSource: host=localhost port=5432 user=postgres password= dbname=postgres sslmode=disable\n  outboxTable: outbox\n  limits:\n    minPollInterval: 1s\n    heartbeatTimeout: 5s\n    maxInFlightRecords: 1000\n    minMetricsInterval: 5s\n    sendConcurrency: 4\n    sendBuffer: 10\nlogging:\n  level: Debug\n```\n\n#### Start `reaper`\n```sh\nreaper -f reaper.yaml\n```\n\n### Embedded mode\n`goharvest` can be run in the same process as your application.\n\n#### Add the dependency\n```sh\ngo get -u github.com/obsidiandynamics/goharvest\n```\n\n#### Create and start a `Harvest` instance\n```go\nimport \"github.com/obsidiandynamics/goharvest\"\n```\n\n```go\n// Configure the harvester. It will use its own database and Kafka connections under the hood.\nconfig := Config{\n  BaseKafkaConfig: KafkaConfigMap{\n    \"bootstrap.servers\": \"localhost:9092\",\n  },\n  DataSource: \"host=localhost port=5432 user=postgres password= dbname=postgres sslmode=disable\",\n}\n\n// Create a new harvester.\nharvest, err := New(config)\nif err != nil {\n  panic(err)\n}\n\n// Start harvesting in the background.\nerr = harvest.Start()\nif err != nil {\n  panic(err)\n}\n\n// Wait indefinitely for the harvester to end.\nlog.Fatal(harvest.Await())\n```\n\n### Using a custom logger\n`goharvest` uses `log.Printf` for output by default. Logger configuration is courtesy of the Scribe façade, from [\u003ccode\u003elibstdgo\u003c/code\u003e](https://github.com/obsidiandynamics/libstdgo). The example below uses a Logrus binding for Scribe.\n\n```go\nimport (\n  \"github.com/obsidiandynamics/goharvest\"\n  scribelogrus \"github.com/obsidiandynamics/libstdgo/scribe/logrus\"\n  \"github.com/sirupsen/logrus\"\n)\n```\n\n```sh\nlog := logrus.StandardLogger()\nlog.SetLevel(logrus.DebugLevel)\n\n// Configure the custom logger using a binding.\nconfig := Config{\n  BaseKafkaConfig: KafkaConfigMap{\n    \"bootstrap.servers\": \"localhost:9092\",\n  },\n  Scribe:     scribe.New(scribelogrus.Bind()),\n  DataSource: \"host=localhost port=5432 user=postgres password= dbname=postgres sslmode=disable\",\n}\n```\n\n### Listening for leader status updates\nJust like `goharvest` uses [NELI](https://github.com/obsidiandynamics/goneli) to piggy-back on Kafka's leader election, you can piggy-back on `goharvest` to get leader status updates:\n\n```go\nlog := logrus.StandardLogger()\nlog.SetLevel(logrus.TraceLevel)\nconfig := Config{\n  BaseKafkaConfig: KafkaConfigMap{\n    \"bootstrap.servers\": \"localhost:9092\",\n  },\n  DataSource: \"host=localhost port=5432 user=postgres password= dbname=postgres sslmode=disable\",\n  Scribe:     scribe.New(scribelogrus.Bind()),\n}\n\n// Create a new harvester and register an event hander.\nharvest, err := New(config)\n\n// Register a handler callback, invoked when an event occurs within goharvest.\n// The callback is completely optional; it lets the application piggy-back on leader\n// status updates, in case it needs to schedule some additional work (other than\n// harvesting outbox records) that should only be run on one process at any given time.\nharvest.SetEventHandler(func(e Event) {\n  switch event := e.(type) {\n  case LeaderAcquired:\n    // The application may initialise any state necessary to perform work as a leader.\n    log.Infof(\"Got event: leader acquired: %v\", event.LeaderID())\n  case LeaderRefreshed:\n    // Indicates that a new leader ID was generated, as a result of having to remark\n    // a record (typically as due to an earlier delivery error). This is purely\n    // informational; there is nothing an application should do about this, other\n    // than taking note of the new leader ID if it has come to rely on it.\n    log.Infof(\"Got event: leader refreshed: %v\", event.LeaderID())\n  case LeaderRevoked:\n    // The application may block the callback until it wraps up any in-flight\n    // activity. Only upon returning from the callback, will a new leader be elected.\n    log.Infof(\"Got event: leader revoked\")\n  case LeaderFenced:\n    // The application must immediately terminate any ongoing activity, on the assumption\n    // that another leader may be imminently elected. Unlike the handling of LeaderRevoked,\n    // blocking in the callback will not prevent a new leader from being elected.\n    log.Infof(\"Got event: leader fenced\")\n  case MeterRead:\n    // Periodic statistics regarding the harvester's throughput.\n    log.Infof(\"Got event: meter read: %v\", event.Stats())\n  }\n})\n\n// Start harvesting in the background.\nerr = harvest.Start()\n```\n\n### Which mode should I use\nRunning `goharvest` in standalone mode using `reaper` is the recommended approach for most use cases, as it fully insulates the harvester from the rest of the application. Ideally, you should deploy `reaper` as a sidecar daemon, to run alongside your application. All the reaper needs is access to the outbox table and the Kafka cluster.\n\nEmbedded `goharvest` is useful if you require additional insights into its operation, which is accomplished by registering an `EventHandler` callback, as shown in the example above. This callback is invoked whenever the underlying leader status changes, which may be useful if you need to schedule additional workloads that should only be run on one process at any given time.\n\n## 3. Write outbox records\n### Directly, using SQL\nYou can write database records from any app, by simply issuing the following `INSERT` statement:\n\n```sql\nINSERT INTO ${outbox_table} (\n  create_time, \n  kafka_topic, \n  kafka_key, \n  kafka_value, \n  kafka_header_keys, \n  kafka_header_values\n)\nVALUES (NOW(), $1, $2, $3, $4, $5)\n```\n\nReplace `${outbox_table}` and bind the query variables as appropriate:\n\n* `kafka_topic` column specifies an arbitrary topic name, which may differ among records.\n* `kafka_key` is a mandatory `string` key. Each record must be published with a specified key, which will affect its placement among the topic's partitions.\n* `kafka_value` is an optional `string` value. If unspecified, the record will be published with a `nil` value, allowing it to be used as a compaction tombstone.\n* `kafka_header_keys` and `kafka_header_values` are arrays that specify the keys and values of record headers. When used each element in `kafka_header_keys` corresponds to an element in `kafka_header_values` at the same index. If not using headers, set both arrays to empty.\n\n\u003e **Note**: **Writing outbox records should be performed in the same transaction as other related database updates.** Otherwise, messaging will not be atomic — the updates may be stably persisted while the message might be lost, and *vice versa*.\n\n### Using `stasher`\nThe `goharvest` library comes with a `stasher` helper package for writing records to an outbox.\n\n#### One-off messages\nWhen one database update corresponds to one message, the easiest approach is to call `Stasher.Stash()`:\n\n```go\nimport \"github.com/obsidiandynamics/goharvest\"\n```\n\n```go\ndb, err := sql.Open(\"postgres\", \"host=localhost port=5432 user=postgres password= dbname=postgres sslmode=disable\")\nif err != nil {\n  panic(err)\n}\ndefer db.Close()\n\nst := New(\"outbox\")\n\n// Begin a transaction.\ntx, _ := db.Begin()\ndefer tx.Rollback()\n\n// Update other database entities in transaction scope.\n\n// Stash an outbox record for subsequent harvesting.\nerr = st.Stash(tx, goharvest.OutboxRecord{\n  KafkaTopic: \"my-app.topic\",\n  KafkaKey:   \"hello\",\n  KafkaValue: goharvest.String(\"world\"),\n  KafkaHeaders: goharvest.KafkaHeaders{\n    {Key: \"applicationId\", Value: \"my-app\"},\n  },\n})\nif err != nil {\n  panic(err)\n}\n\n// Commit the transaction.\ntx.Commit()\n```\n\n#### Multiple messages\nSending multiple messages within a single transaction may be done more efficiently using prepared statements:\n\n```go\n// Begin a transaction.\ntx, _ := db.Begin()\ndefer tx.Rollback()\n\n// Update other database entities in transaction scope.\n// ...\n\n// Formulates a prepared statement that may be reused within the scope of the transaction.\nprestash, _ := st.Prepare(tx)\n\n// Publish a bunch of messages using the same prepared statement.\nfor i := 0; i \u003c 10; i++ {\n  // Stash an outbox record for subsequent harvesting.\n  err = prestash.Stash(goharvest.OutboxRecord{\n    KafkaTopic: \"my-app.topic\",\n    KafkaKey:   \"hello\",\n    KafkaValue: goharvest.String(\"world\"),\n    KafkaHeaders: goharvest.KafkaHeaders{\n      {Key: \"applicationId\", Value: \"my-app\"},\n    },\n  })\n  if err != nil {\n    panic(err)\n  }\n}\n\n// Commit the transaction.\ntx.Commit()\n```\n\n# Configuration\nThere are handful of parameters that for configuring `goharvest`, assigned via the `Config` struct:\n\n\u003ctable\u003e\n  \u003cthead\u003e\n    \u003ctr\u003e\n      \u003cth\u003eParameter\u003c/th\u003e\n      \u003cth\u003eDefault value\u003c/th\u003e\n      \u003cth\u003eDescription\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eBaseKafkaConfig\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003eMap containing \u003ccode\u003ebootstrap.servers=localhost:9092\u003c/code\u003e.\u003c/td\u003e\n      \u003ctd\u003eConfiguration shared by the underlying Kafka producer and consumer clients, including those used for leader election.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eProducerKafkaConfig\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003eEmpty map.\u003c/td\u003e\n      \u003ctd\u003eAdditional configuration on top of \u003ccode\u003eBaseKafkaConfig\u003c/code\u003e that is specific to the producer clients created by \u003ccode\u003egoharvest\u003c/code\u003e for publishing harvested messages. This configuration does not apply to the underlying NELI leader election protocol.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLeaderGroupID\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003eAssumes the filename of the application binary.\u003c/td\u003e\n      \u003ctd\u003eUsed by the underlying leader election protocol as a unique identifier shared by all instances in a group of competing processes. The \u003ccode\u003eLeaderGroupID\u003c/code\u003e is used as Kafka \u003ccode\u003egroup.id\u003c/code\u003e property under the hood, when subscribing to the leader election topic.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLeaderTopic\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003eAssumes the value of \u003ccode\u003eLeaderGroupID\u003c/code\u003e, suffixed with the string \u003ccode\u003e.neli\u003c/code\u003e.\u003c/td\u003e\n      \u003ctd\u003eUsed by NELI as the name of the Kafka topic for orchestrating leader election. Competing processes subscribe to the same topic under an identical consumer group ID, using Kafka's exclusive partition assignment as a mechanism for arbitrating leader status.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eDataSource\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003eLocal Postgres data source \u003ccode\u003ehost=localhost port=5432 user=postgres password= dbname=postgres sslmode=disable\u003c/code\u003e.\u003c/td\u003e\n      \u003ctd\u003eThe database driver-specific data source string.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eOutboxTable\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003e\u003ccode\u003eoutbox\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003eThe name of the outbox table, optionally including the schema name.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eScribe\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003eScribe configured with bindings for \u003ccode\u003elog.Printf()\u003c/code\u003e; effectively the result of running \u003ccode\u003escribe.New(scribe.StandardBinding())\u003c/code\u003e.\u003c/td\u003e\n      \u003ctd\u003eThe logging façade used by the library, preconfigured with your logger of choice. See \u003ca href=\"https://pkg.go.dev/github.com/obsidiandynamics/libstdgo/scribe?tab=doc\"\u003eScribe GoDocs\u003c/a\u003e.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eName\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003eA string in the form \u003ccode\u003e{hostname}_{pid}_{time}\u003c/code\u003e, where \u003ccode\u003e{hostname}\u003c/code\u003e is the result of invoking \u003ccode\u003eos.Hostname()\u003c/code\u003e, \u003ccode\u003e{pid}\u003c/code\u003e is the process ID, and \u003ccode\u003e{time}\u003c/code\u003e is the UNIX epoch time, in seconds.\u003c/td\u003e\n      \u003ctd\u003eThe symbolic name of this instance. This field is informational only, accompanying all log messages.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLimits.MinPollInterval\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003e100 ms\u003c/td\u003e\n      \u003ctd\u003eThe lower bound on the poll interval, preventing the over-polling of Kafka on successive \u003ccode\u003ePulse()\u003c/code\u003e invocations. Assuming \u003ccode\u003ePulse()\u003c/code\u003e is called repeatedly by the application, NELI may poll Kafka at a longer interval than \u003ccode\u003eMinPollInterval\u003c/code\u003e. (Regular polling is necessary to prove client's liveness and maintain internal partition assignment, but polling excessively is counterproductive.)\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLimits.HeartbeatTimeout\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003e5 s\u003c/td\u003e\n      \u003ctd\u003eThe period that a leader will maintain its status, not having received a heartbeat message on the leader topic. After the timeout elapses, the leader will assume a network partition and will voluntarily yield its status, signalling a \u003ccode\u003eLeaderFenced\u003c/code\u003e event to the application.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLimits.QueueTimeout\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003e30 s\u003c/td\u003e\n      \u003ctd\u003eThe maximum period of time a record may be queued after having been marked, before timing out and triggering a remark.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLimits.MarkBackoff\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003e10 ms\u003c/td\u003e\n      \u003ctd\u003eThe backoff delay introduced by the mark thread when a query returns no results, indicating the absence of backlogged records. A mark backoff prevents aggressive querying of the database in the absence of a steady flow of outbox records.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLimits.IOErrorBackoff\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003e500 ms\u003c/td\u003e\n      \u003ctd\u003eThe backoff delay introduced when any of the mark, purge or reset queries encounter a database error.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLimits.MaxInFlightRecords\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003e1000\u003c/td\u003e\n      \u003ctd\u003eAn upper bound on the number of marked records that may be in flight at any given time. I.e. the number of records that have been enqueued with a producer client, for which acknowledgements have yet to be received.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLimits.SendConcurrency\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003e8\u003c/td\u003e\n      \u003ctd\u003eThe number of concurrent shards used for queuing causally unrelated records. Each shard is equipped with a dedicated producer client, allowing for its records to be sent independently of other shards.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLimits.SendBuffer\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003e10\u003c/td\u003e\n      \u003ctd\u003eThe maximum number of marked records that may be buffered for subsequent sending, for any given shard. When the buffer is full, the marker will halt — waiting for records to be sent and for their acknowledgements to flow through.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLimits.MarkQueryRecords\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003e100\u003c/td\u003e\n      \u003ctd\u003eAn upper bound on the number of records that may be marked in any given query. Limiting this number avoids long-running database queries.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr valign=\"top\"\u003e\n      \u003ctd\u003e\u003ccode\u003eLimits.MinMetricsInterval\u003c/code\u003e\u003c/td\u003e\n      \u003ctd\u003e5 s\u003c/td\u003e\n      \u003ctd\u003eThe minimum interval at which throughput metrics are emitted. Metrics are emitted conservatively and may be observed less frequently; in fact, throughput metrics are only emitted upon a successful message acknowledgement, which will not occur during periods of inactivity.\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n# Docs\n[Design](https://github.com/obsidiandynamics/goharvest/wiki/Design)\n\n[Comparison of messaging patterns](https://github.com/obsidiandynamics/goharvest/wiki/Comparison-of-messaging-patterns)\n\n[Comparison of harvesting methods](https://github.com/obsidiandynamics/goharvest/wiki/Comparison-of-harvesting-methods)\n\n[FAQ](https://github.com/obsidiandynamics/goharvest/wiki/FAQ)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobsidiandynamics%2Fgoharvest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fobsidiandynamics%2Fgoharvest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobsidiandynamics%2Fgoharvest/lists"}