{"id":50803366,"url":"https://github.com/uded/koanf-etcd","last_synced_at":"2026-06-12T22:33:30.571Z","repository":{"id":362579118,"uuid":"1259810628","full_name":"uded/koanf-etcd","owner":"uded","description":"A production-grade koanf v2 Provider for etcd v3 — auth/TLS, nested output, watch with reconnect+resume+resync, debounce, BYO clientv3.Client","archived":false,"fork":false,"pushed_at":"2026-06-04T22:54:54.000Z","size":141,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-05T00:08:32.519Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uded.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-04T22:08:42.000Z","updated_at":"2026-06-04T22:54:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/uded/koanf-etcd","commit_stats":null,"previous_names":["uded/koanf-etcd"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/uded/koanf-etcd","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uded%2Fkoanf-etcd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uded%2Fkoanf-etcd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uded%2Fkoanf-etcd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uded%2Fkoanf-etcd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uded","download_url":"https://codeload.github.com/uded/koanf-etcd/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uded%2Fkoanf-etcd/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34265491,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-12T22:33:29.472Z","updated_at":"2026-06-12T22:33:30.552Z","avatar_url":"https://github.com/uded.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# koanf-etcd\n\nA production-grade [koanf](https://github.com/knadh/koanf) v2 Provider for [etcd](https://etcd.io) v3.\n\n[![CI](https://github.com/uded/koanf-etcd/actions/workflows/ci.yml/badge.svg)](https://github.com/uded/koanf-etcd/actions/workflows/ci.yml)\n[![Go Reference](https://pkg.go.dev/badge/github.com/uded/koanf-etcd.svg)](https://pkg.go.dev/github.com/uded/koanf-etcd)\n[![Go Report Card](https://goreportcard.com/badge/github.com/uded/koanf-etcd)](https://goreportcard.com/report/github.com/uded/koanf-etcd)\n[![Release](https://img.shields.io/github/v/release/uded/koanf-etcd?sort=semver)](https://github.com/uded/koanf-etcd/releases)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n\n## Related projects\n\nThis provider composes with two sibling packages in the same family:\n\n- **[koanf-structdefaults](https://github.com/uded/koanf-structdefaults)** — populate a koanf instance from struct-tag defaults. The natural *floor* layer below this provider in the load order.\n- **[koanf-validate](https://github.com/uded/koanf-validate)** — validate the assembled koanf config against struct-tag rules. Pair with this provider's watch loop to gate bad etcd writes (see [Watch + reload recipe](#watch--reload-recipe) below).\n\nRecommended load order: `structdefaults` → file → `koanf-etcd` → env, with `koanf-validate` as the post-load gate.\n\n## Why this exists\n\nThe bundled `github.com/knadh/koanf/providers/etcd` has caused real production incidents because:\n\n1. **No auth, no TLS.** It cannot connect to authenticated or mTLS clusters.\n2. **Returns flat keys including the prefix**, despite a doc comment claiming \"nested.\" This silently breaks merges with nested layers (defaults, files) — overrides drop nondeterministically.\n3. **No value normalization.** A trailing newline from `etcdctl` makes `http://x` and `http://x\\n` distinct.\n4. **No empty-result diagnostics.** A wrong prefix looks like a healthy boot on stale defaults.\n5. **Weak Watch.** Uses `context.Background()`, never reconnects, no compaction recovery, no debounce.\n\nThis package fixes every one of those, by default.\n\n### Comparison\n\n| | Bundled `providers/etcd` | `koanf-etcd` |\n| --- | --- | --- |\n| TLS / auth | ❌ | ✅ |\n| Nested output | ❌ (flat with prefix) | ✅ (unflattened, prefix-trimmed) |\n| Value TrimSpace | ❌ | ✅ (replaceable) |\n| Empty-prefix diagnostics | ❌ | ✅ (strict or callback) |\n| Watch reconnect | ❌ | ✅ (bounded exponential backoff) |\n| Compaction recovery | ❌ | ✅ (resync event) |\n| Resume-from-revision | ❌ | ✅ (no read/watch gap) |\n| Debounce | ❌ | ✅ (configurable window) |\n| BYO `*clientv3.Client` | ❌ | ✅ (headline feature) |\n| Blob mode (one-key documents) | ❌ | ✅ |\n| Pagination | ❌ | ✅ |\n| Atomic multi-key writes | ❌ | ✅ (separate `write` subpackage) |\n\n## Quickstart — tree mode\n\n```go\nimport (\n    \"github.com/knadh/koanf/v2\"\n    ketcd \"github.com/uded/koanf-etcd\"\n)\n\np, err := ketcd.New(\n    ketcd.WithEndpoints(\"localhost:2379\"),\n    ketcd.WithPrefix(\"/svc/\"),\n)\nif err != nil { panic(err) }\ndefer p.Close()\n\nk := koanf.New(\".\")\nif err := k.Load(p, nil); err != nil { panic(err) }\n\nfmt.Println(k.String(\"db.host\"))\n```\n\n## Quickstart — blob mode\n\n```go\nimport (\n    \"github.com/knadh/koanf/parsers/yaml\"\n    \"github.com/knadh/koanf/v2\"\n    ketcd \"github.com/uded/koanf-etcd\"\n)\n\np, _ := ketcd.New(\n    ketcd.WithEndpoints(\"localhost:2379\"),\n    ketcd.WithKey(\"/cfg.yaml\"),\n    ketcd.WithBlob(),\n    ketcd.WithUnflatten(false),\n)\ndefer p.Close()\n\nk := koanf.New(\".\")\nk.Load(p, yaml.Parser())\n```\n\n## BYO `*clientv3.Client`\n\nThe headline feature. Share one client across this provider and your own watchers; control its lifecycle yourself.\n\n```go\ncli, _ := clientv3.New(clientv3.Config{\n    Endpoints: []string{\"etcd:2379\"},\n    TLS:       myTLS,\n    Username:  \"alice\",\n    Password:  os.Getenv(\"ETCD_PASS\"),\n})\ndefer cli.Close()  // koanf-etcd will NOT close this\n\np, _ := ketcd.New(\n    ketcd.WithClient(cli),\n    ketcd.WithPrefix(\"/svc/\"),\n)\n```\n\n`Close()` on the Provider closes only what the Provider built. A BYO client stays alive.\n\n## Watch + reload recipe\n\nThe Provider tells you something changed. You decide how to reload. Below is the production-grade pattern — runs a validator gate and atomically swaps an immutable `*Config`.\n\n```go\ntype Config struct { /* ... */ }\n\nvar current atomic.Pointer[Config]\n\np, _ := ketcd.New(\n    ketcd.WithClient(cli),\n    ketcd.WithPrefix(\"/svc/\"),\n    ketcd.WithDebounce(500 * time.Millisecond),\n    ketcd.WithOnResync(func(reason string, rev int64) {\n        slog.Info(\"etcd resynced\", \"reason\", reason, \"rev\", rev)\n    }),\n)\ndefer p.Close()\n\nreload := func() error {\n    k := koanf.New(\".\")\n    if err := k.Load(p, nil); err != nil { return err }\n    var c Config\n    if err := k.Unmarshal(\"\", \u0026c); err != nil { return err }\n    if err := validate(\u0026c); err != nil { return err }  // koanf-validate\n    current.Store(\u0026c)\n    return nil\n}\nif err := reload(); err != nil { /* fail boot */ }\n\n_ = p.Watch(func(_ any, err error) {\n    if err != nil {\n        slog.Error(\"watch\", \"err\", err)\n        return\n    }\n    if err := reload(); err != nil {\n        slog.Warn(\"reload rejected; keeping last-good\", \"err\", err)\n    }\n})\n```\n\nA bad write to etcd is rejected by `validate`; last-good keeps serving.\n\nFor per-key detail (e.g. to log which keys changed), use `WatchTyped`:\n\n```go\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\n\n_ = p.WatchTyped(ctx, func(evs []ketcd.Event, err error) {\n    for _, e := range evs {\n        slog.Info(\"change\", \"type\", e.Type, \"key\", e.Key, \"rev\", e.Revision)\n    }\n})\n```\n\n## Interop with `koanf-structdefaults` and friends\n\nLoad order: struct-defaults (floor) → file → etcd (live) → env (pins/secrets).\n\n```go\nk := koanf.New(\".\")\nk.Load(structdefaults.Provider(MyConfig{}, \".\"), nil)        // floor\nk.Load(file.Provider(\"config.yaml\"), yaml.Parser())          // file\nk.Load(etcdProvider, nil)                                    // live\nk.Load(env.Provider(\"MYAPP_\", \".\", nil), nil)                // pins\n```\n\nEach later layer overrides earlier ones for keys it provides.\n\n## Security notes\n\n- **Threat model**: see [SECURITY.md](SECURITY.md) for what the library does and does not defend against, plus how to report vulnerabilities.\n- **TLS**: pass `*tls.Config` via `WithTLS` or load cert/key/CA files via `WithTLSFiles`. The two are mutually exclusive.\n- **Auth**: `WithAuth(\"user\", \"pass\")`. Never hardcode — pull from env or a secret manager. For STS-style ephemeral credentials or secret-manager rotation, prefer `WithAuthProvider(fn)`; the function runs once just before client construction, so caller-owned credential buffers can be zeroed out immediately afterwards.\n- **Watch event values**: `Event.Value` from `WatchTyped` is raw, unvalidated bytes from etcd — any process with write access can put any bytes there. Validate before passing to shells, SQL, HTML-rendered logs, or file paths. `Read()` applies the value transform (TrimSpace by default); `WatchTyped` deliberately does not, so consumers parsing structured payloads (JSON/proto) don't pay the round-trip cost.\n- **SRV discovery**: `WithEndpointsFromSRV` trusts DNS at face value. Pair with TLS + `WithTLSServerName` if your DNS path isn't integrity-protected. See SECURITY.md for the full discussion.\n- **BYO client lifecycle**: a BYO client is never closed by the Provider.\n\n## Observability\n\nThe Provider does not log. It emits structured callbacks (`WithOnReconnect`, `WithOnResync`, `WithOnWatchError`, `OnEmpty`) and exposes a `Provider.Stats()` snapshot with cumulative counters (reads, puts, deletes, resyncs, reconnects, watch errors, events delivered, debounce flushes) plus the last-seen timestamps. Wire the snapshot into your RED/USE dashboard of choice; `WithOnWatchError` carries a `WatchErrorClass` so you can branch on transient vs. compaction vs. auth without parsing the error string.\n\n## Atomic multi-key writes\n\netcd has no multi-key atomicity in plain `Put`. The optional `etcdwrite` subpackage provides a transactional `PutAll`:\n\n```go\nimport etcdwrite \"github.com/uded/koanf-etcd/write\"\n\netcdwrite.PutAll(ctx, cli, map[string]string{\n    \"/svc/db.host\": \"newhost\",\n    \"/svc/db.port\": \"5433\",\n    \"/svc/db.user\": \"newuser\",\n})\n```\n\nAll three land at the same revision or none do. A watcher with `WithDebounce` coalesces the resulting events into a single reload.\n\n## Options reference (abridged)\n\nSee `go doc github.com/uded/koanf-etcd` for the full surface. Highlights:\n\n| Option | Default |\n| --- | --- |\n| `WithDelim(s)` | `\".\"` |\n| `WithUnflatten(on)` | `true` |\n| `WithTrimPrefix(on)` | `true` (in prefix mode) |\n| `WithReadTimeout(d)` | `5s` |\n| `WithReconnectBackoff(min, max)` | `1s..2min` |\n| `WithDebounce(d)` | `0` (off) |\n| `WithResumeFromRevision(on)` | `true` |\n| `WithStrict(on)` | `false` |\n\n## Versioning \u0026 Go floor\n\n- **Go 1.25+** (this is higher than koanf v2 itself, which is on 1.23 — see below)\n- **koanf v2** (current minor — currently `v2.3.x`)\n- **etcd client/v3 `v3.5.x`** — pinned to the latest 3.5 patch; the 3.5 line is the widely-deployed LTS and is wire-compatible with 3.4/3.5/3.6 etcd servers\n\nSemVer applies once `v1.0.0` is tagged. Pre-1.0 versions may have breaking changes between minor releases (CHANGELOG calls them out, conventional-commits `!` marker as well).\n\n### Why `go 1.25` when koanf v2 is on `go 1.23`?\n\nThis is a real and deliberate gap. **koanf itself** pulls in a tiny runtime tree (`mapstructure` and friends), all of which still build cleanly on Go 1.23. **This provider** pulls in a much heavier dep tree — `go.etcd.io/etcd/client/v3` and its transitive `google.golang.org/grpc` graph, including a large slice of `golang.org/x/*` modules. In mid-2026 that entire grpc/etcd/x-tools ecosystem migrated their go.mod floor to `1.25.0`. Concretely:\n\n- `go.etcd.io/etcd/client/v3 v3.5.31` — the latest 3.5 patch — requires `go 1.25.0`. The last 3.5 patch that allows `go 1.23.0` is `v3.5.21` (about ten patch releases stale).\n- `google.golang.org/grpc v1.81+` requires `go 1.25.0`. `v1.74.x` was the last `go 1.23`-compatible line.\n- Every `golang.org/x/*` module pulled in by mid-2026 grpc/etcd (`x/net`, `x/sys`, `x/text`, `x/crypto`) declares `go 1.25.0`.\n\nWe tried pinning all of those down to keep our floor at `1.23` to match koanf, and it works mechanically — but the price is real:\n\n| | Stay at `go 1.25` (chosen) | Pin everything down to `go 1.23` |\n| --- | --- | --- |\n| etcd | latest 3.5 patch | `v3.5.21` (~10 patches stale) |\n| grpc | latest stable | `v1.74.2` (~7 minors stale) |\n| `golang.org/x/*` | latest | manually pinned-old, fragile |\n| `govulncheck` | **0 advisories** | several non-callable advisories on older `x/net` + `x/sys` |\n| Maintenance | tracks upstream; clean for 6+ months | needs constant pinning every time a new transitive arrives |\n\nWe picked **`go 1.25`**. Reasoning: this is a brand-new library, the security argument is binding, Go 1.25 has been the stable release line since early 2026, and the cost of asking new adopters to use a 6-month-old Go release is small. If you genuinely need `go 1.23` compatibility (build farm on an older toolchain, vendored CI), pin to `v0.1.x` of this library — that line was on `go 1.23` end-to-end. Pre-`v1.0.0` we may revisit if upstream catches up.\n\nThe `go` directive controls only the consumer floor. The `toolchain` directive in `go.mod` is `go1.25.11`, which means anyone on an older Go toolchain (with the default `GOTOOLCHAIN=auto`) will auto-download `1.25.11` on first build — Go ships toolchain self-management out of the box. Consumers can disable that with `GOTOOLCHAIN=local`, in which case their local Go must satisfy the `1.25` floor.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuded%2Fkoanf-etcd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuded%2Fkoanf-etcd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuded%2Fkoanf-etcd/lists"}