{"id":47630305,"url":"https://github.com/eschizoid/kpipe","last_synced_at":"2026-05-20T19:01:34.497Z","repository":{"id":287136883,"uuid":"963704260","full_name":"eschizoid/kpipe","owner":"eschizoid","description":"Composable Kafka consumer library for building modular, testable JVM data pipelines.","archived":false,"fork":false,"pushed_at":"2026-04-11T21:19:49.000Z","size":3064,"stargazers_count":48,"open_issues_count":3,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-04-11T21:22:23.299Z","etag":null,"topics":["apache-kafka","data-pipelines","event-driven","functional-programming","java","kafka","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eschizoid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-04-10T04:57:24.000Z","updated_at":"2026-04-11T21:19:53.000Z","dependencies_parsed_at":"2025-05-16T07:30:48.762Z","dependency_job_id":"ececbacb-790c-4219-9317-28acc2b813ee","html_url":"https://github.com/eschizoid/kpipe","commit_stats":null,"previous_names":["eschizoid/kpipe"],"tags_count":26,"template":false,"template_full_name":null,"purl":"pkg:github/eschizoid/kpipe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eschizoid%2Fkpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eschizoid%2Fkpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eschizoid%2Fkpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eschizoid%2Fkpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eschizoid","download_url":"https://codeload.github.com/eschizoid/kpipe/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eschizoid%2Fkpipe/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31873607,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"online","status_checked_at":"2026-04-16T02:00:06.042Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","data-pipelines","event-driven","functional-programming","java","kafka","stream-processing"],"created_at":"2026-04-01T23:21:23.114Z","updated_at":"2026-05-20T19:01:34.487Z","avatar_url":"https://github.com/eschizoid.png","language":"Java","funding_links":[],"categories":["进程间通信"],"sub_categories":["Spring Cloud框架"],"readme":"# \u003cimg src=\"img/kpipe_1.png\" width=\"200\" height=\"200\"\u003e\n\n# KPipe\n\nA Kafka consumer library for Java 25. Built around virtual threads, a functional pipeline API, and at-least-once\ndelivery — small enough to read end-to-end in an afternoon.\n\n[![JVM 25+](https://img.shields.io/badge/JVM-25%2B-brightgreen.svg?\u0026logo=openjdk)](https://openjdk.org/projects/jdk/25/)\n[![Build](https://github.com/eschizoid/kpipe/actions/workflows/ci.yaml/badge.svg)](https://github.com/eschizoid/kpipe/actions/workflows/ci.yaml)\n[![Codecov](https://codecov.io/gh/eschizoid/kpipe/graph/badge.svg?token=X50GBU4X7J)](https://codecov.io/gh/eschizoid/kpipe)\n[![Maven Central](https://img.shields.io/maven-central/v/io.github.eschizoid/kpipe-consumer.svg?label=Maven%20Central)](https://central.sonatype.com/artifact/io.github.eschizoid/kpipe-consumer)\n[![Javadoc](https://javadoc.io/badge2/io.github.eschizoid/kpipe-api/javadoc.svg?color=purple)](https://javadoc.io/doc/io.github.eschizoid/kpipe-api)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\nWhat it does:\n\n- Virtual-thread-per-record processing (no thread pool to tune)\n- Composable `UnaryOperator\u003cT\u003e` pipelines that deserialize once and serialize once\n- At-least-once delivery via lowest-pending-offset commits, even with parallel processing\n- In-flight or lag-based backpressure with hysteresis\n- Dead-letter routing, retries, and a circuit breaker — one fluent setter each\n- Multi-topic dispatch (homogeneous or heterogeneous-format) through a single consumer / consumer-group\n- Minimal framework code on the hot path\n\nThe target audience is anyone running Kafka consumers that transform, enrich, or route — and would rather not glue their\nown consumer loop together every time.\n\n### Two API surfaces\n\nThere are two public entry points; pick whichever matches the shape of your problem:\n\n| Surface                                                | What it gives you                                                                                                                                                                   | When to use                                                                    |\n| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |\n| **`KPipe` fluent facade** (`kpipe-api`)                | 5-line `KPipe.json(\"topic\", props).pipe(...).toConsole().start()`. Returns a `Stream\u003cT\u003e` → `Sink\u003cT\u003e` → `Handle` chain. Immutable, IDE-discoverable.                                 | The common path — most users start here.                                       |\n| **Registry + Builder explicit API** (`kpipe-consumer`) | `MessageProcessorRegistry` + `KPipeConsumer.Builder`. Multi-step, supports custom registries, shared pipelines, custom offset managers, periodic metrics reporting via the builder. | Custom offset managers, multi-pipeline-per-consumer, advanced lifecycle hooks. |\n\nThe facade is a thin layer on top of the explicit API, so dropping down when you outgrow it doesn't cost anything. See\n[`docs/escape-hatches.md`](docs/escape-hatches.md) for the full capability map and worked examples of the explicit-only\nfeatures (custom `OffsetManager`, rebalance listeners, pre-shared registries, etc.).\n\n---\n\n## Getting started\n\n### 1. Add the dependencies\n\nFor the 5-line fluent path (recommended), pull `kpipe-api` plus the format module(s) you need:\n\n```kotlin\n// Gradle (Kotlin) — JSON via the fluent API\nimplementation(\"io.github.eschizoid:kpipe-api:1.12.0\")\nimplementation(\"io.github.eschizoid:kpipe-format-json:1.12.0\")\n```\n\n```xml\n\u003c!-- Maven — JSON via the fluent API --\u003e\n\u003cdependency\u003e\n  \u003cgroupId\u003eio.github.eschizoid\u003c/groupId\u003e\n  \u003cartifactId\u003ekpipe-api\u003c/artifactId\u003e\n  \u003cversion\u003e1.12.0\u003c/version\u003e\n\u003c/dependency\u003e\n\u003cdependency\u003e\n  \u003cgroupId\u003eio.github.eschizoid\u003c/groupId\u003e\n  \u003cartifactId\u003ekpipe-format-json\u003c/artifactId\u003e\n  \u003cversion\u003e1.12.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n`kpipe-api` transitively pulls `kpipe-consumer` + `kpipe-producer` + `kpipe-core`. Skip `kpipe-api` only if you want the\nexplicit registry / builder API (see \"Advanced API\" further down) — for that case, depend on `kpipe-consumer` directly.\n\nThere's also a `kpipe-bom` so you only pin one version across modules — use it via `dependencyManagement` (Maven) or\n`enforcedPlatform` (Gradle) and drop versions from the individual `kpipe-*` dependencies.\n\n\u003cdetails\u003e\n\u003csummary\u003eModule catalog \u0026 other build tools\u003c/summary\u003e\n\n| Module                            | What it gives you                                                                                                                                           |\n| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `kpipe-api`                       | High-level fluent entry point: `KPipe`, `Stream\u003cT\u003e`, `Sink\u003cT\u003e`, `Handle`                                                                                    |\n| `kpipe-bom`                       | Maven BOM — pins all `kpipe-*` artifacts to matching versions                                                                                               |\n| `kpipe-core`                      | Low-level building blocks: registries, `MessageFormat`, `MessageSink`, operators, `BatchSink`                                                               |\n| `kpipe-metrics`                   | Metrics interfaces (`ConsumerMetrics`, `ProducerMetrics`) + log-based reporters                                                                             |\n| `kpipe-metrics-otel`              | OpenTelemetry-backed implementation (opt-in)                                                                                                                |\n| `kpipe-tracing-otel`              | W3C trace context propagation through Kafka headers (opt-in)                                                                                                |\n| `kpipe-schema-registry-confluent` | Confluent Schema Registry client — `lookupById` + `lookupBySubjectVersion` (opt-in)                                                                         |\n| `kpipe-producer`                  | Kafka producer wrapper, `KafkaMessageSink`, `Tracer` SPI                                                                                                    |\n| `kpipe-consumer`                  | `KPipeConsumer` (hosts lifecycle, metrics-reporter thread, shutdown hook), `BackpressureController`, `CircuitBreakerController`, `ConsumerHealthController` |\n| `kpipe-format-json`               | `JsonFormat`, `JsonConsoleSink`                                                                                                                             |\n| `kpipe-format-avro`               | `AvroFormat`, `AvroConsoleSink`                                                                                                                             |\n| `kpipe-format-protobuf`           | `ProtobufFormat`, `ProtobufConsoleSink`                                                                                                                     |\n\n**Gradle (Kotlin) with BOM**\n\n```kotlin\nimplementation(platform(\"io.github.eschizoid:kpipe-bom:1.12.0\"))\nimplementation(\"io.github.eschizoid:kpipe-api\")\nimplementation(\"io.github.eschizoid:kpipe-format-json\")\n// add kpipe-metrics-otel only if you want OpenTelemetry-backed metrics\nimplementation(\"io.github.eschizoid:kpipe-metrics-otel\")\n```\n\n**Gradle (Groovy)**\n\n```groovy\nimplementation platform('io.github.eschizoid:kpipe-bom:1.12.0')\nimplementation 'io.github.eschizoid:kpipe-api'\nimplementation 'io.github.eschizoid:kpipe-format-json'\n```\n\n**Maven (with BOM)**\n\n```xml\n\u003cdependencyManagement\u003e\n  \u003cdependencies\u003e\n    \u003cdependency\u003e\n      \u003cgroupId\u003eio.github.eschizoid\u003c/groupId\u003e\n      \u003cartifactId\u003ekpipe-bom\u003c/artifactId\u003e\n      \u003cversion\u003e1.12.0\u003c/version\u003e\n      \u003ctype\u003epom\u003c/type\u003e\n      \u003cscope\u003eimport\u003c/scope\u003e\n    \u003c/dependency\u003e\n  \u003c/dependencies\u003e\n\u003c/dependencyManagement\u003e\n\n\u003cdependencies\u003e\n  \u003cdependency\u003e\n    \u003cgroupId\u003eio.github.eschizoid\u003c/groupId\u003e\n    \u003cartifactId\u003ekpipe-api\u003c/artifactId\u003e\n  \u003c/dependency\u003e\n  \u003cdependency\u003e\n    \u003cgroupId\u003eio.github.eschizoid\u003c/groupId\u003e\n    \u003cartifactId\u003ekpipe-format-json\u003c/artifactId\u003e\n  \u003c/dependency\u003e\n\u003c/dependencies\u003e\n```\n\n**SBT**\n\n```sbt\nlibraryDependencies += \"io.github.eschizoid\" % \"kpipe-api\" % \"1.12.0\"\nlibraryDependencies += \"io.github.eschizoid\" % \"kpipe-format-json\" % \"1.12.0\"\n```\n\n\u003c/details\u003e\n\n### 2. Hello, KPipe — five lines\n\n```java\nimport org.kpipe.KPipe;\nimport static org.kpipe.registry.Operators.removeFields;\n\nKPipe.json(\"events\", kafkaProps)\n    .pipe(msg -\u003e { msg.put(\"ts\", System.currentTimeMillis()); return msg; })\n    .pipe(removeFields(\"password\"))\n    .toConsole()\n    .start();   // returns AutoCloseable Handle (call .close() to shut down)\n```\n\nA working JSON Kafka consumer that strips the `password` field, stamps a timestamp, and logs to console. Behind the\nscenes the chain assembles a `MessageProcessorRegistry` and a `KPipeConsumer` — you don't have to touch either of them.\n\n### 2b. Production wiring — ten lines\n\nThe same shape with the operational stack turned on: retry, backpressure with default watermarks, a dead-letter topic,\nand a custom sink instead of the console. Everything below is one-liner toggle on top of the §2 chain.\n\n```java\ntry (var handle = KPipe.json(\"orders\", kafkaProps)\n    .pipe(enrich)\n    .filter(order -\u003e order.total() \u003e 0)\n    .withRetry(3, Duration.ofMillis(100))\n    .withBackpressure()                            // pause at 10k in-flight, resume at 7k\n    .withDeadLetterTopic(\"orders.dlq\")             // bad records flow here after retries exhaust\n    .onFailed(cause -\u003e log.warn(\"processing failed\", cause))   // observer, not a handler — log only\n    .toCustom(WarehouseSink.create())\n    .start()) {\n  handle.awaitShutdown();\n}\n```\n\n`Handle` is `AutoCloseable` and `close()` calls `shutdownGracefully(Duration.ofSeconds(5))` by default. The DLQ wires a\n`KafkaProducer` from the same properties; provide your own via `.withDeadLetterBundle(...)` if you need a different\nconfiguration. See [`docs/escape-hatches.md`](docs/escape-hatches.md) for the full set of explicit-only options (custom\n`OffsetManager`, rebalance listeners, multi-topic heterogeneous dispatch).\n\n### 3. The full fluent surface\n\nThe `Stream\u003cT\u003e` returned by `KPipe.json/avro/protobuf/bytes/custom(...)` is the API. Type a `.` after it and the IDE\nshows you everything that exists:\n\n| Method                                                             | What it does                                                            |\n| ------------------------------------------------------------------ | ----------------------------------------------------------------------- |\n| `.pipe(UnaryOperator\u003cT\u003e op)`                                       | append an operator to the pipeline                                      |\n| `.filter(Predicate\u003cT\u003e keep)`                                       | drop messages where predicate returns false                             |\n| `.peek(Consumer\u003cT\u003e sideEffect)`                                    | observe without modifying (logging, metrics)                            |\n| `.when(Predicate, ifTrue, ifFalse)`                                | branch the pipeline conditionally                                       |\n| `.skipBytes(int n)`                                                | drop a leading wire-format prefix (Confluent: 5 for Avro, 6 for Proto)  |\n| `.withRetry(int max, Duration backoff)`                            | configure retry behavior                                                |\n| `.withBackpressure()` / `.withBackpressure(high, low)`             | enable backpressure with default or explicit watermarks                 |\n| `.withSequentialProcessing(boolean)`                               | force one-at-a-time per partition                                       |\n| `.withCircuitBreaker(double threshold, int window, Duration open)` | open the circuit when sink failure rate exceeds threshold (see below)   |\n| `.withTracer(Tracer t)`                                            | propagate W3C trace context through Kafka headers                       |\n| `.withDeadLetterTopic(String)`                                     | route failed records to a DLQ topic after retries are exhausted         |\n| `.withErrorHandler(Consumer\u003cProcessingError\u003cbyte[]\u003e\u003e)`             | custom failure handler (DB, alerting, anything not a Kafka topic)       |\n| `.withMetrics(ConsumerMetrics m)`                                  | wire OTel or custom metrics (default `ConsumerMetrics.noop()`)          |\n| `.withPollTimeout(Duration d)`                                     | override Kafka poll timeout (default 100ms)                             |\n| `.toConsole()`                                                     | terminate with the format-appropriate console sink                      |\n| `.toCustom(MessageSink\u003cT\u003e sink)`                                   | terminate with your own sink                                            |\n| `.toMulti(MessageSink\u003cT\u003e... sinks)`                                | fan-out to multiple sinks                                               |\n| `.toBatch(BatchSink\u003cT\u003e sink, BatchPolicy policy)`                  | terminate with a batch sink (size / age flush, optional per-record DLQ) |\n\nThe terminal `Sink\u003cT\u003e.start()` returns a `Handle` exposing `isHealthy()`, `metrics()`, `awaitShutdown(Duration)`,\n`shutdownGracefully(Duration)`, and `close()`.\n\n### 4. Consuming from multiple topics\n\nTwo flavors, depending on whether the topics share a payload type.\n\nHomogeneous — many topics, one shared pipeline (the \"partitioned-by-region versions of the same event\" case):\n\n```java\nKPipe.json(Set.of(\"events-us\", \"events-eu\", \"events-ap\"), kafkaProps)\n    .pipe(addTimestamp)\n    .toCustom(captureSink)\n    .start();\n```\n\nThe `Collection\u003cString\u003e` overload exists on every entry point — `KPipe.json/avro/protobuf/bytes/custom`.\n\nHeterogeneous — many topics, each with its own payload type and its own pipeline, all dispatched through one consumer /\none consumer-group / one offset manager:\n\n```java\nKPipe.multi(kafkaProps)\n    .json(\"events-json\",  s -\u003e s.pipe(addTimestamp).toCustom(jsonSink))\n    .avro(\"events-avro\",  s -\u003e s.filter(active).toCustom(avroSink))\n    .bytes(\"events-raw\",  s -\u003e s.toCustom(rawSink))\n    .start();\n```\n\nEach route gets its own typed `Stream\u003cT\u003e`. Records arriving for topics not registered with a route are dropped at\n`WARNING` and their offsets are still committed (no infinite retry on a config error).\n\n### 5. Common operator patterns\n\n`org.kpipe.registry.Operators` exposes pure-function helpers ready to drop into `.pipe(...)`:\n\n```java\nimport static org.kpipe.registry.Operators.*;\n\nKPipe.json(\"events\", kafkaProps)\n    .filter(msg -\u003e \"active\".equals(msg.get(\"status\")))            // drop inactive\n    .peek(msg -\u003e log.info(\"processing {}\", msg.get(\"id\")))        // log without modifying\n    .pipe(rename(\"user_id\", \"userId\"))                            // rename a field\n    .pipe(removeFields(\"password\", \"ssn\"))                        // strip sensitive fields\n    .pipe(safe(msg -\u003e riskyEnrich(msg)))                          // wrap in error-handling\n    .toConsole()\n    .start();\n```\n\nThe full operator vocabulary: `filter`, `drop`, `peek`, `map`, `compose`, `safe`, `requireField`, `rename`,\n`removeFields`, `addField`, `transformField`. All return `UnaryOperator\u003cT\u003e` (or `UnaryOperator\u003cMap\u003cString, Object\u003e\u003e` for\nthe `Map`-typed ones).\n\n---\n\n## When to use KPipe\n\nKPipe is a good fit if you're building Kafka consumer microservices, event enrichment pipelines, or lightweight\ntransformation services — especially anything I/O-bound (REST calls, DB lookups) where virtual threads pay off. If\nyou're already moving toward Java 25 concurrency, KPipe assumes that world by default.\n\nIt's not trying to replace Kafka Streams or Flink. The scope ends at \"consume, transform, route, produce\" — no\nwindowing, no joins, no state stores.\n\n---\n\n## Why not Kafka Streams or Spring Kafka?\n\nKPipe is for code-first pipelines with minimal infrastructure overhead.\n\n| Capability                       | Kafka Streams | Spring Kafka            | Reactor Kafka | KPipe |\n| -------------------------------- | ------------- | ----------------------- | ------------- | ----- |\n| Full stream processing framework | Yes           | No                      | No            | No    |\n| Lightweight consumer pipelines   | Partial       | Yes                     | Yes           | Yes   |\n| Virtual-thread friendly          | No            | Listener-container only | No            | Yes   |\n| Functional pipeline API          | Yes           | No (annotations)        | Yes           | Yes   |\n| Minimal runtime dependencies     | No            | No (Spring Boot)        | Yes           | Yes   |\n| JPMS modules (no Spring context) | No            | No                      | Partial       | Yes   |\n| Native multi-topic dispatch      | Yes (DSL)     | Per `@KafkaListener`    | Manual        | Yes   |\n\nKPipe sits between raw `KafkaConsumer` code and full streaming frameworks.\n\n### Coming from Spring Kafka?\n\nSpring Kafka covers the same niche — \"I want a Kafka consumer with retries, DLQ, backpressure, and a typed payload\" —\nbut bundles Spring Boot, the application context, and annotation-driven wiring. KPipe targets the same workload with no\nSpring runtime, no annotation processor, no `KafkaListenerContainerFactory` to configure, and a fluent API that's\ndiscoverable via IDE autocomplete.\n\nWhat changes:\n\n- No Spring Boot on the classpath. KPipe runs on plain `java -jar` — eight JPMS modules, no reflection, no AOP, no\n  application context to manage.\n- Virtual threads by default. Spring Kafka's listener container is still thread-pool-per-partition; KPipe runs\n  thread-per-record on virtual threads, which is where I/O-bound enrichment scales without pool tuning.\n- No `@KafkaListener` indirection. Pipelines are values you build in `main` and pass around. No classpath scanning, no\n  proxied beans, no `@EnableKafka` toggle to remember.\n- Per-topic typing without a class per listener. `KPipe.multi(props).json(\"a\", ...).avro(\"b\", ...)` wires heterogeneous\n  routes through one consumer group and one offset manager. Spring's equivalent is one `@KafkaListener` method per\n  topic, each with its own container.\n- Errors throw instead of getting swallowed. `MessagePipeline.apply()` throws on failure; `null` means \"intentionally\n  filtered\" and only that. Spring Kafka's default `ErrorHandlingDeserializer` + `RetryableTopic` chain is more flexible\n  but also easier to misconfigure into silent drops.\n- Metrics are pluggable. `ProducerMetrics` and `ConsumerMetrics` are interfaces; the `kpipe-metrics-otel` adapter is\n  opt-in. No `MeterRegistry` autowiring.\n\nRough migration sketch — a Spring Kafka listener:\n\n```java\n// Spring Kafka\n@KafkaListener(topics = \"events\", groupId = \"my-app\")\npublic void onMessage(ConsumerRecord\u003cString, Map\u003cString, Object\u003e\u003e rec) {\n  rec.value().put(\"ts\", System.currentTimeMillis());\n  sink.accept(rec.value());\n}\n```\n\nbecomes:\n\n```java\n// KPipe\nKPipe.json(\"events\", kafkaProps)\n    .pipe(msg -\u003e { msg.put(\"ts\", System.currentTimeMillis()); return msg; })\n    .toCustom(sink::accept)\n    .start();\n```\n\nRetry, DLQ, backpressure, and graceful shutdown are fluent calls on the same `Stream\u003cT\u003e`. No `RetryTemplate`, no\n`DeadLetterPublishingRecoverer`, no `ContainerProperties.AckMode` to pick.\n\nIf you depend on `@KafkaListener` lifecycle integration with Spring Boot actuator, transactional Kafka producers wired\ninto `@Transactional`, or the broader Spring ecosystem (security, config-server, sleuth tracing), stay where you are.\nKPipe doesn't have those.\n\n---\n\n## Architecture and reliability\n\nKPipe is a lightweight Kafka consumer library that leans hard on Java 25 features — virtual threads, sealed types,\nrecords, JPMS — to keep the runtime predictable.\n\n### 1. Modular Architecture (JPMS)\n\nKPipe ships eight focused JPMS modules with a clean dependency direction (no cycles, no sideways leaks):\n\n```\nkpipe-metrics ← kpipe-core ← kpipe-consumer\n                          ← kpipe-producer\n                          ← kpipe-format-{json, avro, protobuf}\n                          ← kpipe-api  (KPipe fluent facade)\nkpipe-metrics-otel              ← kpipe-metrics    (opt-in OTel metrics)\nkpipe-tracing-otel              ← kpipe-producer   (opt-in OTel tracing)\nkpipe-schema-registry-confluent ← kpipe-core       (opt-in Confluent SR client)\nkpipe-bom                                          (Maven BOM — pins versions)\n```\n\n- **kpipe-core**: format-agnostic pipeline machinery — `MessageFormat`, `MessagePipeline`, `MessageProcessorRegistry`,\n  `MessageSink`, `CompositeMessageSink`, `RegistryKey`, `RegistryFunctions`. No Kafka, no third-party runtime deps.\n- **kpipe-metrics**: pure interfaces (`ProducerMetrics`, `ConsumerMetrics`) plus log-based reporters. No OTel API on the\n  classpath.\n- **kpipe-metrics-otel**: OpenTelemetry implementation (`OtelConsumerMetrics`, `OtelProducerMetrics`). Add this only if\n  you want OTel-backed metrics.\n- **kpipe-producer**: Kafka producer wrapper, `KafkaMessageSink` (in `org.kpipe.producer.sink`), `Tracer` SPI.\n- **kpipe-consumer**: `KPipeConsumer` (hosts lifecycle: `start` / `close` / `awaitShutdown` / `shutdownGracefully` /\n  `waitForInFlightDrain` + optional metrics-reporter thread and JVM shutdown hook), `BackpressureController`,\n  `CircuitBreakerController`, `ConsumerHealthController`, `KafkaOffsetManager`, `HttpHealthServer`, `consumer.metrics`\n  reporters.\n- **kpipe-tracing-otel**: W3C trace context propagation — extract on consume, inject on produce / DLQ. Opt-in module\n  (see \"Distributed tracing\" below).\n- **kpipe-schema-registry-confluent**: Confluent Schema Registry client (`ConfluentSchemaResolver`) for schema-by-ID and\n  by-subject-version lookup. Opt-in (see \"Confluent Schema Registry\" below).\n- **kpipe-format-json / -avro / -protobuf**: per-format `XFormat.INSTANCE`, processors, and console sinks. Pull only the\n  format(s) you need.\n\nUse KPipe in a modular project:\n\n```java\nmodule my.application {\n  requires org.kpipe.consumer; // includes core + producer + metrics transitively\n  requires org.kpipe.format.json; // add only the formats you use\n  requires org.kpipe.metrics.otel; // optional — enables OTel-backed metrics\n}\n```\n\n### 2. Single SerDe cycle\n\nMost pipelines do `byte[] → Object → byte[]` at every step. KPipe doesn't:\n\n- Deserialize once into a mutable representation (`Map` for JSON, `GenericRecord` for Avro) via `MessagePipeline`.\n- Apply a chain of `UnaryOperator` functions to that same object.\n- Serialize back to `byte[]` once at the end.\n- Typed sinks can attach directly to the pipeline and receive the object before final serialization, skipping it\n  entirely.\n\nThe win is fewer allocations and less CPU spent on intermediate SerDe steps. The benchmark numbers are in\n`benchmarks/README.md`.\n\n### 3. Virtual threads\n\nKPipe runs on Java virtual threads (Project Loom) for concurrency.\n\n- Thread-per-record. Each message gets its own virtual thread, so I/O-bound enrichment scales without explicit pool\n  sizing.\n- No `ThreadLocal` on the hot path. `ThreadLocal` doesn't compose with thread-per-record — every record would get a\n  fresh map, defeating any caching intent. If a future need for thread-local-like state turns up (tenant context, span\n  propagation), we'll reach for `ScopedValue`; the codebase currently has neither.\n\n### 4. At-least-once delivery via lowest pending offset\n\nKPipe never commits past an in-flight record. The implementation:\n\n- `OffsetManager` is an interface — Kafka-backed by default, but you can plug in external storage.\n- Every in-flight offset is tracked in a `ConcurrentSkipListSet` per partition (see `KafkaOffsetManager`).\n- Offset 102 cannot be committed until 101 finishes, even if 102 completes first. No gaps.\n- On crash, the consumer resumes from the last committed offset. Some records may be reprocessed — standard\n  at-least-once behavior — but nothing is skipped.\n\n### 5. Parallel vs. sequential processing\n\nTwo modes, picked based on whether per-partition order matters:\n\n- Parallel (default). Stateless transformations like enrichment or masking. High throughput on virtual threads. Offsets\n  commit by lowest pending offset.\n- Sequential (`.withSequentialProcessing(true)`). Stateful transformations where per-partition order matters (balance\n  updates, sequence-dependent events). One message per partition at a time. Backpressure switches to monitoring consumer\n  lag — the gap between partition end-offset and consumer position — since in-flight count is always ≤ 1.\n\n### 6. External offset management\n\nKafka-backed offset storage is the default. For exactly-once processing or external coordination needs, plug in your own\n`OffsetManager`.\n\n1.  **Seek on Assignment**: When partitions are assigned, fetch the last processed offset from your database and call\n    `consumer.seek(partition, offset + 1)`.\n2.  **Update on Processed**: Implement `markOffsetProcessed` to save the offset to the database.\n\n```java\npublic class PostgresOffsetManager\u003cK, V\u003e implements OffsetManager\u003cK, V\u003e {\n\n  private final Consumer\u003cK, V\u003e consumer;\n\n  // ... DB connection ...\n\n  @Override\n  public void markOffsetProcessed(final ConsumerRecord\u003cK, V\u003e record) {\n    // SQL: UPDATE offsets SET offset = ? WHERE partition = ?\n  }\n\n  @Override\n  public ConsumerRebalanceListener createRebalanceListener() {\n    return new ConsumerRebalanceListener() {\n      @Override\n      public void onPartitionsAssigned(final Collection\u003cTopicPartition\u003e partitions) {\n        for (final var tp : partitions) {\n          long lastOffset = fetchFromDb(tp);\n          consumer.seek(tp, lastOffset + 1);\n        }\n      }\n      // ...\n    };\n  }\n  // ...\n}\n```\n\n### 7. Error handling and retries\n\nError handling is layered:\n\n- Retries: `.withRetry(maxRetries, backoff)` for transient failures.\n- Dead-letter topic:\n  ```java\n  final var consumer = KPipeConsumer.\u003cbyte[]\u003ebuilder()\n    .withDeadLetterTopic(\"events-dlq\") // sent here after retries are exhausted\n    .build();\n  ```\n- Custom error handler: `.withErrorHandler(...)` to route failures somewhere other than a Kafka topic — an external DB,\n  alerting system, whatever.\n- Per-processor and per-sink wrapping: `MessageProcessorRegistry.withOperatorErrorHandling()` and\n  `MessageProcessorRegistry.withSinkErrorHandling()` swallow failures at the boundary so one bad processor doesn't take\n  down the pipeline.\n\n### 8. Backpressure\n\nWhen a downstream sink (database, HTTP API, another Kafka topic) is slow, KPipe pauses Kafka polling instead of letting\nin-flight work or lag pile up unbounded.\n\nIt uses two watermarks with hysteresis so it doesn't oscillate:\n\n- High watermark — pause polling when the monitored metric crosses this value.\n- Low watermark — resume polling when the metric drops back to or below this value.\n\n#### Strategies\n\nKPipe picks the strategy based on processing mode:\n\n| Mode               | Strategy     | Monitored              | Why                                                |\n| :----------------- | :----------- | :--------------------- | :------------------------------------------------- |\n| Parallel (default) | In-flight    | Active virtual threads | Cap concurrent in-flight work to bound memory      |\n| Sequential         | Consumer lag | Total unread messages  | In-flight is always ≤ 1, so lag is the real signal |\n\n##### Parallel mode: in-flight count\n\nMany records run concurrently on virtual threads. The controller watches how many are currently in-flight (started but\nnot finished).\n\n- High watermark default: 10,000\n- Low watermark default: 7,000\n\n##### Sequential mode: consumer lag\n\nSequential mode processes one record per partition at a time, so in-flight is always ≤ 1. KPipe watches consumer lag\nacross all assigned partitions instead:\n\n```\nlag = Σ (endOffset - position)\n```\n\n- `endOffset` — highest offset available in the partition.\n- `position` — offset of the next record this consumer will fetch.\n\n- High watermark default: 10,000\n- Low watermark default: 7,000\n\n#### Configuration\n\n```java\nfinal var consumer = KPipeConsumer.\u003cbyte[]\u003ebuilder()\n  .withProperties(kafkaProps)\n  .withTopic(\"events\")\n  .withPipeline(pipeline)\n  // Default watermarks: 10k high / 7k low\n  .withBackpressure()\n  // Or explicit:\n  // .withBackpressure(5_000, 3_000)\n  .build();\n```\n\nBackpressure is on by default with 10k/7k watermarks. Override with `.withBackpressure(high, low)` for your workload.\n\nPauses log at WARNING, resumes at INFO. Two dedicated metrics track them: `backpressurePauseCount` and\n`backpressureTimeMs`.\n\n### 9. Circuit breaker\n\nBackpressure caps in-flight work and prevents memory blow-ups, but it doesn't stop the bleeding when a downstream sink\nis _failing_ (DB connection refused, HTTP 503). Without a circuit breaker, every record during an outage runs\n`maxRetries + 1` attempts and lands in the DLQ. The breaker stops the cascade by pausing polling once the sink failure\nrate crosses a threshold, then probes recovery with a single record after a cool-down window.\n\n```java\nimport java.time.Duration;\nimport org.kpipe.KPipe;\n\nKPipe.json(\"events\", kafkaProps)\n    .pipe(enrich)\n    .withCircuitBreaker(\n        0.5,                          // failure threshold (50% over the window)\n        100,                          // window size (last N outcomes)\n        Duration.ofSeconds(30))       // open duration before half-open probe\n    .toCustom(flakySink)\n    .start();\n```\n\nThe state machine is the standard CLOSED → OPEN → HALF_OPEN → CLOSED cycle:\n\n- **CLOSED**: outcomes feed a sliding window. When the rolling failure rate exceeds the threshold, transition to OPEN.\n- **OPEN**: poll is paused. After `openDuration`, transition to HALF_OPEN.\n- **HALF_OPEN**: poll resumes; the next record is the probe. Success → CLOSED (and the window resets). Failure → back to\n  OPEN for another `openDuration`.\n\nCB and backpressure pause through the same `ConsumerHealthController` (bitmask of MANUAL / BACKPRESSURE /\nCIRCUIT_BREAKER sources), so they don't fight each other — releasing the backpressure source while the CB still holds\nkeeps the consumer paused, and vice-versa.\n\nOTel counters (opt-in via `kpipe-metrics-otel`):\n\n| Instrument                                     | Type    | Description                                  |\n| ---------------------------------------------- | ------- | -------------------------------------------- |\n| `kpipe.consumer.circuit_breaker.trips`         | Counter | Times the breaker transitioned CLOSED → OPEN |\n| `kpipe.consumer.circuit_breaker.state_changes` | Counter | Any state transition, attribute-tagged       |\n| `kpipe.consumer.circuit_breaker.time_open`     | Counter | Total ms spent in OPEN                       |\n\nWhen `.withCircuitBreaker(...)` is omitted, the consumer never trips and the counters stay at zero.\n\n### 10. Distributed tracing (W3C trace context)\n\nKPipe propagates W3C trace context (`traceparent` / `tracestate` Kafka headers) end-to-end: extract on consume, inject\non produce and on DLQ writes. The implementation lives in the opt-in `kpipe-tracing-otel` module — `kpipe-core` and\n`kpipe-consumer` are dependency-free, you bring the OTel SDK at runtime.\n\nAdd the dependency:\n\n```kotlin\nimplementation(\"io.github.eschizoid:kpipe-tracing-otel:1.12.0\")\n```\n\nWire it on the stream:\n\n```java\nimport io.opentelemetry.api.OpenTelemetry;\nimport org.kpipe.KPipe;\nimport org.kpipe.tracing.otel.OtelTracer;\n\nfinal OpenTelemetry otel = /* GlobalOpenTelemetry.get() or your SDK */;\n\nKPipe.json(\"events\", kafkaProps)\n    .withTracer(new OtelTracer(otel, \"events-consumer\"))\n    .pipe(enrich)\n    .toCustom(producerSink)\n    .start();\n```\n\nWhat it does on the hot path:\n\n1. On poll, extracts the upstream context from `traceparent` / `tracestate` headers using the standard W3C propagator.\n2. Starts a CONSUMER span with `messaging.kafka.{topic,partition,offset}` attributes; the upstream span is the parent.\n3. Closes the span in a nested `finally` so a throwing user callback can't leak the scope.\n4. On produce (`KafkaMessageSink.accept`) and DLQ writes (`KPipeProducer.sendToDlq`), injects the _current_ context into\n   outbound headers — downstream consumers see the same trace.\n\nWithout `.withTracer(...)`, `Tracer.noop()` is used: zero allocation, no OTel API on the classpath.\n\n### 11. Graceful shutdown and interrupts\n\nKPipe respects JVM signals without losing records:\n\n- Interrupts trigger a coordinated shutdown. They don't cause records to be skipped.\n- If a record's processing is interrupted mid-flight (during retry backoff or transformation), its offset is not marked\n  as processed. The next consumer instance picks it up — at-least-once still holds during shutdown.\n\n```java\n// Initiate graceful shutdown with 5-second timeout\nboolean allProcessed = runner.shutdownGracefully(5000);\n\n// Or register a JVM shutdown hook via the builder (default off)\nKPipeConsumer.\u003cbyte[]\u003ebuilder()\n  .withShutdownHook(true)\n  // ...\n  .build();\n```\n\n---\n\n## Working with messages\n\nPipelines deserialize once, transform many times, serialize once. Operators are `UnaryOperator\u003cT\u003e` where `T` is the\nformat's typed payload — `Map\u003cString, Object\u003e` for JSON, `GenericRecord` for Avro, `Message` for Protobuf. The generic\nhelpers in [`Operators`](lib/kpipe-core/src/main/java/org/kpipe/registry/Operators.java) (`filter`, `drop`, `peek`,\n`map`, `compose`, `safe`, plus the map-specific `addField`, `removeFields`, `transformField`, `requireField`, `rename`)\ncover the common cases. For anything else, inline lambdas using the format's native API.\n\n### JSON\n\nAdd `kpipe-format-json`. Operators are `UnaryOperator\u003cMap\u003cString, Object\u003e\u003e`:\n\n```java\nimport static org.kpipe.registry.Operators.*;\n\nimport org.kpipe.format.json.JsonFormat;\nimport org.kpipe.registry.MessageProcessorRegistry;\nimport org.kpipe.registry.RegistryKey;\n\nfinal var registry = new MessageProcessorRegistry(\"myApp\");\n\nfinal var stampKey = RegistryKey.json(\"addTimestamp\");\nregistry.registerOperator(stampKey, addField(\"processedAt\", System.currentTimeMillis()));\n\nfinal var sanitizeKey = RegistryKey.json(\"sanitize\");\nregistry.registerOperator(sanitizeKey, removeFields(\"password\", \"ssn\"));\n\nfinal var lowerEmailKey = RegistryKey.json(\"lowerEmail\");\nregistry.registerOperator(lowerEmailKey, transformField(\"email\", v -\u003e ((String) v).toLowerCase()));\n\n// Single deserialization → many transformations → single serialization\nfinal var pipeline = registry.pipeline(JsonFormat.INSTANCE)\n    .add(sanitizeKey)\n    .add(lowerEmailKey)\n    .add(stampKey)\n    .build();\n```\n\n### Avro\n\nAdd `kpipe-format-avro`. Operators are `UnaryOperator\u003cGenericRecord\u003e`:\n\n```java\nimport org.kpipe.format.avro.AvroFormat;\nimport org.kpipe.format.avro.AvroRegistryKey;\nimport org.kpipe.registry.MessageProcessorRegistry;\n\nfinal var registry = new MessageProcessorRegistry(\"myApp\");\n\n// Build an AvroFormat bound to a single schema. Use new AvroFormat(schema) when you already have a\n// parsed Schema, or AvroFormat.of(schemaJson) for inline JSON. For multiple schemas keyed by name\n// use AvroSchemaCatalog and pass catalog.get(\"user\") into AvroFormat.\nfinal var format = AvroFormat.of(\"\"\"\n  {\"type\":\"record\",\"name\":\"User\",\"namespace\":\"com.kpipe\",\"fields\":[\n    {\"name\":\"id\",\"type\":\"string\"},{\"name\":\"name\",\"type\":\"string\"}\n  ]}\"\"\");\n\n// Avro records are schema-bound: use inline lambdas with the native Avro API for value transforms.\n// `Operators.filter`, `tap`, `compose` etc. work for any payload type, including GenericRecord.\nfinal var lowerNameKey = AvroRegistryKey.of(\"lowerName\");\nregistry.registerOperator(lowerNameKey, record -\u003e {\n  if (record.get(\"name\") != null) record.put(\"name\", record.get(\"name\").toString().toLowerCase());\n  return record;\n});\n\nfinal var pipeline = registry.pipeline(format).add(lowerNameKey).build();\n\n// For Confluent Wire Format (1 magic byte + 4-byte schema ID), skip the prefix:\nfinal var confluentPipeline = registry.pipeline(format).skipBytes(5).add(lowerNameKey).build();\n```\n\n#### Confluent Schema Registry\n\nFor Confluent-style deployments where schemas live in a registry rather than on the classpath, the\n`kpipe-schema-registry-confluent` module ships an HTTP client and an in-process cache. Per-record auto-lookup is wired\ninto `AvroFormat`: the format reads the wire envelope (1-byte magic + 4-byte schema ID) off each record, resolves the\nwriter's schema, caches it by ID, and decodes the remaining bytes against it. This is the schema-evolution-correct path\n— every record decodes against its actual writer schema, so producer evolution doesn't silently corrupt consumer output\nthe way a static-fetch-at-startup pattern would.\n\nAdd the dependency:\n\n```kotlin\nimplementation(\"io.github.eschizoid:kpipe-schema-registry-confluent:1.12.0\")\n```\n\nWire the fluent facade with the SR-shorthand factory:\n\n```java\nimport java.time.Duration;\nimport org.kpipe.KPipe;\nimport org.kpipe.schemaregistry.confluent.CachedSchemaResolver;\nimport org.kpipe.schemaregistry.confluent.ConfluentSchemaResolver;\n\n// One resolver for the process; CachedSchemaResolver caches by ID with no TTL (Confluent SR IDs\n// are immutable, so cache-by-ID is trivially correct).\ntry (final var resolver = new CachedSchemaResolver(\n        new ConfluentSchemaResolver(\"http://schema-registry:8081\", Duration.ofSeconds(10)));\n     final var handle = KPipe.avro(\"orders\", kafkaProps, resolver)   // ← one-line SR consumer\n         .pipe(record -\u003e enrich(record))\n         .toCustom(WarehouseSink.create())\n         .start()) {\n  handle.awaitShutdown();\n}\n```\n\nThe factory above is equivalent to `KPipe.avro(AvroFormat.withRegistry(resolver), \"orders\", props)`. The format reads\nthe envelope per record. Do **not** combine `withSchemaRegistry(...)` with `skipBytes(5)` — the format already consumes\nthe envelope. The static-mode path is still supported for shops with strict append-only evolution who fetch the schema\nonce at startup:\n\n```java\ntry (final var resolver = new ConfluentSchemaResolver(\"http://schema-registry:8081\")) {\n  final var schemaJson = resolver.lookupBySubjectVersion(\"orders-value\", \"latest\");\n  final var format = AvroFormat.of(schemaJson);\n  // Pass `format` to KPipe.avro(topic, props, format) and pair with .skipBytes(5).\n}\n```\n\nThe explicit `lookupById(int)` and `lookupBySubjectVersion(subject, version)` calls remain available on\n`ConfluentSchemaResolver` for users who want to manage their own caching or fetch a known schema at startup.\n\n**Cache semantics.** `CachedSchemaResolver` uses a `ConcurrentHashMap` with no TTL and no LRU eviction. Confluent SR\nschema IDs are immutable — ID 42 today means the same schema tomorrow — so cache-by-ID is trivially correct and cache\ncardinality is naturally bounded (typically tens of distinct schemas across the lifetime of a topic, even with active\nevolution). The first record carrying a new schema ID costs one HTTP round trip; every subsequent record with that ID is\na hit. Hit / miss / size counters are exposed via the resolver's accessors and can be bound to OTel via\n`PipelineMetricsObserver.bindSchemaRegistryCache(...)` (see [Observability](#observability) below).\n\n**Scope.** Avro only for now. Protobuf SR auto-lookup needs runtime `.proto` text compilation and has not shipped — use\n`kpipe-format-protobuf` with a compiled descriptor and `skipBytes(6)` for the single-top-level-message case. No schema\npublishing (Confluent's own producer client handles that). No compatibility checks at the consumer — those run at\nregistration time inside SR.\n\n### Protobuf\n\nAdd `kpipe-format-protobuf`. Operators are `UnaryOperator\u003cMessage\u003e`. Protobuf messages are immutable, so every transform\nbuilds a new message via `toBuilder().setField(...).build()`.\n\n```java\nimport org.kpipe.format.protobuf.ProtobufFormat;\nimport org.kpipe.format.protobuf.ProtobufRegistryKey;\nimport org.kpipe.registry.MessageProcessorRegistry;\n\n// Build a ProtobufFormat bound to a single descriptor. Use ProtobufDescriptorCatalog for keyed\n// lookup of multiple descriptors.\nfinal var format = new ProtobufFormat(CustomerProto.Customer.getDescriptor());\nfinal var registry = new MessageProcessorRegistry(\"myApp\", format);\n\nfinal var clearEmailKey = ProtobufRegistryKey.of(\"clearEmail\");\nregistry.registerOperator(clearEmailKey, msg -\u003e {\n  final var emailField = msg.getDescriptorForType().findFieldByName(\"email\");\n  return msg.toBuilder().clearField(emailField).build();\n});\n\n// Register the protobuf console sink yourself (defaults are no longer auto-registered)\nfinal var protoLoggingKey = ProtobufRegistryKey.of(\"protobufLogging\");\nregistry.registerSink(protoLoggingKey, new org.kpipe.format.protobuf.ProtobufConsoleSink\u003c\u003e());\n\nfinal var pipeline = registry\n  .pipeline(format)\n  .add(clearEmailKey)\n  .toSink(protoLoggingKey)\n  .build();\n```\n\n---\n\n## Message sinks\n\nSinks are where processed messages go. `MessageSink` is just a functional `Consumer\u003cT\u003e`:\n\n```java\n@FunctionalInterface\npublic interface MessageSink\u003cT\u003e extends Consumer\u003cT\u003e {}\n```\n\n### Built-in sinks\n\nConsole sinks live in their format modules. The registry doesn't auto-register them — register the ones you want:\n\n```java\nimport org.kpipe.format.json.JsonConsoleSink;\nimport org.kpipe.format.avro.AvroConsoleSink;\nimport org.kpipe.format.protobuf.ProtobufConsoleSink;\n\n// Direct instantiation\nfinal var jsonConsoleSink = new JsonConsoleSink\u003cMap\u003cString, Object\u003e\u003e();\nfinal var avroConsoleSink = new AvroConsoleSink\u003cGenericRecord\u003e(schema);\nfinal var protobufConsoleSink = new ProtobufConsoleSink\u003cMessage\u003e();\n\n// Register and use via the pipeline builder\nregistry.registerSink(JsonFormat.JSON_LOGGING, jsonConsoleSink);\n\nfinal var pipeline = registry\n  .pipeline(JsonFormat.INSTANCE)\n  .add(RegistryKey.json(\"sanitize\"))\n  .toSink(JsonFormat.JSON_LOGGING)\n  .build();\n```\n\n### Custom sinks\n\n```java\nfinal MessageSink\u003cMap\u003cString, Object\u003e\u003e databaseSink = (processedMap) -\u003e {\n  databaseService.insert(processedMap);\n};\n```\n\n### Registering sinks\n\n`MessageProcessorRegistry` holds operators and sinks in two namespaces under the same key shape.\n`registerOperator(key, op)` and `registerSink(key, sink)` are the entry points; lookups use `getOperator(key)` and\n`getSink(key)`. Per-namespace utilities (`getAllSinks`, `getSinkMetrics`, `unregisterSink`, `clearSinks`,\n`compositeSink`) keep the surfaces separate without forcing users through a sub-object.\n\n```java\nfinal var registry = new MessageProcessorRegistry();\n\n// Register a sink under a typed key\nfinal var dbKey = RegistryKey.of(\"database\", Map.class);\nregistry.registerSink(dbKey, databaseSink);\n\n// Use it in a pipeline\nfinal var pipeline = registry.pipeline(JsonFormat.INSTANCE).add(RegistryKey.json(\"enrich\")).toSink(dbKey).build();\n```\n\n### Error handling in sinks\n\n```java\n// Wrap a sink or operator with error handling (suppresses exceptions, logs errors)\nfinal var safeSink = MessageProcessorRegistry.withSinkErrorHandling(riskySink);\nfinal var safeOperator = MessageProcessorRegistry.withOperatorErrorHandling(riskyOperator);\n\nregistry.registerSink(RegistryKey.json(\"safeDatabase\"), safeSink);\n```\n\n### Kafka producer sink\n\nTo produce processed messages back to a Kafka topic, use `KafkaMessageSink` (in `org.kpipe.producer.sink`):\n\n```java\nimport org.kpipe.producer.KPipeProducer;\nimport org.kpipe.producer.sink.KafkaMessageSink;\n\nfinal var producer = KPipeProducer.\u003cbyte[], byte[]\u003ebuilder().withProperties(kafkaProps).build();\n\nfinal var pipeline = registry\n  .pipeline(JsonFormat.INSTANCE)\n  .add(RegistryKey.json(\"transform\"))\n  .toSink(KafkaMessageSink.of(producer.getProducer(), \"output-topic\", JsonFormat.INSTANCE::serialize))\n  .build();\n```\n\n### Composite sink (fanout)\n\n`CompositeMessageSink` (in `org.kpipe.sink`, part of `kpipe-core`) broadcasts to multiple sinks. A failure in one sink\ndoesn't stop the others from receiving the message:\n\n```java\nimport org.kpipe.sink.CompositeMessageSink;\n\nfinal var compositeSink = new CompositeMessageSink\u003c\u003e(List.of(postgresSink, consoleSink));\n\nfinal var pipeline = registry.pipeline(JsonFormat.INSTANCE).toSink(compositeSink).build();\n```\n\n### Batch sinks\n\nSingle-record sinks pay the destination's per-call cost on every message. When that cost is non-trivial — a JDBC commit,\nan HTTP POST, an S3 PUT — batching amortizes it. `BatchSink\u003cT\u003e` is a `Function\u003cList\u003cT\u003e, BatchResult\u003e` that flushes at a\nconfigurable size or age:\n\n```java\nimport java.time.Duration;\nimport org.kpipe.KPipe;\nimport org.kpipe.sink.BatchPolicy;\nimport org.kpipe.sink.BatchSink;\n\nKPipe.json(\"events\", kafkaProps)\n    .pipe(addTimestamp)\n    .toBatch(\n        BatchSink.ofVoid(batch -\u003e jdbc.bulkInsert(batch)),     // void-style: success on return, fail on throw\n        new BatchPolicy(100, Duration.ofSeconds(5)))           // flush at 100 records OR 5 seconds, whichever first\n    .start();\n```\n\n`BatchSink.ofVoid(...)` wraps a void-style consumer (throw → whole-batch DLQ). For per-record outcomes — say a bulk HTTP\nAPI that returns which rows failed — return `BatchResult` directly with the succeeded / failed indexes; the wrapper\nroutes only the failures to the DLQ.\n\nHighlights:\n\n- **Both modes.** Sequential and parallel processing both work; in parallel mode the buffer participates in the\n  in-flight backpressure metric so a slow batch sink can't let the buffer grow unbounded.\n- **Multi-topic.** Compose with `KPipe.multi(...)` — each route can choose `.toBatch(...)` independently.\n- **Coverage contract enforced.** A `BatchResult` that doesn't cover every position `[0, batchSize)` is treated as a\n  contract violation and routed to the DLQ rather than silently marked processed.\n- **Shutdown drain.** A final flush runs before the offset manager closes, so partially-filled buffers commit cleanly.\n\nHeadline number from `BatchSinkLatencyBenchmark` at 1ms-per-call sink latency: **84× throughput at batch=100** versus\nthe per-record control. Full numbers in [`benchmarks/README.md`](benchmarks/README.md).\n\n---\n\n## Built-in metrics\n\n### Programmatic access\n\n```java\nfinal var metrics = consumer.getMetrics();\nlog.log(Level.INFO, \"Messages received: \" + metrics.get(\"messagesReceived\"));\nlog.log(Level.INFO, \"Successfully processed: \" + metrics.get(\"messagesProcessed\"));\nlog.log(Level.INFO, \"Processing errors: \" + metrics.get(\"processingErrors\"));\nlog.log(Level.INFO, \"Messages in-flight: \" + metrics.get(\"inFlight\"));\n// Backpressure metrics (present only when withBackpressure() is configured)\nlog.log(Level.INFO, \"Backpressure pauses: \" + metrics.get(\"backpressurePauseCount\"));\nlog.log(Level.INFO, \"Time spent paused (ms): \" + metrics.get(\"backpressureTimeMs\"));\n```\n\n### OpenTelemetry\n\nOTel is opt-in via the `kpipe-metrics-otel` module. `kpipe-metrics` ships interfaces only and doesn't pull\n`opentelemetry-api` onto your classpath. Add `kpipe-metrics-otel` (plus your OTel SDK at runtime) to wire real\ntelemetry:\n\n```java\nimport org.kpipe.metrics.otel.OtelConsumerMetrics;\n\nfinal var consumer = KPipeConsumer.\u003cbyte[]\u003ebuilder()\n  .withProperties(kafkaProps)\n  .withTopic(\"events\")\n  .withPipeline(pipeline)\n  .withMetrics(new OtelConsumerMetrics(openTelemetry, \"my-pipeline\"))\n  .build();\n```\n\nWhen `withMetrics(...)` is omitted, `ConsumerMetrics.noop()` / `ProducerMetrics.noop()` is used. Zero allocation, no\nOTel API on the classpath.\n\n| Instrument                                     | Type      | Description                                        |\n| ---------------------------------------------- | --------- | -------------------------------------------------- |\n| `kpipe.consumer.messages.received`             | Counter   | Records polled from Kafka                          |\n| `kpipe.consumer.messages.processed`            | Counter   | Records successfully processed                     |\n| `kpipe.consumer.messages.errors`               | Counter   | Records that failed processing                     |\n| `kpipe.consumer.processing.duration`           | Histogram | Per-record processing time (ms)                    |\n| `kpipe.consumer.messages.inflight`             | Gauge     | Current number of in-flight messages               |\n| `kpipe.consumer.backpressure.pauses`           | Counter   | Times backpressure paused the consumer             |\n| `kpipe.consumer.backpressure.time`             | Counter   | Total time paused due to backpressure              |\n| `kpipe.consumer.circuit_breaker.trips`         | Counter   | Times the breaker tripped CLOSED → OPEN            |\n| `kpipe.consumer.circuit_breaker.state_changes` | Counter   | Any CB state transition (tagged with target state) |\n| `kpipe.consumer.circuit_breaker.time_open`     | Counter   | Total time the breaker spent in OPEN (ms)          |\n| `kpipe.producer.messages.sent`                 | Counter   | Records successfully produced                      |\n| `kpipe.producer.messages.failed`               | Counter   | Records that failed to produce                     |\n| `kpipe.producer.dlq.sent`                      | Counter   | Records sent to DLQ                                |\n\n---\n\n## Consumer lifecycle\n\nSince 1.13.0 the consumer hosts its own lifecycle — start, periodic metrics reporting, JVM shutdown hook, in-flight\ndrain, graceful shutdown. The standalone `KPipeRunner` was deleted in the runner+tracker fold; everything is on\n`KPipeConsumer` directly.\n\n```java\nfinal var consumer = KPipeConsumer.\u003cbyte[]\u003ebuilder()\n  .withProperties(kafkaProps)\n  .withTopic(\"events\")\n  .withPipeline(pipeline)\n  .withMetricsReporters(List.of(\n    ConsumerMetricsReporter.forConsumer(c -\u003e consumer.getMetrics()),\n    EntryMetricsReporter.forProcessors(processorRegistry)\n  ))\n  .withMetricsInterval(Duration.ofSeconds(30))\n  .withShutdownHook(true)   // installs Runtime.getRuntime().addShutdownHook(consumer::close)\n  .build();\n\nconsumer.start();\nconsumer.awaitShutdown();             // blocks until close() completes\n```\n\nUse try-with-resources for explicit cleanup:\n\n```java\ntry (final var consumer = KPipeConsumer.\u003cbyte[]\u003ebuilder()\n    .withProperties(kafkaProps)\n    .withTopic(\"events\")\n    .withPipeline(pipeline)\n    .build()) {\n  consumer.start();\n  consumer.awaitShutdown(Duration.ofMinutes(5));   // bounded wait\n}\n```\n\nTargeted operations: `consumer.shutdownGracefully(Duration)` initiates close with a custom in-flight drain budget and\nreturns whether the drain finished cleanly; `consumer.waitForInFlightDrain(Duration)` blocks until the in-flight counter\nhits zero (useful during reconfiguration without closing).\n\nThe facade (`KPipe.json(...).start()`) wraps this in a `Handle` so users of the fluent path never see these methods\ndirectly — the handle's `awaitShutdown` / `shutdownGracefully` / `close` delegate straight through.\n\n---\n\n## Full application example\n\nThe [`examples/`](examples/) directory has complete working apps. Below is a condensed sketch.\n\n\u003cdetails\u003e\n\u003csummary\u003eExpand condensed application example\u003c/summary\u003e\n\n```java\npublic class KPipeApp implements AutoCloseable {\n\n  private static final System.Logger LOGGER = System.getLogger(KPipeApp.class.getName());\n  private final KPipeConsumer\u003cbyte[]\u003e consumer;\n\n  static void main() {\n    final var config = AppConfig.fromEnv();\n    try (final var app = new KPipeApp(config)) {\n      app.start();\n      app.awaitShutdown();\n    } catch (final Exception e) {\n      LOGGER.log(Level.ERROR, \"Fatal error in application\", e);\n      System.exit(1);\n    }\n  }\n\n  public KPipeApp(final AppConfig config) {\n    final var processorRegistry = new MessageProcessorRegistry(JsonFormat.INSTANCE);\n    processorRegistry.register(JsonFormat.JSON_LOGGING, new JsonConsoleSink\u003c\u003e());\n\n    final var commandQueue = new ConcurrentLinkedQueue\u003cConsumerCommand\u003e();\n\n    consumer = KPipeConsumer.\u003cbyte[]\u003ebuilder()\n      .withProperties(KafkaConsumerConfig.createConsumerConfig(config.bootstrapServers(), config.consumerGroup()))\n      .withTopic(config.topic())\n      .withDeadLetterTopic(config.topic() + \".dlq\")\n      .withPipeline(\n        processorRegistry\n          .pipeline(JsonFormat.INSTANCE)\n          .add(RegistryKey.json(\"addSource\"), RegistryKey.json(\"markProcessed\"), RegistryKey.json(\"addTimestamp\"))\n          .toSink(JsonFormat.JSON_LOGGING)\n          .build()\n      )\n      .withCommandQueue(commandQueue)\n      .withOffsetManagerProvider((c) -\u003e\n        KafkaOffsetManager.builder(c).withCommandQueue(commandQueue).withCommitInterval(Duration.ofSeconds(30)).build()\n      )\n      .withMetricsInterval(config.metricsInterval())\n      .withShutdownHook(true)\n      .build();\n  }\n\n  public void start() {\n    consumer.start();\n  }\n\n  public boolean awaitShutdown() {\n    return consumer.awaitShutdown();\n  }\n\n  public void close() {\n    consumer.close();\n  }\n}\n```\n\n**Environment variables:**\n\n```bash\nexport KAFKA_BOOTSTRAP_SERVERS=localhost:9092\nexport KAFKA_CONSUMER_GROUP=my-group\nexport KAFKA_TOPIC=json-events\nexport PROCESSOR_PIPELINE=addSource,markProcessed,addTimestamp\nexport METRICS_INTERVAL_SEC=30\nexport SHUTDOWN_TIMEOUT_SEC=5\n```\n\n\u003c/details\u003e\n\n---\n\n## Requirements\n\n- Java 25 or newer\n- Gradle (for building from source)\n- [kcat](https://github.com/edenhill/kcat) for ad-hoc testing\n- Docker for local Kafka via Testcontainers\n\n---\n\n## Testing\n\nThere's a `docker-compose.yaml` for spinning up Kafka, Zookeeper, and Confluent Schema Registry locally.\n\n```bash\n# Format code and build all modules\n./gradlew clean spotlessApply build\n\n# Build the consumer app container and start all services\ndocker compose build --no-cache --build-arg APP=\u003cjson|avro|protobuf|demo\u003e\ndocker compose down -v\ndocker compose up -d\n\n# Publish test messages\nfor i in {1..10}; do echo \"{\\\"id\\\":$i,\\\"message\\\":\\\"Test message $i\\\"}\" | \\\n  kcat -P -b kafka:9092 -t json-topic; done\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eWorking with the Schema Registry and Avro\u003c/summary\u003e\n\n```bash\n# Register an Avro schema\ncurl -X POST \\\n  -H \"Content-Type: application/vnd.schemaregistry.v1+json\" \\\n  --data \"{\\\"schema\\\": $(cat lib/kpipe-consumer/src/test/resources/avro/customer.avsc | jq tostring)}\" \\\n  http://localhost:8081/subjects/com.kpipe.customer/versions\n\n# Read registered schema\ncurl -s http://localhost:8081/subjects/com.kpipe.customer/versions/latest | jq -r '.schema' | jq --indent 2 '.'\n\n# Produce an Avro message using kafka-avro-console-producer\ncat \u003c\u003c'JSON' | docker run -i --rm --network kpipe_default \\\n-v \"$PWD/lib/kpipe-consumer/src/test/resources/avro/customer.avsc:/tmp/customer.avsc:ro\" \\\nconfluentinc/cp-schema-registry:8.2.0 \\\nsh -ec 'kafka-avro-console-producer \\\n  --bootstrap-server kafka:9092 \\\n  --topic avro-topic \\\n  --property schema.registry.url=http://schema-registry:8081 \\\n  --property value.schema=\"$(cat /tmp/customer.avsc)\"'\n{\"id\":1,\"name\":\"Mariano Gonzalez\",\"email\":{\"string\":\"mariano@example.com\"},\"active\":true,\"registrationDate\":1635724800000,\"address\":{\"com.kpipe.customer.Address\":{\"street\":\"123 Main St\",\"city\":\"Chicago\",\"zipCode\":\"00000\",\"country\":\"USA\"}},\"tags\":[\"premium\",\"verified\"],\"preferences\":{\"notifications\":\"email\"}}\nJSON\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eWorking with the Schema Registry and Protobuf\u003c/summary\u003e\n\n```bash\n# Register a Protobuf schema\ncurl -X POST \\\n  -H \"Content-Type: application/vnd.schemaregistry.v1+json\" \\\n  --data \"{\\\"schemaType\\\": \\\"PROTOBUF\\\", \\\"schema\\\": $(cat lib/kpipe-consumer/src/test/resources/protobuf/customer.proto | jq -Rs .)}\" \\\n  http://localhost:8081/subjects/com.kpipe.customer-protobuf/versions\n\n# Read registered schema\ncurl -s http://localhost:8081/subjects/com.kpipe.customer-protobuf/versions/latest | jq '.'\n\n# Produce a Protobuf message using kafka-protobuf-console-producer\ncat \u003c\u003c'JSON' | docker run -i --rm --network kpipe_default \\\nconfluentinc/cp-schema-registry:8.2.0 \\\nsh -ec 'kafka-protobuf-console-producer \\\n  --bootstrap-server kafka:9092 \\\n  --topic protobuf-topic \\\n  --property schema.registry.url=http://schema-registry:8081 \\\n  --property value.schema.id=1'\n{\"id\":\"1\",\"name\":\"Mariano Gonzalez\",\"email\":\"mariano@example.com\",\"active\":true,\"registrationDate\":\"1635724800000\",\"tags\":[\"premium\",\"verified\"],\"preferences\":{\"notifications\":\"email\"}}\nJSON\n```\n\n\u003c/details\u003e\n\n---\n\n## Advanced patterns\n\n### Enum-based registry\n\nFor statically typed bulk registration, define operators as an enum implementing `UnaryOperator\u003cT\u003e`:\n\n```java\npublic enum StandardProcessors implements UnaryOperator\u003cMap\u003cString, Object\u003e\u003e {\n  TIMESTAMP(JsonMessageProcessor.addTimestampOperator(\"ts\")),\n  SOURCE(JsonMessageProcessor.addFieldOperator(\"src\", \"app\"));\n\n  private final UnaryOperator\u003cMap\u003cString, Object\u003e\u003e op;\n\n  StandardProcessors(final UnaryOperator\u003cMap\u003cString, Object\u003e\u003e op) {\n    this.op = op;\n  }\n\n  @Override\n  public Map\u003cString, Object\u003e apply(final Map\u003cString, Object\u003e t) {\n    return op.apply(t);\n  }\n}\n\n// Bulk register all enum constants\nregistry.registerEnum(Map.class, StandardProcessors.class);\n\n// Now they can be used by name in configuration\n// PROCESSOR_PIPELINE=TIMESTAMP,SOURCE\n```\n\n### Conditional processing\n\n```java\nfinal var pipeline = registry\n  .pipeline(JsonFormat.INSTANCE)\n  .when(\n    (map) -\u003e \"VIP\".equals(map.get(\"level\")),\n    (map) -\u003e {\n      map.put(\"priority\", \"high\");\n      return map;\n    },\n    (map) -\u003e {\n      map.put(\"priority\", \"low\");\n      return map;\n    }\n  )\n  .build();\n```\n\n### Filtering messages\n\nReturn `null` from an operator to skip a record. KPipe stops processing it — no downstream operators, no sink. The\noffset is still committed (the record is treated as intentionally filtered, not failed).\n\n```java\nregistry.registerOperator(RegistryKey.json(\"filter\"), map -\u003e {\n  if (\"internal\".equals(map.get(\"type\"))) return null; // Skip this message\n  return map;\n});\n```\n\n### Thread safety and resource management\n\n- Processors should be stateless and thread-safe.\n- Don't add `ThreadLocal` of your own — under thread-per-record it gets a new map for every message and doesn't cache\n  anything. If you need thread-local-like state in a future feature (tenant context, span propagation), reach for\n  `ScopedValue` instead.\n- Side-effectful processors (DB calls, HTTP) need to be safe under high concurrency. With virtual threads you can easily\n  have thousands in flight; pool sizing on connection pools matters more than thread count.\n\n---\n\n## Performance\n\nA few specifics worth calling out, since \"fast\" without context is meaningless. Performance always depends on workload\nshape (I/O vs CPU bound), partitioning, and message size — these are micro-level optimizations.\n\n- Zero-copy magic-byte handling. For Avro from Confluent Schema Registry, KPipe takes an `offset` parameter to skip\n  magic bytes and schema IDs without `Arrays.copyOfRange`.\n- DslJson for JSON parsing. Cuts parse overhead and GC pressure compared to Jackson on the hot path.\n\nThe latest parallel benchmark in [`benchmarks/README.md`](benchmarks/README.md) shows KPipe ahead of Confluent Parallel\nConsumer on throughput, with a higher allocation footprint. Scenario-specific — not a blanket claim.\n\n---\n\n## Key-level ordering (payments, balances, anything sequential)\n\nIf your domain requires per-entity ordering — Authorize before Capture, balance updates in sequence — Kafka already\nguarantees per-partition ordering. Use that:\n\n- Have your producer key by the entity (`transaction_id`, `customer_id`, etc). Kafka routes all messages with the same\n  key to the same partition.\n- KPipe commits per-partition lowest-pending-offset, so as long as related events share a key, they land on one\n  partition and KPipe processes them in order without skipping.\n- For strict per-partition serial processing (one record at a time), turn on `.withSequentialProcessing(true)`.\n\n---\n\n## Observability\n\nTwo OTel pieces ship in `kpipe-metrics-otel`:\n\n- **`OtelConsumerMetrics`** — implements `ConsumerMetrics`. Wire on the consumer builder / `Stream.withMetrics(...)` and\n  get the standard `kpipe.consumer.*` counters and histograms (received, processed, errors, processing duration,\n  in-flight gauge, backpressure pauses, circuit-breaker state changes).\n- **`PipelineMetricsObserver`** — implements `Consumer\u003cResult\u003c?\u003e\u003e`. Hand to `Stream.peekResult(observer)` and every\n  `Passed` / `Filtered` / `Failed` outcome increments the matching `kpipe.pipeline.*` counter. This is how you make the\n  §12 \"processed counter rises but sink stays at 0\" condition visible at the metrics layer instead of having to scrape\n  logs.\n\n```java\nimport io.opentelemetry.api.GlobalOpenTelemetry;\nimport org.kpipe.metrics.otel.OtelConsumerMetrics;\nimport org.kpipe.metrics.otel.PipelineMetricsObserver;\n\nfinal var otel = GlobalOpenTelemetry.get();\nfinal var consumerMetrics = new OtelConsumerMetrics(otel, \"orders-consumer\");\nfinal var observer = new PipelineMetricsObserver(otel, \"orders\");\n\ntry (var handle = KPipe.json(\"orders\", kafkaProps)\n    .pipe(enrich)\n    .withMetrics(consumerMetrics)             // standard kpipe.consumer.* metrics\n    .peekResult(observer)                     // per-Result-variant counters\n    .toCustom(WarehouseSink.create())\n    .start()) {\n  handle.awaitShutdown();\n}\n```\n\nIf you're using Confluent SR auto-lookup, bind the cache counters too so cache hit rate is visible alongside the\npipeline outcomes:\n\n```java\nfinal var resolver = new CachedSchemaResolver(new ConfluentSchemaResolver(\"http://schema-registry:8081\"));\n\nfinal var observer = new PipelineMetricsObserver(otel, \"orders\").bindSchemaRegistryCache(\n  resolver::hitCount,\n  resolver::missCount,\n  () -\u003e (long) resolver.size()\n);\n```\n\nThe observer takes `LongSupplier`s rather than the resolver itself so `kpipe-metrics-otel` doesn't acquire a transitive\ndependency on `kpipe-schema-registry-confluent`. Wire whichever cache you actually use; the suppliers don't care.\n\n---\n\n## Metrics dashboard\n\nThere's a local observability stack under `infra/observability/` that runs via Docker Compose:\n\n- OpenTelemetry Collector — receives OTLP metrics from KPipe, exports to Prometheus.\n- Prometheus — scrapes the collector and stores time-series data.\n- Grafana — pre-provisioned with a \"KPipe Overview\" dashboard covering all consumer and producer metrics.\n\nAny of the example apps brings this stack up via `docker compose`. To point your own collector at a running KPipe app,\nset the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable:\n\n```bash\nOTEL_EXPORTER_OTLP_ENDPOINT=http://your-collector:4318\nOTEL_METRICS_EXPORTER=otlp\n```\n\nOpen Grafana at [http://localhost:3000](http://localhost:3000) (admin/admin) for the KPipe Overview dashboard.\n\n---\n\n## Running the demo\n\n`examples/demo` runs JSON, Avro, and Protobuf pipelines side-by-side in one app with observability wired up.\n\n```bash\n# Brings up Kafka, Schema Registry, OTel Collector, Prometheus, Grafana, the demo app, and seeds data\n./scripts/run-demo.sh\n\n# Or, by hand:\ncd examples/demo\ndocker compose up --build\n```\n\nThe script starts Kafka (KRaft mode) + Schema Registry, creates topics, registers the Avro and Protobuf schemas, brings\nup the observability stack, then builds and starts the demo app with all three pipelines and seeds JSON messages so\nthere's something to process immediately.\n\nGrafana is at [http://localhost:3000](http://localhost:3000); the app health endpoint is at\n[http://localhost:8080/health](http://localhost:8080/health).\n\n---\n\n## Inspiration\n\nKPipe leans on:\n\n- [Project Loom](https://openjdk.org/projects/loom/) for virtual threads.\n- [DslJson](https://github.com/ngs-doo/dsl-json) for the JSON hot path.\n\n---\n\n## Contributing\n\nCustom processors, metrics hooks, retry strategies — PRs welcome.\n\n---\n\n## License\n\nApache 2.0. See [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feschizoid%2Fkpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feschizoid%2Fkpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feschizoid%2Fkpipe/lists"}