{"id":20377867,"url":"https://github.com/konqi/schematic-kafka","last_synced_at":"2025-04-12T07:52:17.477Z","repository":{"id":37794457,"uuid":"365555081","full_name":"konqi/schematic-kafka","owner":"konqi","description":"Protocol agnostic typescript implementation to encode and decode kafka messages using the Confluent Schema Registry","archived":false,"fork":false,"pushed_at":"2025-04-03T18:07:09.000Z","size":688,"stargazers_count":10,"open_issues_count":5,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-12T07:52:05.315Z","etag":null,"topics":["avro","json","kafka","protobuf","schema","schema-registry","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/konqi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-08T15:58:14.000Z","updated_at":"2025-01-02T09:47:08.000Z","dependencies_parsed_at":"2025-01-02T10:32:37.196Z","dependency_job_id":"e234f00b-fbfd-4c62-b2c8-03539da0271e","html_url":"https://github.com/konqi/schematic-kafka","commit_stats":{"total_commits":176,"total_committers":17,"mean_commits":"10.352941176470589","dds":"0.49431818181818177","last_synced_commit":"8fb4edce54e93f301d5d75708fae7edc4360b80d"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konqi%2Fschematic-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konqi%2Fschematic-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konqi%2Fschematic-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konqi%2Fschematic-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/konqi","download_url":"https://codeload.github.com/konqi/schematic-kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248537033,"owners_count":21120690,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","json","kafka","protobuf","schema","schema-registry","typescript"],"created_at":"2024-11-15T01:46:50.494Z","updated_at":"2025-04-12T07:52:17.452Z","avatar_url":"https://github.com/konqi.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"![workflow-status](https://github.com/konqi/schematic-kafka/actions/workflows/build-actions.yml/badge.svg) ![test-status](https://github.com/konqi/schematic-kafka/actions/workflows/test-actions.yml/badge.svg) [![Coverage Status](https://coveralls.io/repos/github/konqi/schematic-kafka/badge.svg?branch=main)](https://coveralls.io/github/konqi/schematic-kafka?branch=main)\n\n# schematic-kafka\n\nThis package provides a **schema type agnostic** kafka message encoder/decoder. It works fine with whatever protocol you may want to use, but it doesn't take care of this aspect. Have a look at the code sample below to understand what this means.\n\n## Quickstart\n\n### Install\n\n```bash\nnpm install avsc schematic-kafka\n# or\nyarn add avsc schematic-kafka\n```\n\n### Use\n\n```typescript\nimport { KafkaRegistryHelper, SchemaType } from \"schematic-kafka\"\nimport { parse, Type as AVSCInstance } from \"avsc\"\n\n// create instance\nconst registry = new KafkaRegistryHelper({ baseUrl: \"https://schemaRegistryHost:8081\" })\n  // adding a custom schema handler for AVRO\n  // as mentioned before, the library doesn't take care of this\n  .withSchemaHandler(SchemaType.AVRO, (schema: string) =\u003e {\n    // if you want to customize your encoder, this is where you'd do it\n    const avsc: AVSCInstance = parse(schema)\n    return {\n      encode: (message: any) =\u003e {\n        return avsc.toBuffer(message)\n      },\n      decode: (message: Buffer) =\u003e {\n        return avsc.fromBuffer(message)\n      },\n    }\n  })\n\n// decode a message from kafka\n// AVSC returns parsed json, so decodedMessage is a ready to use object\nconst decodedMessage = await registry.decode(rawMessageFromKafka)\n\n// encode a message with a schema\n// where\n// - subject    is the kafka topic plus the (-key, -value) postfix\n// - message    the actual message to send (this has to be in whatever format\n//              the schema handler defined above expects in the encode-function)\n// - schemaType (optional if already registerd, default: AVRO) AVRO/PROTOBUF/JSON\n// - schema     (optional if already registerd) serialized schema to be used\n// returns      a Buffer that you can send to the kafka broker\nconst encodeResult = await registry.encodeForSubject(subject, message, SchemaType.AVRO, schema)\n```\n\nFor more examples, take a look at `src/kafka-registry-helper.testcontainersspec.ts`.\n\n## How this library works\n\nThis is how a kafka message looks like when you send or receive it.\n\n```\n[ 1 byte  | 0      | 0 indicates this message is schema encoded ]\n[ 4 bytes | number | schema id                                  ]\n[ n bytes | msg    | protocol encoded message                   ]\n```\n\nThe first byte being a zero tells us that the following four bytes contain the schema id. With this schema id we can request the schema type (AVRO, PROTOBUF or JSON) and schema (serialized representation of the schema for the corresponding schema type) from the schema registry.\n\nThis library can decodes whole kafka message header and then calls the appropriate decoder that you provide with the schema as argument.\n\n## Documentation\n\nThe package exports the following things\n\n- `KafkaRegistryHelper` is the wrapper that handles encoding/decoding of messages, it also registeres your payload schemas if the registry doesn't know about them yet\n- `SchemaRegistryClient` is the low level api client that communicates with the schema-registry\n- `SchemaType` Is an enum containing the available schema types (AVRO, PROTOBUF, JSON)\n\n`KafkaRegistryHelper` applies a cache for some of the schema registry requests of `SchemaRegistryClient` to go easy on the network traffic and load.\n\n### Client SSL authentication\n\nThis library uses node's http/https request. As such you can provide an Agent to modify your requests.\n\n```typescript\nimport { Agent } from \"https\"\n\nconst agent = new Agent({\n  key: readFileSync(\"./client.key\"),\n  cert: readFileSync(\"./client.cert\"),\n})\nnew KafkaRegistryHelper({ baseUrl: \"https://schemaRegistryHost:8081\", agent })\n...\n```\n\n### Basic authentication\n\n```typescript\nnew KafkaRegistryHelper({ baseUrl: \"https://schemaRegistryHost:8081\", username: \"username\", password: \"password })\n\n// OR\n\nnew KafkaRegistryHelper({ baseUrl: \"https://username:password@schemaRegistryHost:8081\" })\n```\n\n## feature x\n\nTODO - document things hint: Most methods have jsdoc comments on them. Have a look.\n\n# TODO\n\nA few things that need to be done:\n\n- ✅ increase test coverage to at least 85%\n- ✅ stretch goal: test coverage of at least 95%\n- ✅ Add missing jsdoc information\n- ✅ fix caching mechanism for non-primitive values\n- ✅ create working sample/testcontainers-test for PROTOBUF messages\n- ✅ create a working example for how to use this package with kafka.js\n- create example for how to configure nestjs kafka-module serializer/deserializer\n- create working example for use of referenced schemas (AVRO/PROTOBUF)\n\n# Motivation\n\nI created this library out of necessity. I wanted to use [confluent-schema-registry](https://github.com/kafkajs/confluent-schema-registry). However I wasn't able to get a client-certificate TLS-secured connection to a schema-registry working. All I got was an SSL error with code 42 in an underlying layer. I wasn't able to configure mappersmith, which handles http connections. I'm aware that a recent merge should fix that, but the head version from the repo didn't work for me either. I also didn't like the list dependencies 😉. I had a look at [avro-schema-registry](https://github.com/bencebalogh/avro-schema-registry) which can't use SSL authentication at all. Being a nice little library, I decided to fork and make the changes and, well, things got a little out of hand... Now everything is pure typescript and the package handles just the schema-registry aspect.\n\n# Dependencies\n\nThe module has just the tslib as dependency.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkonqi%2Fschematic-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkonqi%2Fschematic-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkonqi%2Fschematic-kafka/lists"}