{"id":16537790,"url":"https://github.com/multicatch/kafka-json-processor","last_synced_at":"2025-10-26T05:10:18.616Z","repository":{"id":65154692,"uuid":"526276383","full_name":"multicatch/kafka-json-processor","owner":"multicatch","description":"A processor that reads JSONs from Kafka topics, processes them and puts them in other selected Kafka topic.","archived":false,"fork":false,"pushed_at":"2023-02-06T12:29:08.000Z","size":1417,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-14T18:57:28.821Z","etag":null,"topics":["kafka","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/multicatch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-18T15:50:07.000Z","updated_at":"2023-01-03T12:17:15.000Z","dependencies_parsed_at":"2023-02-19T07:25:19.318Z","dependency_job_id":null,"html_url":"https://github.com/multicatch/kafka-json-processor","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/multicatch/kafka-json-processor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multicatch%2Fkafka-json-processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multicatch%2Fkafka-json-processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multicatch%2Fkafka-json-processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multicatch%2Fkafka-json-processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/multicatch","download_url":"https://codeload.github.com/multicatch/kafka-json-processor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multicatch%2Fkafka-json-processor/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264952074,"owners_count":23688010,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","rust"],"created_at":"2024-10-11T18:43:28.069Z","updated_at":"2025-10-26T05:10:13.578Z","avatar_url":"https://github.com/multicatch.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka JSON Processor\n\nA processor that reads JSONs from Kafka topics, processes them and puts them in other selected Kafka topic.\n\n```text\nKafka topic #1 -\u003e JSON message -\u003e kafka-json-processor -\u003e new JSON message -\u003e Kafka topic #2\n```\n\nProcessors used by kafka-json-processor are configured by a template.yaml file. \nYou can customize the process of processing messages or extend processors with custom ones.\n\nIn fact this utility generates Rust code based on a template in YAML. \nCompiled code in customized cases is generally faster, \nas it does not need to interpret the config and runs instructions directly.\n\nThis project is split into the following subprojects:\n* [The generator](kjp-generator) - generates a project based on `template.yaml` and available processor generators.\n* [The processor generators](kjp-generator-generators) - a set of scripts with code generators with some predefined functions for your custom processor.\n* [The plugin framework](kjp-generator-plugin) - base for your custom code generator (if you want to write it in Rust, not as a script).\n* [The core dependency](kafka-json-processor-core) - used in generated projects, prevents boilerplate.\n\n## How to use?\n\nIn short, the steps to run your custom processor are the following:\n\n1. Prepare your `template.yaml` with your desired processor configuration (e.g. copy field, extract date from message etc. - [see example](template-examples/basic.yaml)).\n2. Generate JSON processor with [the generator](kjp-generator) (and [processor generators](kjp-generator-generators), you can also use your own) and compile the generated project.\n3. Prepare `processor.properties` with rdkafka (Kafka client) configuration - [see example](./processor.properties) (put this file in the same directory as your executable).\n4. Run your executable (to see logs set the following environment variable: `RUST_LOG=info`, e.g. in bash you can just run `RUST_LOG=info ./your_executable`).\n\n## Test your processor\n\nYou may want to test if the generated processor is correct before deploying it.\nTo test it, in the generated project, create a `simulations` directory. \nIn this directory, create another directory (or directories, depending on the `template.yaml`) with the name `${input_topic}_${output_topic}`,\nwhere `${input_topic}` is the name of the input_topic from `template.yaml` and `${output_topic}` is the name of the output_topic from `template.yaml`.\n\nFor example, if you have the following stream in your `tempate.yaml`:\n```yaml\nstreams:\n  - input_topic: sometopic\n    output_topic: target\n```\n\nThen create the following directory structure:\n```text\n\u003cproject_directory\u003e\n \u003e simulations\n | \u003e sometopic_target\n```\n\nIn the `${input_topic}_${output_topic}` directory (in this case - `sometopic_target`), create text files with the input message and expected output.\nFor example, given the template [`all_processors.yaml`](template-examples/all_processors.yaml) (see template-examples), I have prepared some test data in [`simulations/in_out`](simulations/in_out).\nThe test files always have two headers - `[Input]` (for input JSON) and `[Expected]` (for expected processed message).\n\nTo run the simulation, run `cargo test` in the generated project. \nSee [`kjp-generator/tests/integration_test.rs`](kjp-generator/tests/integration_test.rs) for a complete example.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmulticatch%2Fkafka-json-processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmulticatch%2Fkafka-json-processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmulticatch%2Fkafka-json-processor/lists"}