{"id":20515803,"url":"https://github.com/kuper-tech/protokaf","last_synced_at":"2025-04-10T09:52:31.604Z","repository":{"id":45382445,"uuid":"403666306","full_name":"Kuper-Tech/protokaf","owner":"Kuper-Tech","description":"Kafka producer and consumer tool in protobuf format.","archived":false,"fork":false,"pushed_at":"2024-08-01T10:10:38.000Z","size":76,"stargazers_count":148,"open_issues_count":4,"forks_count":7,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-24T08:47:45.858Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Kuper-Tech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-06T15:08:37.000Z","updated_at":"2025-02-05T09:37:44.000Z","dependencies_parsed_at":"2023-02-08T20:15:16.975Z","dependency_job_id":"a1551fa5-5a9a-404c-877d-e68512a1f33d","html_url":"https://github.com/Kuper-Tech/protokaf","commit_stats":null,"previous_names":["kuper-tech/protokaf","sbermarket-tech/protokaf"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kuper-Tech%2Fprotokaf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kuper-Tech%2Fprotokaf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kuper-Tech%2Fprotokaf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kuper-Tech%2Fprotokaf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Kuper-Tech","download_url":"https://codeload.github.com/Kuper-Tech/protokaf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248198172,"owners_count":21063626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T21:24:47.206Z","updated_at":"2025-04-10T09:52:31.580Z","avatar_url":"https://github.com/Kuper-Tech.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# protokaf\n\nKafka producer and consumer tool in protobuf format.\n\n## Features\n- Consume and produce messages using Protobuf protocol\n- Trace messages with Jaeger\n- Create custom templates for one or multiple messages and produce them to Kafka\n\n## Install\n```sh\ngo install github.com/kuper-tech/protokaf@latest\n```\n\n## Configuration\nConfiguration file is optional, so you can skip this section.\n\nIn order for Protokaf to work, it needs to know how to reach your Kafka broker. First option is to provide `--broker` each time you invoke Protokaf. Another option is to use a configuration file. You can provide Protokaf with a configuration file with option `-F ..` on the command line. Or by default Protokaf will search its config files in `.protokaf.yaml` and `$HOME/.protokaf.yaml` respectively.\n\n**Example of `.protokaf.yaml`**\n```yaml\ndebug: true\nbroker: \"\u003caddr\u003e:\u003cport\u003e\"\nkafka-auth-dsn: \"SCRAM-SHA-512:\u003cnamespace\u003e:\u003cpasswd\u003e\"\nproto: \"\u003cdir\u003e/\u003cprotofile\u003e\"\n```\n\n## Help\n```sh\n$ protokaf help\n```\n\n## List metadata\n```sh\n$ protokaf list [-t \u003ctopic\u003e(,\u003ctopic\u003e...)]\n1 brokers:\n broker 1 \"127.0.0.1:9093\"\n2 topics:\n  topic \"test-topic\", partitions: 1\n    partition 0, leader 1, replicas: [1] (offline: []), isrs: [1]\n  topic \"test\", partitions: 1\n    partition 0, leader 1, replicas: [1] (offline: []), isrs: [1]\n```\n\n## Produce\n### Help\n```sh\n$ protokaf produce -h\n```\n\n### Examples\nThis proto file will be used in the examples below. \n\n`api/example.proto`\n```protobuf\nsyntax = \"proto3\";\n\npackage example;\n\nmessage HelloRequest {\n  string name = 1;\n  int32 age = 2;\n}\n```\n\n**A simple produce message**\n```sh\n$ protokaf produce HelloRequest \\\n    --broker kafka:9092 \\\n    --proto internal/proto/testdata/example.proto \\\n    --topic test \\\n    --data '{\"name\": \"Alice\", \"age\": 11}'\n```\n\n**Produce message with headers**\n```sh\n$ protokaf produce HelloRequest \\\n    --broker kafka:9092 \\\n    --proto internal/proto/testdata/example.proto \\\n    --topic test \\\n    --header \"priority=high\" \\\n    --header \"application=protokaf\" \\\n    --data '{\"name\": \"Alice\", \"age\": 11}'\n```\n\n**Produce message with \u003ca href=\"#template\"\u003etemplate\u003c/a\u003e**\n```sh\n$ protokaf produce HelloRequest \\\n    --broker kafka:9092 \\\n    --proto internal/proto/testdata/example.proto \\\n    --topic test \\\n    --data '{\"name\": {{randomFemaleName | quote}}, \"age\": {{randomNumber 10 20}}}' \\\n    --count 10 \\\n    --seed 42\n```\n\n**Produce message with Kafka auth**\n```sh\n$ protokaf produce HelloRequest \\\n    --broker kafka:9093 \\\n    --kafka-auth-dsn \"SCRAM-SHA-512:login:passwd\" \\\n    --proto internal/proto/testdata/example.proto \\\n    --topic test \\\n    --data '{\"name\": \"Alice\", \"age\": 11}'\n```\n\n**Read data from stdin or flag**\n\nRead message `HelloRequest` from `stdin`, produce to `test` topic\n```sh\n$ echo '{\"name\": \"Alice\", \"age\": 11}' | protokaf produce HelloRequest -t test\n```\n\nRead message `HelloRequest` from `-d` value, produce to `test` topic\n```sh\n$ protokaf produce HelloRequest -t test -d '{\"name\": \"Alice\", \"age\": 11}'\n```\n\n### Template\u003ca id=\"template\"\u003e\u003c/a\u003e\n**Template options**\n* `--seed \u003cint\u003e` You can set number greater then zero to produce the same pseudo-random sequence of messages\n* `--count \u003cint\u003e` Useful for generating messages with random data\n* `--concurrency \u003cint\u003e` Number of message senders to run concurrently for const concurrency producing\n\n**Show all template functions**\n```sh\n$ protokaf produce --template-functions-print\n```\n\n## Build json template by proto file\nThis can be useful for creating body for produce command\n```sh\n$ protokaf build HelloRequest --proto internal/proto/testdata/example.proto\n```\nFor proto file\n```protobuf\nsyntax = \"proto3\";\n\npackage example;\n\nmessage HelloRequest {\n  enum Status {\n    PENDING = 0;\n    COMPLETED = 1;\n  }\n\n  string name = 1;\n  int32 age = 2;\n  optional float amount = 4;\n  Status status = 5;\n  repeated string keys = 6;\n}\n```\ncommand will print\n```json\n{\n  \"name\": \"\",\n  \"age\": 0,\n  \"amount\": 0,\n  \"status\": \"PENDING\",\n  \"keys\": [\n    \"\"\n  ]\n}\n\n```\n\n## Consume\n### Help\n```sh\n$ protokaf help consume\n```\n\n### Examples\n```sh\n$ protokaf consume HelloRequest \\\n    --broker kafka:9092 \\\n    --proto internal/proto/testdata/example.proto \\\n    --group mygroup \\\n    --topic test\n```\n\n**Read messages from Kafka `test` topic, use group `mygroup`, print to `stdout`**\n```sh\n$ protokaf consume HelloRequest -G mygroup -t test\n```\n\n**Read the last `10` messages from `test` topic, then exit**\n```sh\n$ protokaf consume HelloRequest -G mygroup -t test -c 10\n```\n\n**Read from offset `5` messages from `test` topic**\n```sh\n$ protokaf consume HelloRequest -G mygroup -t test -o 5\n```\n\n## Testing\n\n### Prepare test environment and running tests\n```sh\nmake docker-dev-up\nmake kafka-users\nmake test\nmake install # optional (you can use 'go run . \u003cargs\u003e \u003cflags\u003e')\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuper-tech%2Fprotokaf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkuper-tech%2Fprotokaf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuper-tech%2Fprotokaf/lists"}