{"id":13709689,"url":"https://github.com/twmb/kcl","last_synced_at":"2025-04-13T00:43:35.775Z","repository":{"id":35071180,"uuid":"180713970","full_name":"twmb/kcl","owner":"twmb","description":"Your one stop shop to do anything with Kafka. Producing, consuming, transacting, administrating; 0.8.0 through 3.2+","archived":false,"fork":false,"pushed_at":"2024-03-13T23:30:29.000Z","size":469,"stargazers_count":208,"open_issues_count":9,"forks_count":21,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-13T00:43:29.662Z","etag":null,"topics":["client","go","golang","kafka","kafka-client"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/twmb.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-11T04:26:06.000Z","updated_at":"2025-03-10T14:25:19.000Z","dependencies_parsed_at":"2024-06-18T18:31:06.384Z","dependency_job_id":null,"html_url":"https://github.com/twmb/kcl","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twmb%2Fkcl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twmb%2Fkcl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twmb%2Fkcl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twmb%2Fkcl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/twmb","download_url":"https://codeload.github.com/twmb/kcl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248650418,"owners_count":21139672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["client","go","golang","kafka","kafka-client"],"created_at":"2024-08-02T23:00:43.942Z","updated_at":"2025-04-13T00:43:35.753Z","avatar_url":"https://github.com/twmb.png","language":"Go","funding_links":[],"categories":["Repositories"],"sub_categories":[],"readme":"kcl\n===\n\n## Contents\n\n- [Introduction](#introduction)\n- [Getting Started / Installation](#getting-started)\n- [Stability Status](#stability-status)\n- [Configuration](#configuration)\n- [Autocompletion](#autocompletion)\n- [Transactions](#transactions)\n- [Group Consuming](#group-consuming)\n- [API at a Glance](#api-at-a-glance)\n- [Examples](#examples)\n\n## Introduction\n\nkcl is a complete, pure Go command line Kafka client. Think of it as your\none stop shop to do anything you want to do with Kafka. Producing, consuming,\ntransacting, administrating, and so on.\n\nUnlike the small size of [kafkacat][1], this binary is ~12M compiled.\nIt is, however, still fast, has rich consuming and producing formatting\noptions, and a complete Kafka administration interface.\n\n[1]: https://github.com/edenhill/kafkacat\n\n## Stability Status\n\nI consider the current API **relatively** stable. Once this hits a 1.x release,\nthe API will be even more stable. I would like to get some definitive\nbroader usage of the client before deeming things unchanging.\n\nI've spent a good amount of time integration testing my [franz-go][2] client\nthat this program uses. The main thing I have currently been unable to test is\nclosest replica fetching, which is only theoretically supported. It is worth it\nto read the stability status in the franz-go repo as well if using this client.\n\n[2]: https://github.com/twmb/franz-go/\n\nI would love confirmation that this program has been used more broadly, and\nwould love to start a \"Users\" section below. With this confirmation, I will\npush a 1.x release.\n\n## Getting Started\n\nIf you have a go installation, you can simply\n\n```\ngo install github.com/twmb/kcl@latest\n```\n\nThis will install kcl from the latest release. You can optionally suffix the\n`go get` with `@v#.#.#` to install a specific version.\n\nOtherwise, you can download a release from the\n[releases](https://github.com/twmb/kcl/releases) page.\n\n## Configuration\n\nkcl supports configuration through a config file, environment variables, and\nconfig flag overrides, with the config values being defined in that order.\nBy default, kcl searches your OS's user config dir for a `kcl` directory and\na `config.toml` file in that directory. The default path can be overridden.\nAs well, multiple configs can easily swapped between with `kcl myconfig`.\n\nThe configuration supports TLS, SASL (currently PLAIN and SCRAM), seed brokers,\nand a timeout for requests that take timeouts.\n\nTo learn more about configuration, use `kcl myconfig help`.\n\n## Autocompletion\n\nThanks to [cobra][2], autocompletion exists for bash, zsh, and powershell.\n\n[3]: https://github.com/spf13/cobra\n\nAs an example of what to put in your .bashrc,\n\n```bash\nif [ -f /etc/bash_completion ] \u0026\u0026 ! shopt -oq posix; then\n    . /etc/bash_completion\n    . \u003c(kcl misc gen-autocomplete -kbash)\nfi\n```\n\n## Transactions\n\nTransactions are supported by consuming a batch of records, executing a command\nand passing the records to the command's STDIN, reading the modified records\nvia the command's STDOUT, and publishing those records.\n\nInput to the program and output from the program is controlled through the same\nsyntax as consuming and producing, and the `--rw` flag is a shortcut to say that\nthe input and output will use the same format.\n\nAs an example, the following command:\n\n```\nkcl transact --rw '%V{b4}%v' -dtxn -g group -t foo -x mytxn -v ./command\n```\n\nreads topic `foo` in group `group`, executes `./command`, writes all record\nvalues to it prefixed with the four byte big endian value length, reads\nback records in the same format, and produces to topic `txn` all using the\ntransactional id `mytxn`.\n\nAdditionally, you can use the special command \"mirror\" to have poor man's\nmirrormarker:\n\n```\nkcl transact -x mytxn -g group -t srcTopic -d destTopic -v mirror\n```\n\n## Group Consuming\n\nGroup consuming is supported with the `-g` or `--group` flag to `kcl consume`\nor `kcl transact`. The default balancer is the cooperative-sticky balancer,\nwhich was introduced with incremental rebalancing in Kafka 2.4.0. This balancer\nis incompatible with the previous eager balancers (roundrobin, range, sticky),\nthus if you are using kcl with existing groups that have members using eager\nbalancing strategies, be sure to specify a different balancer.\n\n## API at a glance\n\nBe sure to `help` any command before using it to understand the full syntax.\n\n```\nkcl\n consume                            -- consume records\n\n produce                            -- produce records\n\n transact                           -- transactional consuming \u0026 producing\n\n metadata                           -- print broker, cluster, and topic information\n\n group\n   list                             -- list consumer groups\n   describe                         -- describe consumer groups\n   delete                           -- delete consumer groups\n   offset-delete                    -- forcefully delete committed offsets a group (see KIP-496)\n\n topic\n   create                         -- create topics\n   delete                         -- delete topics\n   add-partitions                 -- add partitions to topics\n\n misc\n   api-versions                     -- print api versions for requests\n   probe-version                    -- probe for the currently running Kafka version\n   gen-autocomplete                 -- generate cli autocompletion\n   errcode                          -- print the error name and desc for an error number\n   errtext                          -- print the error name and desc for an error code / all errors\n   raw-req                          -- issue a raw request from input JSON\n   list-offsets                     -- list offsets for topics and partitions\n\n admin\n   delete-records                   -- delete record deletion for partitions based off input offsets\n   elect-leaders                    -- trigger leader elections\n\n   acl\n     create                         -- create ACLs\n     describe                       -- describe ACLs\n     delete                         -- delete ACLs\n\n   client-quotas\n     alter                          -- alter client quotas\n     describe                       -- describe client quotas\n\n   configs\n     alter                          -- alter broker, topic, broker-logger, etc. configs\n     describe                       -- describe broker, topic, broker-logger, etc. configs\n\n   dtoken\n     create                         -- create delegation tokens\n     renew                          -- renew delegation tokens\n     describe                       -- describe delegation tokens\n     expire                         -- expire delegation tokens\n\n   group                            -- duplicate of `kcl group` top level command\n\n   logdirs\n     alter                          -- alter log directories that partitions are in\n     describe                       -- describe log directories that partitions are in\n\n   partas\n     alter                          -- alter partition assignments\n     list                           -- list partition reassignments\n\n   topic                            -- duplicate of `kcl topic` top level command\n\n   user-scram\n     alter                          -- alter user scram\n     describe                       -- describe user scram\n\n myconfig\n   unlink                           -- unlink the kcl config symlink\n   link                             -- link a kcl config symlink\n   dump                             -- dump the kcl configuration\n   help                             -- print kcl configuration help\n   ls                               -- list files in the kcl config directory\n```\n\n## Examples\n\n### Consuming\n\n#### ...from topic foo, printing the values of records\n\n```\nkcl consume foo\n```\n\n#### ...from topic foo with advanced printing\n\n```\nkcl consume foo -f \"KEY=%k, VALUE=%v, HEADERS=%{%h{ '%k'='%v' }}\\n\"\n```\n\n#### ...from topics foo and bar with group grup\n\n```\nkcl consume -g grup foo bar\n```\n\n### Producing\n\n#### ...a newline delimited value to topic foo\n\n```\necho fubar | kcl produce foo\n```\n\n#### ...a bunch of newline delimited values in file baz to topic foo\n\n```\nkcl produce foo \u003c baz\n```\n\n#### ...key bar, value foo to topic foo in an obscurely formatted way\n\n```\necho barfoo | kcl produce foo -f'%K{3}%V{3}%v%k\\n'\n```\n\n#### ...key bizzy, value bazzy to topic foo, pulling the key and value from a line\n\n```\necho \"key: bizzy, value: bazzy\" | kcl produce foo  -f 'key: %k, value: %v\\n'\n```\n\n#### ...key k, value v, headers k1 v1 and k2 v2 to topic foo, with ascii length prefixed strings\n\n```\necho \"1 k 1 v 2 2 h1 2 v1 2 h2 2 v2 \" | kcl produce foo -f '%K %k %V %v %H %h{%K %k %V %v }\\n'\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwmb%2Fkcl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftwmb%2Fkcl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwmb%2Fkcl/lists"}