{"id":17921798,"url":"https://github.com/3axap4ehko/kafka-console","last_synced_at":"2025-10-16T11:22:42.870Z","repository":{"id":57288047,"uuid":"247346699","full_name":"3axap4eHko/kafka-console","owner":"3axap4eHko","description":"CLI kafka client","archived":false,"fork":false,"pushed_at":"2025-06-06T00:34:17.000Z","size":1555,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-06T00:49:12.859Z","etag":null,"topics":["kafka","kafka-cli","kafka-client"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/3axap4eHko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-03-14T20:14:43.000Z","updated_at":"2025-06-06T00:34:19.000Z","dependencies_parsed_at":"2023-10-11T02:29:42.731Z","dependency_job_id":"4e87b8f0-4dda-4ad6-bd43-80bf84e464e8","html_url":"https://github.com/3axap4eHko/kafka-console","commit_stats":{"total_commits":228,"total_committers":3,"mean_commits":76.0,"dds":0.2192982456140351,"last_synced_commit":"e1dc0fb1bf4d42fa217b053f0ff07f00c85de4d4"},"previous_names":[],"tags_count":251,"template":false,"template_full_name":null,"purl":"pkg:github/3axap4eHko/kafka-console","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3axap4eHko%2Fkafka-console","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3axap4eHko%2Fkafka-console/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3axap4eHko%2Fkafka-console/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3axap4eHko%2Fkafka-console/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/3axap4eHko","download_url":"https://codeload.github.com/3axap4eHko/kafka-console/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3axap4eHko%2Fkafka-console/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259561735,"owners_count":22876874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-cli","kafka-client"],"created_at":"2024-10-28T20:36:12.336Z","updated_at":"2025-10-16T11:22:42.860Z","avatar_url":"https://github.com/3axap4eHko.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Console CLI\r\n\r\nA powerful and easy-to-use command-line interface for Apache Kafka operations.\r\n\r\n[![NPM version][npm-image]][npm-url]\r\n[![Downloads][downloads-image]][npm-url]\r\n\r\n## Table of Contents\r\n\r\n- [Features](#features)\r\n- [Installation](#installation)\r\n- [Quick Start](#quick-start)\r\n- [Commands](#commands)\r\n  - [Consuming Messages](#consuming-messages)\r\n  - [Producing Messages](#producing-messages)\r\n  - [Topic Management](#topic-management)\r\n  - [Cluster Information](#cluster-information)\r\n- [Authentication](#authentication)\r\n- [Message Formats](#message-formats)\r\n- [Environment Variables](#environment-variables)\r\n- [Common Use Cases](#common-use-cases)\r\n- [Troubleshooting](#troubleshooting)\r\n- [License](#license)\r\n\r\n## Features\r\n\r\n- ✅ **Consumer \u0026 Producer** - Full support for consuming and producing messages\r\n- ✅ **Multiple Authentication Methods** - Plain, SCRAM-SHA-256/512, AWS IAM, OAuth Bearer\r\n- ✅ **Flexible Message Formats** - JSON, JavaScript, raw text, or custom formatters\r\n- ✅ **Consumer Groups** - Full consumer group support with offset management\r\n- ✅ **Time-based Consumption** - Read messages from specific timestamps\r\n- ✅ **SSL/TLS Support** - Secure connections to Kafka clusters\r\n- ✅ **Topic Management** - Create, delete, and inspect topics\r\n- ✅ **Headers Support** - Read and write message headers\r\n- ✅ **GZIP Compression** - Automatic compression support\r\n- ✅ **TypeScript** - Full TypeScript support\r\n\r\n## Installation\r\n\r\n### Global Installation (Recommended)\r\n```bash\r\nnpm install -g kafka-console\r\n```\r\n\r\n### Local Installation\r\n```bash\r\nnpm install kafka-console\r\n```\r\n\r\n### Using without Installation\r\n```bash\r\nnpx kafka-console [command]\r\n```\r\n\r\n## Quick Start\r\n\r\n### 1. List all topics\r\n```bash\r\nkafka-console list --brokers localhost:9092\r\n```\r\n\r\n### 2. Consume messages from a topic\r\n```bash\r\nkafka-console consume my-topic --brokers localhost:9092\r\n```\r\n\r\n### 3. Produce a message to a topic\r\n```bash\r\necho '{\"message\": \"Hello Kafka!\"}' | kafka-console produce my-topic --brokers localhost:9092\r\n```\r\n\r\n## Commands\r\n\r\n### Consuming Messages\r\n\r\n```bash\r\nkafka-console consume \u003ctopic\u003e [options]\r\n```\r\n\r\n#### Options\r\n| Option | Description | Default |\r\n|--------|-------------|---------|\r\n| `-g, --group \u003cgroup\u003e` | Consumer group name | `kafka-console-consumer-{timestamp}` |\r\n| `-f, --from \u003cfrom\u003e` | Start position (timestamp/ISO date/0 for beginning) | latest |\r\n| `-c, --count \u003ccount\u003e` | Number of messages to read | unlimited |\r\n| `-s, --skip \u003cskip\u003e` | Number of messages to skip | 0 |\r\n| `-o, --output \u003cfile\u003e` | Write output to file | stdout |\r\n| `-d, --data-format \u003cformat\u003e` | Message format (json/js/raw/custom) | json |\r\n| `-p, --pretty` | Pretty print JSON output | false |\r\n\r\n#### Examples\r\n\r\n**Consume from beginning and pretty print:**\r\n```bash\r\nkafka-console consume my-topic --from 0 --pretty\r\n```\r\n\r\n**Consume last 10 messages:**\r\n```bash\r\nkafka-console consume my-topic --count 10\r\n```\r\n\r\n**Consume from specific timestamp:**\r\n```bash\r\nkafka-console consume my-topic --from \"2024-01-01T00:00:00Z\"\r\n```\r\n\r\n**Consume with specific consumer group:**\r\n```bash\r\nkafka-console consume my-topic --group my-consumer-group\r\n```\r\n\r\n**Save output to file:**\r\n```bash\r\nkafka-console consume my-topic --output messages.json\r\n```\r\n\r\n**Extract specific fields with jq:**\r\n```bash\r\nkafka-console consume my-topic | jq '.value.userId'\r\n```\r\n\r\n### Producing Messages\r\n\r\n```bash\r\nkafka-console produce \u003ctopic\u003e [options]\r\n```\r\n\r\n#### Options\r\n| Option | Description | Default |\r\n|--------|-------------|---------|\r\n| `-i, --input \u003cfile\u003e` | Read input from file | stdin |\r\n| `-d, --data-format \u003cformat\u003e` | Message format (json/js/raw/custom) | json |\r\n| `-h, --header \u003cheader\u003e` | Add message header (format: key:value) | none |\r\n| `-w, --wait \u003cms\u003e` | Wait time between messages | 0 |\r\n\r\n#### Examples\r\n\r\n**Produce single message:**\r\n```bash\r\necho '{\"user\": \"john\", \"action\": \"login\"}' | kafka-console produce my-topic\r\n```\r\n\r\n**Produce from file:**\r\n```bash\r\nkafka-console produce my-topic --input messages.json\r\n```\r\n\r\n**Produce with headers:**\r\n```bash\r\necho '{\"data\": \"test\"}' | kafka-console produce my-topic --header \"source:api\" --header \"version:1.0\"\r\n```\r\n\r\n**Produce multiple messages from JSON array:**\r\n```bash\r\ncat users.json | jq -c '.[]' | kafka-console produce my-topic\r\n```\r\n\r\n**Produce with key (for partitioning):**\r\n```bash\r\necho '{\"key\": \"user123\", \"value\": {\"name\": \"John\"}}' | kafka-console produce my-topic\r\n```\r\n\r\n### Topic Management\r\n\r\n#### Create Topic\r\n```bash\r\nkafka-console topic:create my-new-topic\r\n```\r\n\r\n#### Delete Topic\r\n```bash\r\nkafka-console topic:delete old-topic\r\n```\r\n\r\n#### Show Topic Offsets\r\n```bash\r\nkafka-console topic:offsets my-topic\r\n```\r\n\r\n#### Show Topic Offsets for Specific Timestamp\r\n```bash\r\nkafka-console topic:offsets my-topic \"2024-01-01T00:00:00Z\"\r\n```\r\n\r\n### Cluster Information\r\n\r\n#### List All Topics\r\n```bash\r\nkafka-console list\r\n```\r\n\r\n#### List Including Internal Topics\r\n```bash\r\nkafka-console list --all\r\n```\r\n\r\n#### Show Cluster Metadata\r\n```bash\r\nkafka-console metadata\r\n```\r\n\r\n#### Show Topic Configuration\r\n```bash\r\nkafka-console config --resource topic --resourceName my-topic\r\n```\r\n\r\n## Authentication\r\n\r\n### SSL/TLS Connection\r\n```bash\r\nkafka-console consume my-topic \\\r\n  --brokers broker1:9093,broker2:9093 \\\r\n  --ssl\r\n```\r\n\r\n### SASL/PLAIN\r\n```bash\r\nkafka-console consume my-topic \\\r\n  --brokers broker:9093 \\\r\n  --ssl \\\r\n  --mechanism plain \\\r\n  --username myuser \\\r\n  --password mypassword\r\n```\r\n\r\n### SASL/SCRAM-SHA-256\r\n```bash\r\nkafka-console consume my-topic \\\r\n  --brokers broker:9093 \\\r\n  --ssl \\\r\n  --mechanism scram-sha-256 \\\r\n  --username myuser \\\r\n  --password mypassword\r\n```\r\n\r\n### AWS IAM\r\n```bash\r\nkafka-console consume my-topic \\\r\n  --brokers broker:9093 \\\r\n  --ssl \\\r\n  --mechanism aws \\\r\n  --access-key-id AKIAXXXXXXXX \\\r\n  --secret-access-key XXXXXXXXXX \\\r\n  --session-token XXXXXXXXXX\r\n```\r\n\r\n### OAuth Bearer\r\n```bash\r\nkafka-console consume my-topic \\\r\n  --brokers broker:9093 \\\r\n  --ssl \\\r\n  --mechanism oauthbearer \\\r\n  --oauth-bearer \"eyJhbGciOiJIUzI1NiIs...\"\r\n```\r\n\r\n## Message Formats\r\n\r\n### JSON Format (Default)\r\nMessages are parsed as JSON:\r\n```bash\r\necho '{\"name\": \"Alice\", \"age\": 30}' | kafka-console produce my-topic\r\n```\r\n\r\n### Raw Format\r\nMessages are sent as plain text:\r\n```bash\r\necho \"Plain text message\" | kafka-console produce my-topic --data-format raw\r\n```\r\n\r\n### JavaScript Format\r\nMessages can contain JavaScript exports:\r\n```bash\r\necho 'module.exports = { timestamp: Date.now() }' | kafka-console produce my-topic --data-format js\r\n```\r\n\r\n### Custom Formatter\r\nCreate a custom formatter module:\r\n\r\n```javascript\r\n// formatter/custom.js\r\nmodule.exports = {\r\n  encode: (value) =\u003e Buffer.from(JSON.stringify(value)),\r\n  decode: (buffer) =\u003e JSON.parse(buffer.toString())\r\n};\r\n```\r\n\r\nUse the custom formatter:\r\n```bash\r\nkafka-console consume my-topic --data-format ./formatter/custom.js\r\n```\r\n\r\n## Environment Variables\r\n\r\nSet environment variables to avoid repeating common options:\r\n\r\n```bash\r\nexport KAFKA_BROKERS=broker1:9092,broker2:9092\r\nexport KAFKA_USERNAME=myuser\r\nexport KAFKA_PASSWORD=mypassword\r\nexport KAFKA_MECHANISM=plain\r\nexport KAFKA_TIMEOUT=30000\r\n```\r\n\r\nAll supported environment variables:\r\n- `KAFKA_BROKERS` - Comma-separated list of brokers\r\n- `KAFKA_TIMEOUT` - Operation timeout in milliseconds\r\n- `KAFKA_MECHANISM` - SASL mechanism\r\n- `KAFKA_USERNAME` - SASL username\r\n- `KAFKA_PASSWORD` - SASL password\r\n- `KAFKA_AUTH_ID` - AWS authorization identity\r\n- `KAFKA_ACCESS_KEY_ID` - AWS access key ID\r\n- `KAFKA_SECRET_ACCESS_KEY` - AWS secret access key\r\n- `KAFKA_SESSION_TOKEN` - AWS session token\r\n- `KAFKA_OAUTH_BEARER` - OAuth bearer token\r\n\r\n## Common Use Cases\r\n\r\n### Monitor Topic in Real-time\r\n```bash\r\nkafka-console consume logs --group monitor-group --pretty\r\n```\r\n\r\n### Replay Messages from Yesterday\r\n```bash\r\nkafka-console consume events --from \"$(date -d yesterday --iso-8601)\"\r\n```\r\n\r\n### Copy Messages Between Topics\r\n```bash\r\nkafka-console consume source-topic | kafka-console produce destination-topic\r\n```\r\n\r\n### Filter Messages\r\n```bash\r\nkafka-console consume all-events | jq 'select(.value.type == \"ERROR\")' \r\n```\r\n\r\n### Count Messages in Topic\r\n```bash\r\nkafka-console consume my-topic --from 0 | wc -l\r\n```\r\n\r\n### Sample Messages\r\n```bash\r\nkafka-console consume large-topic --count 100 --pretty\r\n```\r\n\r\n### Debug Message Headers\r\n```bash\r\nkafka-console consume my-topic | jq '.headers'\r\n```\r\n\r\n## Troubleshooting\r\n\r\n### Connection Issues\r\n\r\n**Problem:** Cannot connect to Kafka broker\r\n```bash\r\nError: KafkaJSConnectionError: Connection timeout\r\n```\r\n**Solution:** \r\n- Verify broker addresses are correct\r\n- Check network connectivity: `telnet broker-host 9092`\r\n- Ensure security groups/firewalls allow connection\r\n- For Docker: use host network or proper port mapping\r\n\r\n### Authentication Failures\r\n\r\n**Problem:** Authentication failed\r\n```bash\r\nError: KafkaJSProtocolError: SASL authentication failed\r\n```\r\n**Solution:**\r\n- Verify credentials are correct\r\n- Check SASL mechanism matches broker configuration\r\n- Ensure SSL is enabled if required: `--ssl`\r\n\r\n### Consumer Group Issues\r\n\r\n**Problem:** Not receiving messages\r\n**Solution:**\r\n- Check consumer group offset: `kafka-console topic:offsets my-topic --group my-group`\r\n- Reset to beginning: `--from 0`\r\n- Use a new consumer group name\r\n\r\n### Message Format Errors\r\n\r\n**Problem:** JSON parsing errors\r\n```bash\r\nSyntaxError: Unexpected token...\r\n```\r\n**Solution:**\r\n- Verify message format matches specified data-format\r\n- Use `--data-format raw` for non-JSON messages\r\n- Check for malformed JSON with: `jq . \u003c input.json`\r\n\r\n### Performance Issues\r\n\r\n**Problem:** Slow message consumption\r\n**Solution:**\r\n- Increase batch size in consumer configuration\r\n- Use multiple consumer instances with same group\r\n- Check network latency to brokers\r\n\r\n### SSL/TLS Issues\r\n\r\n**Problem:** SSL handshake failed\r\n**Solution:**\r\n- Ensure `--ssl` flag is used\r\n- Verify broker SSL port (usually 9093)\r\n- Check certificate validity\r\n\r\n## License\r\n\r\nLicense [The MIT License](http://opensource.org/licenses/MIT)\r\nCopyright (c) 2024 Ivan Zakharchanka\r\n\r\n[npm-url]: https://www.npmjs.com/package/kafka-console\r\n[downloads-image]: https://img.shields.io/npm/dw/kafka-console.svg?maxAge=43200\r\n[npm-image]: https://img.shields.io/npm/v/kafka-console.svg?maxAge=43200","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3axap4ehko%2Fkafka-console","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F3axap4ehko%2Fkafka-console","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3axap4ehko%2Fkafka-console/lists"}