{"id":35672332,"url":"https://github.com/bborbe/kafka-topic-reader","last_synced_at":"2026-01-05T19:03:43.186Z","repository":{"id":306088420,"uuid":"921079072","full_name":"bborbe/kafka-topic-reader","owner":"bborbe","description":null,"archived":false,"fork":false,"pushed_at":"2025-11-28T09:06:18.000Z","size":6177,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-11-30T17:53:10.967Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bborbe.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-23T09:49:35.000Z","updated_at":"2025-11-28T09:06:21.000Z","dependencies_parsed_at":"2025-07-23T16:25:44.468Z","dependency_job_id":"8f2156a0-4277-4000-8f1c-bf78a0f13776","html_url":"https://github.com/bborbe/kafka-topic-reader","commit_stats":null,"previous_names":["bborbe/kafka-topic-reader"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/bborbe/kafka-topic-reader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bborbe%2Fkafka-topic-reader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bborbe%2Fkafka-topic-reader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bborbe%2Fkafka-topic-reader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bborbe%2Fkafka-topic-reader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bborbe","download_url":"https://codeload.github.com/bborbe/kafka-topic-reader/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bborbe%2Fkafka-topic-reader/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28218057,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2026-01-05T02:00:06.358Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-05T19:01:58.696Z","updated_at":"2026-01-05T19:03:38.826Z","avatar_url":"https://github.com/bborbe.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Topic Reader\n\nA Go service that provides HTTP access to read messages from Kafka topics. The service exposes a REST API to consume Kafka messages with pagination and filtering support.\n\n## Features\n\n- **HTTP API**: Read Kafka messages via REST endpoints\n- **Binary Filtering**: Filter messages by binary pattern matching\n- **Pagination**: Support for offset-based pagination with configurable limits\n- **Monitoring**: Prometheus metrics and health check endpoints\n- **Error Reporting**: Integration with Sentry for error tracking\n\n## Quick Start\n\n### Running the Service\n\n```bash\n# Run with command-line arguments\ngo run main.go -- \\\n  --listen=\":8080\" \\\n  --kafka-brokers=\"localhost:9092\" \\\n  --sentry-dsn=\"your-sentry-dsn-here\"\n\n# Run with environment variables (alternative)\nexport KAFKA_BROKERS=\"localhost:9092\"\nexport LISTEN=\":8080\"\nexport SENTRY_DSN=\"your-sentry-dsn-here\"\ngo run main.go\n\n# For local development, you can use a dummy Sentry DSN\ngo run main.go -- \\\n  --listen=\":8080\" \\\n  --kafka-brokers=\"localhost:9092\" \\\n  --sentry-dsn=\"https://dummy@dummy.ingest.sentry.io/dummy\"\n```\n\n### Docker\n\n```bash\n# Build Docker image\nmake build\n\n# Run with Docker\ndocker run -p 8080:8080 -e KAFKA_BROKERS=\"localhost:9092\" kafka-topic-reader\n```\n\n## API Reference\n\n### Read Messages\n\n```\nGET /read\n```\n\nRead Kafka messages with query parameters:\n\n**Parameters:**\n- `topic` (required) - Kafka topic name\n- `partition` (required) - Kafka partition number  \n- `offset` (required) - Starting offset (supports negative values for relative positioning)\n- `limit` (optional, default: 100) - Maximum number of records to return\n- `filter` (optional, max: 1024 bytes) - Binary substring filter for raw message values (exact byte matching, case-sensitive)\n\n**Example:**\n```bash\n# Read 10 messages from topic \"events\", partition 0, starting at offset 100\ncurl \"http://localhost:8080/read?topic=events\u0026partition=0\u0026offset=100\u0026limit=10\"\n\n# Filter messages containing \"error\"\ncurl \"http://localhost:8080/read?topic=logs\u0026partition=0\u0026offset=0\u0026filter=error\"\n\n# Use negative offset to read from end\ncurl \"http://localhost:8080/read?topic=events\u0026partition=0\u0026offset=-10\u0026limit=10\"\n```\n\n**Response:**\n```json\n{\n  \"records\": [\n    {\n      \"key\": \"message-key\",\n      \"value\": {\"data\": \"message content\"},\n      \"offset\": 100,\n      \"partition\": 0,\n      \"topic\": \"events\"\n    }\n  ],\n  \"nextOffset\": 101\n}\n```\n\n### Health Checks\n\n- `GET /healthz` - Health check endpoint\n- `GET /readiness` - Readiness check endpoint  \n- `GET /metrics` - Prometheus metrics\n\n### Log Level Management\n\n- `POST /setloglevel/{level}` - Dynamic log level adjustment\n\n## Configuration\n\nThe application supports both command-line arguments and environment variables:\n\n### Required Parameters\n- `--kafka-brokers` / `KAFKA_BROKERS` - Comma-separated list of Kafka broker addresses\n- `--listen` / `LISTEN` - HTTP server listen address (e.g., \":8080\")\n- `--sentry-dsn` / `SENTRY_DSN` - Sentry error reporting DSN\n\n### Optional Parameters  \n- `--sentry-proxy` / `SENTRY_PROXY` - Sentry proxy URL\n\n**Note for Development**: While Sentry DSN is marked as required, you can use a dummy DSN for local development.\n\n**Note**: Command-line arguments take precedence over environment variables.\n\n## Binary Filtering\n\nThe service supports binary pattern matching on raw Kafka message values:\n\n- **Case-sensitive**: Exact byte matching without case conversion\n- **Binary safe**: Works with any binary data, not just text\n- **Efficient**: Filtering happens before message conversion\n- **Size limit**: Filter parameter limited to 1024 bytes for security\n\n**Examples:**\n```bash\n# Filter JSON messages containing specific field\ncurl \"http://localhost:8080/read?topic=api-logs\u0026partition=0\u0026offset=0\u0026filter=user_id\"\n\n# Filter binary data (URL-encoded)\ncurl \"http://localhost:8080/read?topic=binary-data\u0026partition=0\u0026offset=0\u0026filter=%00%01%FF\"\n```\n\n## Development\n\n### Building and Testing\n\n```bash\n# Run full precommit pipeline (required before commits)\nmake precommit\n\n# Individual commands\nmake ensure        # Tidy and verify go modules\nmake format        # Format code with goimports-reviser\nmake generate      # Generate code (mocks, etc.)\nmake test          # Run tests with race detection and coverage\nmake check         # Run vet, errcheck, and vulnerability checks\n\n# Run all tests\ngo test -mod=mod ./...\n```\n\n### Docker Operations\n\n```bash\nmake build         # Build Docker image\nmake upload        # Push Docker image to registry\nmake clean         # Remove Docker image\n```\n\n## Architecture\n\nThe service is built using:\n\n- **Kafka Client**: IBM Sarama library for Kafka operations\n- **HTTP Router**: Gorilla Mux for request routing\n- **Testing**: Ginkgo v2 with Gomega for BDD-style testing\n- **Monitoring**: Prometheus metrics integration\n- **Error Handling**: Context-aware error handling with Sentry integration\n\n## License\n\nThis project is licensed under the BSD-style license. See the LICENSE file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbborbe%2Fkafka-topic-reader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbborbe%2Fkafka-topic-reader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbborbe%2Fkafka-topic-reader/lists"}