{"id":35672296,"url":"https://github.com/bborbe/kafka","last_synced_at":"2026-01-05T19:03:24.153Z","repository":{"id":226206269,"uuid":"768009306","full_name":"bborbe/kafka","owner":"bborbe","description":null,"archived":false,"fork":false,"pushed_at":"2025-12-30T14:28:31.000Z","size":8243,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-01-03T08:05:04.053Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bborbe.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-03-06T09:56:45.000Z","updated_at":"2025-12-30T14:28:35.000Z","dependencies_parsed_at":"2024-05-10T13:49:07.805Z","dependency_job_id":"a677a8a6-0820-49f4-b041-6d4ab3737a01","html_url":"https://github.com/bborbe/kafka","commit_stats":null,"previous_names":["bborbe/kafka"],"tags_count":72,"template":false,"template_full_name":null,"purl":"pkg:github/bborbe/kafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bborbe%2Fkafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bborbe%2Fkafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bborbe%2Fkafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bborbe%2Fkafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bborbe","download_url":"https://codeload.github.com/bborbe/kafka/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bborbe%2Fkafka/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28218056,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2026-01-05T02:00:06.358Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-05T19:01:50.304Z","updated_at":"2026-01-05T19:03:19.738Z","avatar_url":"https://github.com/bborbe.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Library\n\n[![CI](https://github.com/bborbe/kafka/actions/workflows/ci.yml/badge.svg)](https://github.com/bborbe/kafka/actions/workflows/ci.yml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/bborbe/kafka)](https://goreportcard.com/report/github.com/bborbe/kafka)\n[![Go Reference](https://pkg.go.dev/badge/github.com/bborbe/kafka.svg)](https://pkg.go.dev/github.com/bborbe/kafka)\n[![License](https://img.shields.io/badge/License-BSD%202--Clause-blue.svg)](LICENSE)\n\nA production-ready Kafka abstraction library for Go, built on top of IBM's Sarama client. Provides a clean interface for Kafka operations while adding essential features like metrics, batch processing, transaction support, and comprehensive message handling patterns.\n\n## Features\n\n- 🚀 **Production Ready**: Built for high-throughput, low-latency applications\n- 🔧 **Interface-Driven Architecture**: Composition-friendly with extensive interface support\n- 📊 **Built-in Metrics**: Prometheus integration throughout all components\n- 🔄 **Batch Processing**: Efficient batch message handling with configurable parameters\n- 🔒 **Transaction Support**: Atomic message processing with transaction decorators\n- 🎯 **Flexible Patterns**: Function types, decorators, and composition patterns\n- 🛡️ **TLS Support**: Built-in TLS configuration for secure connections\n- ⚡ **Offset Management**: Multiple offset tracking strategies and fallback behavior\n\n---\n\n## Table of Contents\n\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n  - [Basic Producer](#basic-producer)\n  - [JSON Message Sender](#json-message-sender)\n  - [Basic Consumer](#basic-consumer)\n- [Advanced Usage](#advanced-usage)\n  - [Batch Processing](#batch-processing)\n  - [Metrics Integration](#metrics-integration)\n  - [Transaction Support](#transaction-support)\n  - [Performance Tuning](#performance-tuning)\n- [Architecture](#architecture)\n- [Testing](#testing)\n- [Development](#development)\n- [Dependencies](#dependencies)\n- [License](#license)\n\n---\n\n## Installation\n\n```bash\ngo get github.com/bborbe/kafka\n```\n\n## Quick Start\n\n### Basic Producer\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"log\"\n\n    \"github.com/bborbe/kafka\"\n    \"github.com/IBM/sarama\"\n)\n\nfunc main() {\n    ctx := context.Background()\n    brokers := kafka.ParseBrokers(\"localhost:9092\")\n\n    // Create producer with optional Sarama config\n    producer, err := kafka.NewSyncProducer(ctx, brokers)\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer producer.Close()\n\n    // Send message\n    msg := \u0026sarama.ProducerMessage{\n        Topic: \"my-topic\",\n        Value: sarama.StringEncoder(\"Hello Kafka!\"),\n    }\n\n    partition, offset, err := producer.SendMessage(ctx, msg)\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    log.Printf(\"Message sent to partition %d at offset %d\", partition, offset)\n}\n```\n\n### JSON Message Sender\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"log\"\n\n    \"github.com/bborbe/kafka\"\n    \"github.com/bborbe/log\"\n)\n\nfunc main() {\n    ctx := context.Background()\n    brokers := kafka.ParseBrokers(\"localhost:9092\")\n\n    // Create producer\n    producer, err := kafka.NewSyncProducer(ctx, brokers)\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer producer.Close()\n\n    // Create JSON sender with producer and log sampler factory\n    logSamplerFactory := log.NewLogSamplerFactory()\n    sender := kafka.NewJsonSender(producer, logSamplerFactory)\n\n    // Define a value type for your data\n    type EventValue struct {\n        UserID    int    `json:\"user_id\"`\n        Action    string `json:\"action\"`\n        Timestamp string `json:\"timestamp\"`\n    }\n\n    // Create and send update\n    value := EventValue{\n        UserID:    123,\n        Action:    \"login\",\n        Timestamp: \"2023-01-01T00:00:00Z\",\n    }\n\n    key := kafka.NewKey(\"user-123\")\n    err = sender.SendUpdate(ctx, kafka.Topic(\"events\"), key, value)\n    if err != nil {\n        log.Fatal(err)\n    }\n}\n```\n\n### Basic Consumer\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"log\"\n\n    \"github.com/bborbe/kafka\"\n    \"github.com/bborbe/log\"\n    \"github.com/IBM/sarama\"\n)\n\nfunc main() {\n    ctx := context.Background()\n    brokers := kafka.ParseBrokers(\"localhost:9092\")\n\n    // Create Sarama client\n    config := sarama.NewConfig()\n    config.Version = sarama.V2_6_0_0\n    saramaClient, err := sarama.NewClient(brokers.Hosts(), config)\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer saramaClient.Close()\n\n    // Create message handler\n    handler := kafka.MessageHandlerFunc(func(ctx context.Context, msg *sarama.ConsumerMessage) error {\n        log.Printf(\"Received message: %s\", string(msg.Value))\n        return nil\n    })\n\n    // Create consumer with log sampler factory\n    logSamplerFactory := log.NewLogSamplerFactory()\n    consumer := kafka.NewSimpleConsumer(\n        saramaClient,\n        kafka.Topic(\"my-topic\"),\n        kafka.OffsetNewest,\n        handler,\n        logSamplerFactory,\n    )\n\n    // Start consuming\n    if err := consumer.Consume(ctx); err != nil {\n        log.Fatal(err)\n    }\n}\n```\n\n## Advanced Usage\n\n### Batch Processing\n\n```go\n// Create batch handler from your existing handler\nbatchHandler := kafka.NewMessageHandlerBatch(\n    handler,\n    kafka.ParseBatchSize(100),                    // Process 100 messages at once\n    kafka.NewMessageHandlerBatchDelay(time.Second), // Or wait 1 second\n)\n\n// Use with NewSimpleConsumerBatch for batch processing\nconsumer := kafka.NewSimpleConsumerBatch(\n    saramaClient,\n    kafka.Topic(\"my-topic\"),\n    kafka.OffsetNewest,\n    batchHandler,\n    100, // batch size\n    logSamplerFactory,\n)\n```\n\n### Metrics Integration\n\n```go\n// Wrap producer with metrics\nproducer = kafka.NewSyncProducerMetrics(producer, metrics)\n\n// Wrap message handler with metrics  \nhandler = kafka.NewMessageHandlerMetrics(handler, metrics)\n```\n\n### Transaction Support\n\n```go\n// Create transactional message handler\ntxHandler := kafka.NewMessageHandlerTx(ctx, handler, db)\n\nconsumer, err := kafka.NewSimpleConsumer(ctx, brokers, \"my-topic\", txHandler)\n```\n\n### Performance Tuning\n\nFor high-throughput scenarios, customize the Sarama configuration when creating the client provider:\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"log\"\n    \"time\"\n\n    \"github.com/bborbe/kafka\"\n    \"github.com/bborbe/errors\"\n    \"github.com/IBM/sarama\"\n)\n\nfunc main() {\n    ctx := context.Background()\n    brokers := kafka.ParseBrokers(\"localhost:9092\")\n\n    // Create high-performance client provider with tuned configuration\n    saramaClientProvider, err := kafka.NewSaramaClientProviderByType(\n        ctx,\n        kafka.SaramaClientProviderTypeReused,\n        brokers,\n        func(config *sarama.Config) {\n            config.Consumer.MaxWaitTime = 1000 * time.Millisecond  // Increase from 500ms for larger batches\n            config.Consumer.Fetch.Default = 10 * 1024 * 1024       // 10MB (from 1MB default)\n            config.Consumer.Fetch.Max = 50 * 1024 * 1024           // 50MB max fetch size\n            config.ChannelBufferSize = 1000                         // Buffer 1000 messages (from 256 default)\n        },\n    )\n    if err != nil {\n        log.Fatal(errors.Wrapf(ctx, err, \"create sarama client provider failed\"))\n    }\n    defer saramaClientProvider.Close()\n\n    // Use the provider with your consumer...\n}\n```\n\n**Configuration Parameters:**\n\n- **`Consumer.MaxWaitTime`**: Maximum time broker waits before responding (higher = larger batches but higher latency; default: 500ms)\n- **`Consumer.Fetch.Default`**: Target data to fetch per request (higher = fewer requests, better throughput; default: 1MB)\n- **`Consumer.Fetch.Max`**: Maximum data broker can return (protects against excessive memory usage; default: varies)\n- **`ChannelBufferSize`**: Internal channel buffer size (higher = better throughput under load, more memory; default: 256)\n\n**Trade-offs:** This configuration optimizes for **throughput over latency**. Larger fetch sizes, longer wait times, and bigger buffers increase memory usage and per-message latency but significantly improve overall throughput. Use for high-volume scenarios where batch processing efficiency matters more than individual message latency.\n\n## Architecture\n\n### Core Interfaces\n\n- **`Consumer`** - Basic message consumption interface\n- **`SyncProducer`** - Synchronous message production interface  \n- **`MessageHandler`** - Message processing logic interface\n- **`OffsetManager`** - Offset tracking strategies interface\n\n### Design Patterns\n\n- **Interface-Driven**: All major components are interface-based for easy composition and testing\n- **Decorator Pattern**: Wrap components with additional functionality (metrics, transactions, filtering)\n- **Function Types**: Functional programming support via `ConsumerFunc`, `MessageHandlerFunc`\n\n## Testing\n\nThis library uses [Ginkgo](https://github.com/onsi/ginkgo) and [Gomega](https://github.com/onsi/gomega) for testing.\n\n```bash\n# Run all tests\nmake test\n\n# Run tests with verbose output\nginkgo -v\n\n# Run specific test\ngo test -run TestSpecific\n```\n\n## Development\n\n```bash\n# Install dependencies and run all checks\nmake precommit\n\n# Format code\nmake format\n\n# Generate mocks\nmake generate\n\n# Run linting and security checks\nmake check\n```\n\n## Dependencies\n\n- **Kafka Client**: [IBM Sarama v1.45.2](https://github.com/IBM/sarama)\n- **Testing**: [Ginkgo v2](https://github.com/onsi/ginkgo) \u0026 [Gomega](https://github.com/onsi/gomega)\n- **Metrics**: [Prometheus Client](https://github.com/prometheus/client_golang)\n\n## License\n\nThis project is licensed under the BSD 2-Clause License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbborbe%2Fkafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbborbe%2Fkafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbborbe%2Fkafka/lists"}