{"id":28724483,"url":"https://github.com/anshumanpattnaik/kafka-manager","last_synced_at":"2026-02-26T18:10:45.983Z","repository":{"id":291114429,"uuid":"959178664","full_name":"anshumanpattnaik/kafka-manager","owner":"anshumanpattnaik","description":"A Python library for managing Kafka Producers, Consumers and Topics","archived":false,"fork":false,"pushed_at":"2025-05-02T14:03:00.000Z","size":5786,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-02T14:28:25.445Z","etag":null,"topics":["kafka","kafka-manager","kafka-python","python"],"latest_commit_sha":null,"homepage":"https://kafka-manager-docs.readthedocs.io/en/latest/index.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anshumanpattnaik.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-02T11:48:55.000Z","updated_at":"2025-05-02T14:01:40.000Z","dependencies_parsed_at":"2025-05-02T14:38:30.726Z","dependency_job_id":null,"html_url":"https://github.com/anshumanpattnaik/kafka-manager","commit_stats":null,"previous_names":["anshumanpattnaik/kafka-manager"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/anshumanpattnaik/kafka-manager","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anshumanpattnaik%2Fkafka-manager","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anshumanpattnaik%2Fkafka-manager/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anshumanpattnaik%2Fkafka-manager/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anshumanpattnaik%2Fkafka-manager/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anshumanpattnaik","download_url":"https://codeload.github.com/anshumanpattnaik/kafka-manager/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anshumanpattnaik%2Fkafka-manager/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259960918,"owners_count":22938162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-manager","kafka-python","python"],"created_at":"2025-06-15T10:39:44.987Z","updated_at":"2026-02-26T18:10:40.925Z","avatar_url":"https://github.com/anshumanpattnaik.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"kafka-manager\n=============\n\n.. image:: https://img.shields.io/badge/license-MIT-blue\n    :target: https://github.com/anshumanpattnaik/kafka-manager/blob/main/LICENSE\n\n.. image:: https://codecov.io/gh/anshumanpattnaik/kafka-manager/graph/badge.svg?token=8DES91MFEU\n    :target: https://codecov.io/gh/anshumanpattnaik/kafka-manager\n\nA Kafka Manager is a Python utility class that simplifies Kafka interactions by providing a high-level abstraction for\nmanaging Producers, Consumers, and Topics. It provides a user-friendly interface for developers to implement Kafka\neffectively in their applications, encapsulating the complexity of the Kafka-python library. This abstraction allows\nquicker development and more manageable maintenance of Kafka-related applications.\n\nRead the documentation for more details: https://kafka-manager-docs.readthedocs.io/en/latest/index.html\n\nRequirements\n------------\n* Python 3.7+\n* kafka-python\n\nInstallation\n------------\n.. code:: bash\n\n    $ pip install kafka-manager\n\nFeatures\n--------\n\nProducer Management\n*******************\nIt provides interfaces to start/stop producers, send messages to topics, and check producer running status. It also invokes functions to initialize and terminate producer instances to publish messages to Kafka, and it effectively checks the producer status to ensure that messages are sent successfully.\n\n.. code:: python\n\n    import json\n\n    from kafka_manager.kafka_manager import KafkaManager\n\n    bootstrap_servers = ['localhost:9092']  # Replace with your Kafka broker addresses\n    topic_name = 'example_topic'  # Replace topic name with your choice\n    group_id = 'example_group'  # Replace consumer group ID with your choice\n\n    # Create a KafkaManager instance\n    kafka_manager = KafkaManager(bootstrap_servers=bootstrap_servers)\n\n    # Start the Kafka producer\n    kafka_manager.start_producer();\n\n    # Send Kafka message\n    try:\n        message_payload = json.dumps({\n            \"message_key\": \"message_value\"\n        })\n        metadata = kafka_manager.send_message(topic=topic_name, value=message_payload)\n        if metadata:\n            print(f'Message sent successfully to Kafka topic: \"{topic_name}\"')\n        else:\n            print(f'Failed to send message to Kafka topic: \"{topic_name}\"')\n    except Exception as e:\n        print(f'Error in sending message to Kafka topic: {e}')\n\n    # Stop Kafka producer\n    kafka_manager.stop_producer();\n\nConsumer Management\n*******************\nIt enables configuring various configurations to Create/Manage consumers and provides an interface to start/stop consumers. The Kafka Manager allows developers to create consumers per their application needs, such as different deserialization methods or offset management strategies. It provides a user-defined callback function to consume messages, allowing developers to define custom logic for processing each received message and enabling further data processing.\n\n.. code:: python\n\n    from kafka_manager.kafka_manager import KafkaManager\n\n    bootstrap_servers = ['localhost:9092']  # Replace with your Kafka broker addresses\n    topic_name = 'example_topic'  # Replace topic name with your choice\n    group_id = 'example_group'  # Replace consumer group ID with your choice\n\n    # Create a KafkaManager instance\n    kafka_manager = KafkaManager(bootstrap_servers=bootstrap_servers)\n\n    # Create a Kafka Consumer\n    consumer = kafka_manager.create_consumer(topics=[topic_name], group_id=group_id, auto_offset_reset='earliest')\n\n    # Start the Kafka Consumer\n    kafka_manager.start_consumer(consumer_id=group_id):\n\n    def message_handler(message):\n        \"\"\"\n        This method is a callback function called by the consumer, which handles the received messages when a new message\n        arrives.\n\n        In production real-world application, the received message would be processed as follows:\n        - Perform some business logic\n        - Store the message in a database for further processing.\n        - Message deserialization\n        - etc.\n\n        :param message: Message received from the consumer.\n        \"\"\"\n        print(f'Received message: Partition={message.partition}, Offset={message.offset}, Value={message.value}')\n\n    # Consume Messages\n    kafka_manager.consume_messages(consumer_id=group_id, message_handler=message_handler)\n\nTopic Management\n*******************\nKafka Manager allows developers to create and delete topics dynamically, which serve as categories from which messages are published. It's essential for managing data streams and evolving application requirements.\n\n.. code:: python\n\n    from kafka_manager.kafka_manager import KafkaManager\n\n    bootstrap_servers = ['localhost:9092']  # Replace with your Kafka broker addresses\n    topic_name = 'example_topic'  # Replace topic name with your choice\n    group_id = 'example_group'  # Replace consumer group ID with your choice\n\n    # Create a KafkaManager instance\n    kafka_manager = KafkaManager(bootstrap_servers=bootstrap_servers)\n\n    # For topic management connect to Kafka admin client\n    kafka_manager.connect_admin_client()\n\n    # Create a topic - (if it doesn't exist)\n    kafka_manager.create_topic(topic_name=topic_name, num_partitions=1, replication_factor=1)\n\nAdmin Client\n*******************\nIt provides interfaces to connect to the Kafka Admin client and allows developers to perform administrative operations such as creating and deleting topics. However, the admin-client connection is vital to performing many advanced Kafka management tasks, such as describing cluster configurations and managing Kafka ACLs.\n\n.. code:: python\n\n    from kafka_manager.kafka_manager import KafkaManager\n\n    bootstrap_servers = ['localhost:9092']  # Replace with your Kafka broker addresses\n    topic_name = 'example_topic'  # Replace topic name with your choice\n    group_id = 'example_group'  # Replace consumer group ID with your choice\n\n    # Create a KafkaManager instance\n    kafka_manager = KafkaManager(bootstrap_servers=bootstrap_servers)\n\n    # Connect to Kafka admin client\n    admin_client = kafka_manager.connect_admin_client()\n\n    # Listing Consumer Groups\n    consumers_groups = admin_client.list_consumer_groups()\n    print(consumers_groups)\n\n    # Describing Consumer Groups\n    admin_client.describe_consumer_groups(list(consumers_groups))\n\nError Handling\n*******************\nTo handle errors in Kafka due to network failures, broker failures, or misconfigurations, Kafka Manager handles these exceptions efficiently and ensures application stability.\n\nResource Management\n*******************\nKafka Manager resource management ensures that all connections to Kafka are correctly closed. It provides a close() function for proper shutdown, which prevents resource leaks and potential data corruption. It's essential for maintaining data integrity and managing the Kafka cluster and application.\n\nLicense\n*******\n\nMIT License, See `LICENSE \u003chttps://github.com/anshumanpattnaik/kafka-manager/blob/main/LICENSE\u003e`_.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanshumanpattnaik%2Fkafka-manager","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanshumanpattnaik%2Fkafka-manager","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanshumanpattnaik%2Fkafka-manager/lists"}