{"id":15017584,"url":"https://github.com/bookingcom/perl-net-kafka","last_synced_at":"2025-04-09T19:42:33.053Z","repository":{"id":56163585,"uuid":"217309680","full_name":"bookingcom/perl-Net-Kafka","owner":"bookingcom","description":"Net::Kafka - High-performant Perl client for Apache Kafka","archived":false,"fork":false,"pushed_at":"2023-09-08T14:51:47.000Z","size":73,"stargazers_count":12,"open_issues_count":6,"forks_count":10,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-23T21:45:41.713Z","etag":null,"topics":["apache-kafka","kafka","librdkafka","perl","perl5"],"latest_commit_sha":null,"homepage":null,"language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bookingcom.png","metadata":{"files":{"readme":"README","changelog":"Changes","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-24T13:44:57.000Z","updated_at":"2021-05-26T13:13:08.000Z","dependencies_parsed_at":"2024-06-21T15:19:56.062Z","dependency_job_id":"5a080bde-2407-4e84-8a35-a4834007b085","html_url":"https://github.com/bookingcom/perl-Net-Kafka","commit_stats":{"total_commits":11,"total_committers":1,"mean_commits":11.0,"dds":0.0,"last_synced_commit":"9602fd8b30a18be89a3918dbbc942afcf86e9ed6"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookingcom%2Fperl-Net-Kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookingcom%2Fperl-Net-Kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookingcom%2Fperl-Net-Kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookingcom%2Fperl-Net-Kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bookingcom","download_url":"https://codeload.github.com/bookingcom/perl-Net-Kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248101146,"owners_count":21047914,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","kafka","librdkafka","perl","perl5"],"created_at":"2024-09-24T19:50:42.854Z","updated_at":"2025-04-09T19:42:33.032Z","avatar_url":"https://github.com/bookingcom.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"NAME\n    Net::Kafka - High-performant Perl client for Apache Kafka\n\nSYNOPSIS\n        use Net::Kafka::Producer;\n        use Net::Kafka::Consumer;\n        use AnyEvent;\n\n        # Produce 1 message into \"my_topic\"\n        my $condvar     = AnyEvent-\u003econdvar;\n        my $producer    = Net::Kafka::Producer-\u003enew(\n            'bootstrap.servers' =\u003e 'localhost:9092'\n        );\n        $producer-\u003eproduce(\n            payload =\u003e \"message\",\n            topic   =\u003e \"my_topic\"\n        )-\u003ethen(sub {\n            my $delivery_report = shift;\n            $condvar-\u003esend;\n            print \"Message successfully delivered with offset \" . $delivery_report-\u003e{offset};\n        }, sub {\n            my $error = shift;\n            $condvar-\u003esend;\n            die \"Unable to produce a message: \" . $error-\u003e{error} . \", code: \" . $error-\u003e{code};\n        });\n        $condvar-\u003erecv;\n\n        # Consume message from \"my_topic\"\n        my $consumer    = Net::Kafka::Consumer-\u003enew(\n            'bootstrap.servers'     =\u003e 'localhost:9092',\n            'group.id'              =\u003e 'my_consumer_group',\n            'enable.auto.commit'    =\u003e 'true',\n        );\n\n        $consumer-\u003esubscribe( [ \"my_topic\" ] );\n        while (1) {\n            my $msg = $kafka-\u003econsumer_poll(1000);\n            if ($msg) {\n                if ( $msg-\u003eerr ) {\n                    say \"Error: \", Net::Kafka::Error::to_string($err);\n                }\n                else {\n                    say $msg-\u003epayload;\n                }\n            }\n        }\n\nDESCRIPTION\n    This module provides Perl bindings to librdkafka C client library. It is\n    heavily inspired by Kafka::Librd module originally developed by Pavel\n    Shaydo.\n\n    Please refer to the following modules documentation in order to\n    understand how to use it:\n\n    * `Net::Kafka::Producer' - asynchronous producer interface\n    * `Net::Kafka::Consumer' - consumer interface that supports both Simple\n    and Distributed modes\n\nREQUIREMENTS\n    * GNU make\n    * librdkafka \u003e= 1.0.0\n\nINSTALLATION\n    First install librdkafka\n    (https://github.com/edenhill/librdkafka#installation).\n\n  BUILD FROM CPAN\n        cpanm install Net::Kafka\n\n  BUILD FROM SOURCE\n    Sources are available on Github:\n    https://github.com/bookingcom/perl-Net-Kafka.\n\n        perl Makefile.pl\n        make\n        make test\n        make install\n\nNet::Kafka::Producer\n    The Net::Kafka::Producer module provides interface to librdkafka's\n    producer methods. It utilizes signal pipes, AnyEvent watcher and\n    AnyEvent::XSPromises to make its behaviour asynchronous. Taking that\n    into consideration you need to make sure to properly create condvar and\n    `send'/`recv' in order to collect all outstanding promises. It is highly\n    suggested to familirize yourself with both AnyEvent and\n    AnyEvent::XSPromises modules. See SYNOPSIS for example.\n\n  METHODS\n    new()\n            my $producer = Net::Kafka::Producer-\u003enew(\n                'bootstrap.servers' =\u003e 'localhost:9092'\n            );\n\n        Create an instance of Net::Kafka::Producer. Accept hash where keys\n        are equal to property names of librdkafka (see\n        https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)\n        . Note that only `error_cb' and `stats_cb' callbacks are supported\n        for Producer. Message delivery reports are served automatically\n        through `Promise' based `produce' method (see below).\n\n    produce()\n            my $promise = $producer-\u003eproduce(\n                payload     =\u003e \"my_message\",\n                topic       =\u003e \"my_topic\",\n                key         =\u003e \"my_key\",    # optional\n                timestamp   =\u003e 1234567,   # optional, if not specified current local timestamp will be used\n                partition   =\u003e 0          # optional, if not specified internal librdkafka partitioner will be used\n                headers     =\u003e $headers,  # Optional, see Net::Kafka::Headers\n            )-\u003ethen(sub {\n                my $delivery_report = shift;\n                print \"Message is sent with offset \" . $delivery_report-\u003e{offset};\n            })-\u003ecatch(sub {\n                my $error = shift;\n                print $error-\u003e{error} . \"\\n\";\n            });\n\n        Sends a message to Kafka. Accepts hash with parameters.\n\n        Returns back an instance of `Promise' that will be resolved/rejected\n        later. In case message is successfully send \"resolve\" callback will\n        receive a delievry report in the form of the hash that contains\n        `offset', `partition' and `timestamp'. If message delivery has\n        failed \"reject\" callback will receive a hash that contains `error'\n        (a human readable error description) and (optionally) `error_code'\n        that is equal to librdkafka's error code. All error codes are mapped\n        and exported by `Net::Kafka' module as constants (e.g.\n        `Net::Kafka::RD_KAFKA_RESP_ERR__PREV_IN_PROGRESS') for simplicity.\n\n    partitions_for()\n            my $partitions = $producer-\u003epartitions_for(\"my_topic\", $timeout_ms);\n\n        Returns an `ARRAYREF' that contains partition metadata information\n        about the given topic (leader, replicas, ISR replicas);\n\n    close()\n            $producer-\u003eclose();\n\n        Explicitly closees `Net::Kafka::Producer' instance and underlying\n        librdkafka handles.\n\nNet::Kafka::Consumer\n    The Net::Kafka::Consumer class provides interface to librdkafka's\n    consumer functionality. It supports both \"distributed\" (subscription\n    based) and \"simple\" (manual partition assignment) modes of work.\n\n  METHODS\n    new()\n            my $consumer = Net::Kafka::Consumer-\u003enew(\n                'bootstrap.servers'  =\u003e 'localhost:9092',\n                'group.id'           =\u003e \"my_consumer_group\",\n                'enable.auto.commit' =\u003e \"true\",\n            );\n\n        Create an instance of Net::Kafka::Consumer. Accept hash where keys\n        are equal to property names of librdkafka (see\n        https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)\n        . Note that not all callbacks are supported at the moment. Supported\n        ones are: `error_cb', `rebalance_cb', `commit_cb' and `stats_cb'.\n\n    subscribe()\n            $consumer-\u003esubscribe([ 'my_topic' ]);\n\n        Subscribe to topic set using balanced consumer groups. The main\n        entry-point for \"distributed\" consumer mode - partitions will be\n        assigned automatically using Kafka's GroupApi semantics.\n        Wildcard/regex topics are supported so matching topics will be added\n        to the subscription list.\n\n    unsubscribe()\n            $consumer-\u003eunsubscribe();\n\n        Unsubscribe from the current subscription set.\n\n    assign()\n            # manually assign partitions 0 and 1 to be consumed\n            my $tp_list = Net::Kafka::TopicPartitionList-\u003enew();\n            $tp_list-\u003eadd(\"my_topic\", 0);\n            $tp_list-\u003eadd(\"my_topic\", 1);\n            $consumer-\u003eassign($tp_list);\n\n        Atomic assignment of partitions to consume. The main entry-point for\n        \"simple\" consumer mode - partitions are assigned manually.\n\n    poll()\n            my $message = $consumer-\u003epoll($timeout_ms);\n\n        Poll the consumer for messages or events. Returns instance of\n        `Net::Kafka::Message'. Will block for at most `timeout_ms'\n        milliseconds. An application should make sure to call `poll' at\n        regular intervals.\n\n    committed()\n            my $tp_list = Net::Kafka::TopicPartitionList-\u003enew();\n            $tp_list-\u003eadd(\"my_topic\", 0);\n            $consumer-\u003ecommitted($tp_list);\n            my $offset = $tp_list-\u003eoffset(\"my_topic_, 0);\n\n        Retrieve committed offsets for topics+partitions.\n\n    offsets_for_times()\n            my $tp_list = Net::Kafka::TopicPartitionList-\u003enew();\n            $tp_list-\u003eadd(\"my_topic\", 0);\n            $tp_list-\u003eset_offset(\"my_topic\", 0, 958349923); # timestamp if passed through offset field\n            $consumer-\u003eoffsets_for_times($tp_list);\n            my $offset = $tp_list-\u003eoffset(\"my_topic\");\n\n        Look up the offsets for the given partitions by timestamp.\n\n    pause()\n            my $tp_list = Net::Kafka::TopicPartitionList-\u003enew();\n            $tp_list-\u003eadd(\"my_topic\", 0);\n            $consumer-\u003epause($tp_list); # pauses consumption of partition 0 of \"my_topic\"\n\n        Pause consumption for the provided list of partitions.\n\n    resume()\n            my $tp_list = Net::Kafka::TopicPartitionList-\u003enew();\n            $tp_list-\u003eadd(\"my_topic\", 0);\n            $consumer-\u003eresume($tp_list); # resumes consumption of partition 0 of \"my_topic\"\n\n        Resume consumption for the provided list of partitions.\n\n    subscription()\n            my $topics = $consumer-\u003esubscription();\n\n        Returns the current topic subscription\n\n    partitions_for()\n            my $partitions = $producer-\u003epartitions_for(\"my_topic\");\n\n        Returns an `ARRAYREF' that contains partition metadata information\n        about the given topic (leader, replicas, ISR replicas);\n\n    commit()\n            $consumer-\u003ecommit(); # commit current partition assignment (blocking call)\n            $consumer-\u003ecommit(1); # commit current partition assignment (non-blocking call)\n            my $tp_list = Net::Kafka::TopicPartitionList-\u003enew();\n            $tp_list-\u003eadd(\"my_topic\", 0);\n            $tp_list-\u003eset_offset(\"my_topic\", 0, 12345);\n            $consumer-\u003ecommit(0, $tp_list); # commit $tp_list assignment (blocking call);\n\n        Commit offsets on broker for the provided list of partitions. If no\n        partitions provided current assignment is committed instead.\n\n    commit_message();\n            my $message = $consumer-\u003epoll(1000);\n            $consumer-\u003ecommit_message(0, $message); # commit message (blocking call);\n            $consumer-\u003ecommit_message(1, $message); # commit message (non-blocking call);\n\n        Commit message's offset on broker for the message's partition.\n\n    position()\n            my $position_list = Net::Kafka::TopicPartitionList-\u003enew();\n            $position_list-\u003eadd(\"my_topic\", 0);\n            $consumer-\u003eposition($position_list);\n            my $position = $position_list-\u003eoffset(\"my_topic\", 0);\n\n        Retrieve current positions (offsets) for topics+partitions. The \\p\n        offset field of each requested partition will be set to the offset\n        of the last consumed message + 1, or RD_KAFKA_OFFSET_INVALID in case\n        there was no previous message.\n\n        Note: in this context the last consumed message is the offset\n        consumed by the current librdkafka instance and, in case of\n        rebalancing, not necessarily the last message fetched from the\n        partition.\n\n    seek()\n            $consumer-\u003eseek(\"my_topic\", 0, 12345); # seek partition 0 of \"my_topic\" to offset \"12345\"\n            $consumer-\u003eseek(\"my_topic\", 0, RD_KAFKA_OFFSET_BEGINNING); # seek to the beginning of \"my_topic\" partition 0\n            $consumer-\u003eseek(\"my_topic\", 0, RD_KAFKA_OFFSET_END); # seek to the end of \"my_topic\" partition 0\n\n        Seek consumer for topic+partition to offset which is either an\n        absolute or logical offset.\n\n    query_watermark_offsets()\n            my ($low, $high) = $consumer-\u003equery_watermark_offsets(\"my_topic\", 0);\n\n        Queries Kafka Broker for lowest and highest watermark offsets in the\n        given topic-partition.\n\n    close()\n            $consumer-\u003eclose();\n\n        Close all consumer handles. Make sure to call it before destroying\n        your application to make sure that all outstanding requests to be\n        flushed.\n\nNet::Kafka::Message\n    This class maps to `rd_kafka_message_t' structure from librdkafka and\n    represents message or event. Objects of this class have the following\n    methods:\n\n    err()\n        return error code from the message\n\n    topic()\n        return topic name\n\n    partition()\n        return partition number\n\n    offset()\n        return offset. Note, that the value is truncated to 32 bit if your\n        perl doesn't support 64 bit integers.\n\n    key()\n        return message key\n\n    payload()\n        return message payload\n\n    headers()\n        return a copy of message headers\n\n    detach_headers()\n        return message headers and removes them from the message\n\nNet::Kafka::Headers\n    This class contains a list of Kafka headers (it allows duplicates).\n    Objects of this class have the following methods:\n\n    new()\n        create a new instance\n\n    add(name, value)\n        append a new name/value pair to the header list\n\n    remove(name)\n        remove all headers with the given name, if any\n\n    get_last(name)\n        return the last value associated with a given name\n\n    to_hash()\n        return an hash-of-arrays containing all headers\n\nNet::Kafka::Err\n    This class provides static methods to convert error codes into names and\n    descriptions.\n\n    rd_kafka_get_err_descs()\n            rd_kafka_get_err_descs()\n\n        returns a hash mapping error codes to description strings.\n\n    to_string()\n            to_string($code)\n\n        return the description string for this error code.\n\n    to_name()\n            to_name($code)\n\n        return the name of this error code.\n\nCAVEATS\n    Message offset is truncated to 32 bit if perl is compiled without\n    support for 64 bit integers.\n\nSEE ALSO\n    * https://github.com/edenhill/librdkafka\n    * https://github.com/trinitum/perl-Kafka-Librd\n\nLICENSE AND COPYRIGHT\n    Copyright (C) 2016, 2017 Pavel Shaydo\n\n    Copyright (C) 2018, 2019 Booking.com\n\n    This program is free software; you can redistribute it and/or modify it\n    under the terms of either: the GNU General Public License as published\n    by the Free Software Foundation; or the Artistic License.\n\n    See http://dev.perl.org/licenses/ for more information.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbookingcom%2Fperl-net-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbookingcom%2Fperl-net-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbookingcom%2Fperl-net-kafka/lists"}