{"id":13699985,"url":"https://github.com/arnaud-lb/php-rdkafka","last_synced_at":"2025-05-13T19:13:45.667Z","repository":{"id":29600523,"uuid":"33140616","full_name":"arnaud-lb/php-rdkafka","owner":"arnaud-lb","description":"Production-ready, stable Kafka client for PHP","archived":false,"fork":false,"pushed_at":"2025-02-24T16:10:40.000Z","size":997,"stargazers_count":2141,"open_issues_count":40,"forks_count":267,"subscribers_count":79,"default_branch":"7.x","last_synced_at":"2025-04-27T20:06:43.006Z","etag":null,"topics":["kafka","kafka-client","php","php-extension"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arnaud-lb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"arnaud-lb"}},"created_at":"2015-03-30T18:33:14.000Z","updated_at":"2025-04-24T10:03:59.000Z","dependencies_parsed_at":"2024-06-01T12:59:30.758Z","dependency_job_id":"669c19c5-0b95-4fbc-af44-988b0e76ae03","html_url":"https://github.com/arnaud-lb/php-rdkafka","commit_stats":null,"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arnaud-lb%2Fphp-rdkafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arnaud-lb%2Fphp-rdkafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arnaud-lb%2Fphp-rdkafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arnaud-lb%2Fphp-rdkafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arnaud-lb","download_url":"https://codeload.github.com/arnaud-lb/php-rdkafka/tar.gz/refs/heads/7.x","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254010813,"owners_count":21998995,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-client","php","php-extension"],"created_at":"2024-08-02T20:00:46.795Z","updated_at":"2025-05-13T19:13:45.645Z","avatar_url":"https://github.com/arnaud-lb.png","language":"C","funding_links":["https://github.com/sponsors/arnaud-lb"],"categories":["客户端","Development","队列( Queue )","Clients"],"sub_categories":["Client libraries","PHP"],"readme":"# PHP Kafka client - php-rdkafka\n\n[![Join the chat at https://gitter.im/arnaud-lb/php-rdkafka](https://badges.gitter.im/arnaud-lb/php-rdkafka.svg)](https://gitter.im/arnaud-lb/php-rdkafka?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n[![Supported librdkafka versions: \u003e= 0.11](https://img.shields.io/badge/librdkafka-%3E%3D%200.11-blue.svg)](https://github.com/edenhill/librdkafka/releases) [![Supported Kafka versions: \u003e= 0.8](https://img.shields.io/badge/kafka-%3E%3D%200.8-blue.svg)](https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#broker-version-compatibility) ![Supported PHP versions: 7.x .. 8.x](https://img.shields.io/badge/php-7.x%20..%208.x-blue.svg)\n\nPHP-rdkafka is a **stable**, **production-ready**, and **fast** Kafka client for PHP based on [librdkafka](https://github.com/edenhill/librdkafka).\n\nCurrent version supports PHP \u003e= 8.1.0, librdkafka \u003e= 1.5.3, Kafka \u003e= 0.8. Version [6.x](https://github.com/arnaud-lb/php-rdkafka/tree/6.x) supports PHP 7.x..8.x, librdkafka 0.11..2.x. Older versions support PHP 5.\n\nThe goal of the extension is to be a low-level un-opinionated librdkafka binding focused on production and long term support.\n\nThe high level and low level *consumers*, *producer*, and *metadata* APIs are supported.\n\nDocumentation is available [here](https://arnaud-lb.github.io/php-rdkafka/phpdoc/book.rdkafka.html).\n\n## Table of Contents\n\n1. [Installation](#installation)\n2. [Examples](#examples)\n3. [Usage](#usage)\n   * [Producing](#producing)\n   * [High-level consuming](#high-level-consuming)\n   * [Low-level consuming (legacy)](#low-level-consuming-legacy)\n   * [Low-level consuming from multiple topics / partitions (legacy)](#low-level-consuming-from-multiple-topics--partitions-legacy)\n   * [Using stored offsets](#using-stored-offsets)\n   * [Interesting configuration parameters](#interesting-configuration-parameters)\n     * [queued.max.messages.kbytes](#queuedmaxmessageskbytes)\n     * [topic.metadata.refresh.sparse and topic.metadata.refresh.interval.ms](#topicmetadatarefreshsparse-and-topicmetadatarefreshintervalms)\n     * [internal.termination.signal](#internalterminationsignal)\n4. [Documentation](#documentation)\n5. [Credits](#credits)\n6. [License](#license)\n\n## Installation\n\nhttps://arnaud-lb.github.io/php-rdkafka-doc/phpdoc/rdkafka.setup.html\n\n## Examples\n\nhttps://arnaud-lb.github.io/php-rdkafka-doc/phpdoc/rdkafka.examples.html\n\n## Usage\n\nConfiguration parameters used below can be found in [Librdkafka Configuration reference](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)\n\n### Producing\n\n#### Creating a producer\nFor producing, we first need to create a producer, and to add brokers (Kafka\nservers) to it:\n\n``` php\n\u003c?php\n$conf = new RdKafka\\Conf();\n$conf-\u003eset('log_level', (string) LOG_DEBUG);\n$conf-\u003eset('debug', 'all');\n$rk = new RdKafka\\Producer($conf);\n$rk-\u003eaddBrokers(\"10.0.0.1:9092,10.0.0.2:9092\");\n```\n\n#### Producing messages\n\n\u003e **Warning** Make sure that your producer follows proper shutdown (see below) to not lose messages.  \n\nNext, we create a topic instance from the producer:\n``` php\n\u003c?php\n\n$topic = $rk-\u003enewTopic(\"test\");\n```\n\nFrom there, we can produce as much messages as we want, using the produce\nmethod:\n``` php\n\u003c?php\n\n$topic-\u003eproduce(RD_KAFKA_PARTITION_UA, 0, \"Message payload\");\n```\nThe first argument is the partition. RD_KAFKA_PARTITION_UA stands for\n*unassigned*, and lets librdkafka choose the partition.  \nThe second argument are message flags and should be either 0  \nor `RD_KAFKA_MSG_F_BLOCK` to block produce on full queue.\nThe message payload can be anything.\n\n#### Proper shutdown\n\nThis should be done prior to destroying a producer instance  \nto make sure all queued and in-flight produce requests are completed  \nbefore terminating. Use a reasonable value for `$timeout_ms`.  \n\n\u003e **Warning** Not calling flush can lead to message loss!\n\n```php\n$rk-\u003eflush($timeout_ms);\n```\n\nIn case you don't care about sending messages that haven't been sent yet,\nyou can use `purge()` before calling `flush()`:\n\n```php\n// Forget messages that are not fully sent yet\n$rk-\u003epurge(RD_KAFKA_PURGE_F_QUEUE);\n\n$rk-\u003eflush($timeout_ms);\n```\n\n### High-level consuming\n\nThe RdKafka\\KafkaConsumer class supports automatic partition assignment/revocation. See the example [here](https://arnaud-lb.github.io/php-rdkafka-doc/phpdoc/rdkafka.examples.html#example-1).\n\n### Low-level consuming (legacy)\n\n\u003e **Note** The low-level consumer is a legacy API, please prefer using the high-level consumer\n\nWe first need to create a low level consumer, and to add brokers (Kafka\nservers) to it:\n\n``` php\n\u003c?php\n$conf = new RdKafka\\Conf();\n$conf-\u003eset('log_level', (string) LOG_DEBUG);\n$conf-\u003eset('debug', 'all');\n$rk = new RdKafka\\Consumer($conf);\n$rk-\u003eaddBrokers(\"10.0.0.1,10.0.0.2\");\n```\n\nNext, create a topic instance by calling the `newTopic()` method, and start\nconsuming on partition 0:\n\n``` php\n\u003c?php\n\n$topic = $rk-\u003enewTopic(\"test\");\n\n// The first argument is the partition to consume from.\n// The second argument is the offset at which to start consumption. Valid values\n// are: RD_KAFKA_OFFSET_BEGINNING, RD_KAFKA_OFFSET_END, RD_KAFKA_OFFSET_STORED.\n$topic-\u003econsumeStart(0, RD_KAFKA_OFFSET_BEGINNING);\n```\n\nNext, retrieve the consumed messages:\n\n``` php\n\u003c?php\n\nwhile (true) {\n    // The first argument is the partition (again).\n    // The second argument is the timeout.\n    $msg = $topic-\u003econsume(0, 1000);\n    if (null === $msg || $msg-\u003eerr === RD_KAFKA_RESP_ERR__PARTITION_EOF) {\n        // Constant check required by librdkafka 0.11.6. Newer librdkafka versions will return NULL instead.\n        continue;\n    } elseif ($msg-\u003eerr) {\n        echo $msg-\u003eerrstr(), \"\\n\";\n        break;\n    } else {\n        echo $msg-\u003epayload, \"\\n\";\n    }\n}\n```\n\n### Low-level consuming from multiple topics / partitions (legacy)\n\n\u003e **Note** The low-level consumer is a legacy API, please prefer using the high-level consumer\n\nConsuming from multiple topics and/or partitions can be done by telling\nlibrdkafka to forward all messages from these topics/partitions to an internal\nqueue, and then consuming from this queue:\n\nCreating the queue:\n\n``` php\n\u003c?php\n$queue = $rk-\u003enewQueue();\n```\n\nAdding topic partitions to the queue:\n\n``` php\n\u003c?php\n\n$topic1 = $rk-\u003enewTopic(\"topic1\");\n$topic1-\u003econsumeQueueStart(0, RD_KAFKA_OFFSET_BEGINNING, $queue);\n$topic1-\u003econsumeQueueStart(1, RD_KAFKA_OFFSET_BEGINNING, $queue);\n\n$topic2 = $rk-\u003enewTopic(\"topic2\");\n$topic2-\u003econsumeQueueStart(0, RD_KAFKA_OFFSET_BEGINNING, $queue);\n```\n\nNext, retrieve the consumed messages from the queue:\n\n``` php\n\u003c?php\n\nwhile (true) {\n    // The only argument is the timeout.\n    $msg = $queue-\u003econsume(1000);\n    if (null === $msg || $msg-\u003eerr === RD_KAFKA_RESP_ERR__PARTITION_EOF) {\n        // Constant check required by librdkafka 0.11.6. Newer librdkafka versions will return NULL instead.\n        continue;\n    } elseif ($msg-\u003eerr) {\n        echo $msg-\u003eerrstr(), \"\\n\";\n        break;\n    } else {\n        echo $msg-\u003epayload, \"\\n\";\n    }\n}\n```\n\n### Using stored offsets\n\n#### Broker (default)\nlibrdkafka per default stores offsets on the broker.\n\n#### File offsets (deprecated)\nIf you're using local file for offset storage, then by default the file is created in the current directory, with a\nname based on the topic and the partition. The directory can be changed by setting the ``offset.store.path``\n[configuration property](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md).\n\n### Consumer settings\n\n#### Low-level consumer: auto commit settings\nTo manually control the offset, set `enable.auto.offset.store` to `false`.  \nThe settings `auto.commit.interval.ms` and `auto.commit.enable` will control  \nif the stored offsets will be auto committed to the broker and in which interval.\n\n#### High-level consumer: auto commit settings\nTo manually control the offset, set `enable.auto.commit` to `false`.\n\n#### High level consumer: max.poll.interval.ms\nMaximum allowed time between calls to consume messages for high-level consumers.  \nIf this interval is exceeded the consumer is considered failed and the group will  \nrebalance in order to reassign the partitions to another consumer group member.\n\n#### Consumer group id (general)\n`group.id` is responsible for setting your consumer group ID and it should be unique (and should\nnot change). Kafka uses it to recognize applications and store offsets for them.\n\n``` php\n\u003c?php\n\n$topicConf = new RdKafka\\TopicConf();\n$topicConf-\u003eset(\"auto.commit.interval.ms\", 1e3);\n\n$topic = $rk-\u003enewTopic(\"test\", $topicConf);\n\n$topic-\u003econsumeStart(0, RD_KAFKA_OFFSET_STORED);\n```\n\n### Interesting configuration parameters\n\n[Librdkafka Configuration reference](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)\n\n#### queued.max.messages.kbytes\n\nlibrdkafka will buffer up to 1GB of messages for each consumed partition by default. You can lower memory usage by reducing the value of the ``queued.max.messages.kbytes`` parameter on your consumers.\n\n### topic.metadata.refresh.sparse and topic.metadata.refresh.interval.ms\n\nEach consumer and producer instance will fetch topics metadata at an interval defined by the ``topic.metadata.refresh.interval.ms`` parameter. Depending on your librdkafka version, the parameter defaults to 10 seconds, or 600 seconds.\n\nlibrdkafka fetches the metadata for all topics of the cluster by default. Setting ``topic.metadata.refresh.sparse`` to the string ``\"true\"`` makes sure that librdkafka fetches only the topics he uses.\n\nSetting ``topic.metadata.refresh.sparse`` to ``\"true\"``, and ``topic.metadata.refresh.interval.ms`` to 600 seconds (plus some jitter) can reduce the bandwidth a lot, depending on the number of consumers and topics.\n\n### internal.termination.signal\n\nThis setting allows librdkafka threads to terminate as soon as librdkafka is done with them. This effectively allows your PHP processes / requests to terminate quickly.\n\nWhen enabling this, you have to mask the signal like this:\n\n``` php\n\u003c?php\n// once\npcntl_sigprocmask(SIG_BLOCK, array(SIGIO));\n// any time\n$conf-\u003eset('internal.termination.signal', SIGIO);\n```\n\n### socket.blocking.max.ms (librdkafka \u003c 1.0.0)\n\n\u003e Maximum time a broker socket operation may block. A lower value improves responsiveness at the expense of slightly higher CPU usage.\n\nReducing the value of this setting improves shutdown speed. The value defines the maximum time librdkafka will block in one iteration of a read loop. This also defines how often the main librdkafka thread will check for termination.\n\n### queue.buffering.max.ms\n\nThis defines the maximum and default time librdkafka will wait before sending a batch of messages. Reducing this setting to e.g. 1ms ensures that messages are sent ASAP, instead of being batched.\n\nThis has been seen to reduce the shutdown time of the rdkafka instance, and of the PHP process / request.\n\n## Performance / Low-latency settings\n\nHere is a configuration optimized for low latency. This allows a PHP process / request to send messages ASAP and to terminate quickly.\n\n``` php\n\u003c?php\n\n$conf = new \\RdKafka\\Conf();\n$conf-\u003eset('socket.timeout.ms', 50); // or socket.blocking.max.ms, depending on librdkafka version\nif (function_exists('pcntl_sigprocmask')) {\n    pcntl_sigprocmask(SIG_BLOCK, array(SIGIO));\n    $conf-\u003eset('internal.termination.signal', SIGIO);\n} else {\n    $conf-\u003eset('queue.buffering.max.ms', 1);\n}\n\n$producer = new \\RdKafka\\Producer($conf);\n$consumer = new \\RdKafka\\Consumer($conf);\n```\n\nIt is advised to call poll at regular intervals to serve callbacks. In `php-rdkafka:3.x`  \npoll was also called during shutdown, so not calling it in regular intervals might  \nlead to a slightly longer shutdown. The example below polls until there are no more events in the queue:\n\n```\n$producer-\u003eproduce(...);\nwhile ($producer-\u003egetOutQLen() \u003e 0) {\n    $producer-\u003epoll(1);\n}\n```\n\n## Documentation\n\nhttps://arnaud-lb.github.io/php-rdkafka-doc/phpdoc/book.rdkafka.html  \nThe source of the documentation can be found [here](https://github.com/arnaud-lb/php-rdkafka-doc)\n\n## Asking for Help\n\nIf the documentation is not enough, feel free to ask a questions on the php-rdkafka channels on [Gitter](https://gitter.im/arnaud-lb/php-rdkafka) or [Google Groups](https://groups.google.com/forum/#!forum/php-rdkafka).\n\n## Stubs\n\nBecause your IDE is not able to auto discover php-rdkadka api you can consider usage of external package providing a set of stubs for php-rdkafka classes, functions and constants: [kwn/php-rdkafka-stubs](https://github.com/kwn/php-rdkafka-stubs)\n\n## Contributing\n\nIf you would like to contribute, thank you :)\n\nBefore you start, please take a look at the [CONTRIBUTING document](https://github.com/arnaud-lb/php-rdkafka/blob/master/CONTRIBUTING.md) to see how to get your changes merged in.\n\n## Credits\n\nDocumentation copied from [librdkafka](https://github.com/edenhill/librdkafka).\n\nAuthors: see [contributors](https://github.com/arnaud-lb/php-rdkafka/graphs/contributors).\n\n## License\n\nphp-rdkafka is released under the [MIT](https://github.com/arnaud-lb/php-rdkafka/blob/master/LICENSE) license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farnaud-lb%2Fphp-rdkafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farnaud-lb%2Fphp-rdkafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farnaud-lb%2Fphp-rdkafka/lists"}