{"id":20038062,"url":"https://github.com/perfectlysoft/perfect-kafka","last_synced_at":"2026-03-07T16:31:32.328Z","repository":{"id":57756383,"uuid":"83479750","full_name":"PerfectlySoft/Perfect-Kafka","owner":"PerfectlySoft","description":"An Express Swift Client of Apache Kafka 0.8, the Stream Processing Platform","archived":false,"fork":false,"pushed_at":"2018-03-17T21:54:44.000Z","size":53,"stargazers_count":24,"open_issues_count":1,"forks_count":7,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-25T12:01:56.725Z","etag":null,"topics":["kafka","message-queue","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Swift","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PerfectlySoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-28T21:13:53.000Z","updated_at":"2023-12-26T16:55:05.000Z","dependencies_parsed_at":"2022-11-29T13:20:35.044Z","dependency_job_id":null,"html_url":"https://github.com/PerfectlySoft/Perfect-Kafka","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/PerfectlySoft/Perfect-Kafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerfectlySoft%2FPerfect-Kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerfectlySoft%2FPerfect-Kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerfectlySoft%2FPerfect-Kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerfectlySoft%2FPerfect-Kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PerfectlySoft","download_url":"https://codeload.github.com/PerfectlySoft/Perfect-Kafka/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerfectlySoft%2FPerfect-Kafka/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30221506,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T14:02:48.375Z","status":"ssl_error","status_checked_at":"2026-03-07T14:02:43.192Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","message-queue","stream-processing"],"created_at":"2024-11-13T10:25:16.861Z","updated_at":"2026-03-07T16:31:32.293Z","avatar_url":"https://github.com/PerfectlySoft.png","language":"Swift","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Perfect-Kafka [简体中文](README.zh_CN.md)\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"http://perfect.org/get-involved.html\" target=\"_blank\"\u003e\n        \u003cimg src=\"http://perfect.org/assets/github/perfect_github_2_0_0.jpg\" alt=\"Get Involed with Perfect!\" width=\"854\" /\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/PerfectlySoft/Perfect\" target=\"_blank\"\u003e\n        \u003cimg src=\"http://www.perfect.org/github/Perfect_GH_button_1_Star.jpg\" alt=\"Star Perfect On Github\" /\u003e\n    \u003c/a\u003e  \n    \u003ca href=\"http://stackoverflow.com/questions/tagged/perfect\" target=\"_blank\"\u003e\n        \u003cimg src=\"http://www.perfect.org/github/perfect_gh_button_2_SO.jpg\" alt=\"Stack Overflow\" /\u003e\n    \u003c/a\u003e  \n    \u003ca href=\"https://twitter.com/perfectlysoft\" target=\"_blank\"\u003e\n        \u003cimg src=\"http://www.perfect.org/github/Perfect_GH_button_3_twit.jpg\" alt=\"Follow Perfect on Twitter\" /\u003e\n    \u003c/a\u003e  \n    \u003ca href=\"http://perfect.ly\" target=\"_blank\"\u003e\n        \u003cimg src=\"http://www.perfect.org/github/Perfect_GH_button_4_slack.jpg\" alt=\"Join the Perfect Slack\" /\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://developer.apple.com/swift/\" target=\"_blank\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Swift-3.0-orange.svg?style=flat\" alt=\"Swift 3.0\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://developer.apple.com/swift/\" target=\"_blank\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Platforms-OS%20X%20%7C%20Linux%20-lightgray.svg?style=flat\" alt=\"Platforms OS X | Linux\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"http://perfect.org/licensing.html\" target=\"_blank\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/License-Apache-lightgrey.svg?style=flat\" alt=\"License Apache\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"http://twitter.com/PerfectlySoft\" target=\"_blank\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Twitter-@PerfectlySoft-blue.svg?style=flat\" alt=\"PerfectlySoft Twitter\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"http://perfect.ly\" target=\"_blank\"\u003e\n        \u003cimg src=\"http://perfect.ly/badge.svg\" alt=\"Slack Status\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\n\nThis project provides an express Swift wrapper of librdkafka.\n\nThis package builds with Swift Package Manager and is part of the [Perfect](https://github.com/PerfectlySoft/Perfect) project but can also be used as an independent module.\n\n## Release Notes for MacOS X\n\nBefore importing this library, please install librdkafka first:\n\n```\n$ brew install librdkafka\n```\n\nPlease also note that a proper pkg-config path setting is required:\n\n```\n$ export PKG_CONFIG_PATH=\"/usr/local/lib/pkgconfig\"\n```\n\n## Release Notes for Linux\n\nBefore importing this library, please install librdkafka-dev first:\n\n```\n$ sudo apt-get install librdkafka-dev\n```\n\n## Quick Start\n\n### Kafka Client Configurations\n\nBefore starting any stream operations, it is necessary to apply settings to clients, i.e., producers or consumers.\n\nPerfect Kafka provides two different categories of configuration, i.e. `Kafka.Config()` for global configurations and `Kafka.TopicConfig()` for topic configurations.\n\n#### Initialization of Global Configurations\n\nTo create a configuration set with default value settings, simple call:\n\n``` swift\nlet conf = try Kafka.Config()\n```\n\nor, if another configuration based on an existing one can be also duplicated in such a form:\n\n``` swift\nlet conf = try Kafka.Config()\n// this will keep the original settings and duplicate a new one\nlet conf2 = try Kafka.Config(conf)\n```\n\n#### Initialization of Topic Configurations\n\nTopic configuration shares the same initialization fashion with global configuration.\n\nTo create a topic configuration with default settings, call:\n\n``` swift\nlet conf = try Kafka.TopicConfig()\n```\n\nor, if another configuration based on an existing one can be also duplicated in such a form:\n\n``` swift\nlet conf = try Kafka.TopicConfig()\n// this will keep the original settings and duplicate a new one\nlet conf2 = try Kafka.TopicConfig(conf)\n```\n\n### Access Settings of Configuration\n\nBoth `Kafka.Config` and `Kafka.TopicConfig` have the same api of accessing settings.\n\n#### List All Variables with Value\n\n`Kafka.Config.properties` and `Kafka.TopicConfig.properties` provides dictionary type settings:\n\n``` swift\n// this will print out all variables in a configuration\nprint(conf.properties)\n// for example, it will print out something like:\n// [\"topic.metadata.refresh.fast.interval.ms\": \"250\",\n// \"receive.message.max.bytes\": \"100000000\", ...]\n```\n\n#### Get a Variable Value\n\nCall `get()` to retrieve the value from a specific variable:\n\n``` swift\nlet maxBytes = try conf.get(\"receive.message.max.bytes\")\n// maxBytes would be \"100000000\" by default\n```\n\n#### Set a Variable with New Value\n\nCall `set()` to save settings for a specific variable:\n\n``` swift\n// this will restrict message receiving buffer to 1MB\ntry conf.set(\"receive.message.max.bytes\", \"1048576\")\n```\n\n### Producer\n\nPerfect-Kafka provides a Producer class to send data / message to Kafka hosts. Producer can send a message one at a time, or sent multiple messages in a batch. Messages can be either text string or binary bytes.\n\n``` swift\nlet producer = try Producer(\"VideoTest\")\nlet brokers = producer.connect(brokers: \"host:9092\")\nif brokers \u003e 0 {\n  let _ = try producer.send(message: \"hello, world!\")\n}\n```\n\nBefore sending any actual messages, a few steps are required to setup the connection to Kafka hosts.\n\n#### Producer Instance with a Topic\n\nTo initialize a Producer instance, a topic name is required no matter whether this topic exists in the Kafka hosts or not.\n\nIf the topic didn't exist when connected to Kafka hosts / brokers, `Producer()` would try to create a new one; Otherwise it would use the existing topic for further operations.\n\nFor example, the demo below shows how to start a producer with a topic named \"VideoTest\":\n\n``` swift\nlet producer = try Producer(\"VideoTest\")\n```\n\n#### Connect to Brokers\n\nUse method `connect()` to connect to one or more message brokers, i.e., Kafka hosts ( host and port ):\n\n``` swift\nlet brokers = producer.connect(brokers: \"host1:9092,host2:9092,host3:9092\")\n```\n\nIf success, it will return the number of hosts that connected.\n\nAlternatively, it is also possible to connect to brokers by different parameter fashions, take example, hosts can be an array of string:\n\n``` swift\nlet brokers = producer.connect(brokers: [\"host1:9092\", \"host2:9092\", \"host3:9092\"])\n```\n\nor dictionary:\n``` swift\nlet brokers = producer.connect(brokers: [\"host1\": 9092, \"host2\": 9092, \"host3\": 9092])\n```\n\n#### Send Messages\n\nPerfect Kafka allows to send either text or binary messages to brokers one at a time or in a batch.\n\nMethod|Description|Returns\n------|-----------|-------\n`send(message: String, key: String? = nil)`|a text message with an optional key to send|an Int64 message id\n`send(message: [Int8], key: [Int8] = [])`|a binary message with an optional key to send|an Int64 message id\n`send(messages: [(String, String?)])`|text messages with optional keys in an array|[Int64] message IDs for each message\n`send(messages: [([Int8], [Int8])])`|binary messages with optional keys in an array|[Int64] message IDs for each message\n\n#### Sent or Not\n\nPerfect Kafka `send()` is asynchronous function so the library provides a few extra methods to determine the sending status of each message.\n\n- `OnSent()` callback. If set properly, each message will call this event once actually sent. For example: `producer.OnSent = { print(\"msg #\\($0) was sent\") }`. The only parameter of this event is the Int64 message id returned by `send()`.\n\n- `producer.outbox` is an [Int64] array to indicate the messages in sending queue. *NOTE* As a high performant streaming platform, the existence of messages in outbox doesn't mean that they were failed to send, so don't try to resend these message unless it was explicitly confirmed that they were failed to send.\n\n- `OnError()` callback. Producer will call this event if something wrong, e.g., `producer.OnError = { print(\"error: \\($0)\") }` will print out the error message if happen.\n\n- `flush(_ seconds: Int)` method can help wait seconds for clearing the message queue and flushing the outbox.\n\n### Consumer\n\nBefore actually receiving messages from Kafka with a specific topic, a few procedures are required to initialize a Consumer instance:\n\n``` swift\nlet consumer = try Consumer(\"VideoTest\")\nlet brokers = consumer.connect(brokers: [\"host1\": 9092, \"host2\": 9092, \"host3\": 9092])\nguard brokers \u003e 0 else {\n  // connection failed\n}//end guard\n```\n\n#### Partitions\n\nOnce connected, it is a good idea to get the information from the brokers to see if there are sufficient resources, i.e., partitions, for further operations:\n\n``` swift\nlet info = try consumer.brokerInfo()\nprint(info)\n```\n\nThe above variable `info` is a `MetaData` structure as reference below:\n\nMember|Type|Description\n------|----|-----------\nbrokers|[Broker]|An array of Broker structure\ntopics|[Topic]|An array of Topic structure\n\nStructure `Broker` stores the information of a broker:\n\nMember|Type|Description\n------|----|-----------\nid|Int|Broker Id\nhost|String|Host name of the broker\nport|Int|Host port that listens\n\nThe major content of `Topic` structure is to record how many partitions are using in such a topic:\n\nMember|Type|Description\n------|----|-----------\nname|String|Topic name\nerr|Exception|Topic error reported by broker\npartitions|[Partition]|Partitions of this topic\n\nData structure `Partition` is vitally important to indicate the partition id for messaging:\n\nMember|Type|Description\n------|----|-----------\nid|Int|Partition Id - use this to start / stop messaging\nerr|Exception|Partition error reported by broker\nleader|Int|Leader broker\nreplicas|[Int]|Replica brokers\nisrs|[Int]|In-Sync-Replica brokers\n\nPractically, partition info could be acquired by way below:\n\n``` swift\nlet consumer = try Consumer(\"VideoTest\")\nlet brokers = consumer.connect(brokers: [\"host1\": 9092, \"host2\": 9092, \"host3\": 9092])\nguard brokers \u003e 0 else {\n  // connection failed\n}//end guard\nconsumer.OnArrival = { m in print(\"message : #\\(m.offset) \\(m.text)\")}\nlet info = try consumer.brokerInfo()\nguard info.topics.count \u003e 0 else {\n  // no topic found\n}//end guard\nguard info.topics[0].name == \"VideoTest\" else {\n  // it is not the topic we want\n}//end guar\nlet partitions = info.topics[0].partitions\n```\n\n#### Download Messages From A Partition\n\nCode below shows how to download messages from a partition. In this demo, we assume `let partId = partitions[0].id`:\n\n``` swift\nconsumer.OnArrival = { m in\n  print(\"message #\\(m.offset) : \\(m.text)\")\n}//end event\n\n// start downloading\ntry consumer.start(partition: partId)\n// run until end of program\nwhile(notEndOfProgram) {\n  let total = try consumer.poll(partition: partId)\n  print(\"\\(total) messages arrived in this moment\")\n}//end while\nconsumer.stop(partId)\n```\n\nNow we take a walk through:\n\nFirstly, `OnArrival()` event is a callback with a `Message` data structure:\n\nMember|Type|Description\n------|----|-----------\nerr|Exception|Error: if the message is good or not\ntopic|String|topic name of the message\npartition|Int|partition of the message\nisText|Bool|if the message is a valid UTF-8 text or not\ndata|[Int8]|the original binary data of message body\ntext|String|decoded message body in a UTF-8 string, if `isText`\nkeyIsText|Bool|if the key is a valid UTF-8 text\nkeybuf|[Int8]|the original binary data of optional key\nkey|String|decoded key in a UTF-8 string, if `keyIsText`\noffset|Int64|offset inside the topic\n\nSecondly, call `start()` to start download messages: `func start(_ from: Position = .BEGIN, partition: Int32 = RD_KAFKA_PARTITION_UA)`, here are the parameter details:\n- from: Position, from which position of the messages in the partition to download. Valid value can be `.BEGIN` to indicate downloading every messages from the very beginning, or `.END` to download the most reason one, or `.STORED` to download the previous stored messages in case of failure, or `.SPECIFY(Int64)` to start downloading from a specific location. *NOTE* use `func store(_ offset: Int64, partition: Int32 = RD_KAFKA_PARTITION_UA)` to store a specific message if `.STORED` is needed.\n- partition: Int32, the partition id.\n\nThen Perfect Kafka provides the `poll()` function to wait a short while to listen the activity of a specific partition:\n`func poll(_ timeout: UInt = 10, partition: Int32 = RD_KAFKA_PARTITION_UA)`. The `timeout` is the milliseconds to wait for polling.\n\nFinally, call `stop()` to end the messaging.\n\n## Further Information\nFor more information on the Perfect project, please visit [perfect.org](http://perfect.org).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperfectlysoft%2Fperfect-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperfectlysoft%2Fperfect-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperfectlysoft%2Fperfect-kafka/lists"}