{"id":18007295,"url":"https://github.com/turtlesoupy/haskakafka","last_synced_at":"2025-12-11T23:14:53.305Z","repository":{"id":17476764,"uuid":"20251050","full_name":"turtlesoupy/haskakafka","owner":"turtlesoupy","description":"Kafka bindings for Haskell","archived":false,"fork":false,"pushed_at":"2017-10-04T23:48:45.000Z","size":121,"stargazers_count":72,"open_issues_count":3,"forks_count":14,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-10-30T02:42:34.280Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/turtlesoupy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-05-28T08:54:55.000Z","updated_at":"2022-06-14T19:46:08.000Z","dependencies_parsed_at":"2022-07-26T17:02:15.795Z","dependency_job_id":null,"html_url":"https://github.com/turtlesoupy/haskakafka","commit_stats":null,"previous_names":["cosbynator/haskakafka"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/turtlesoupy%2Fhaskakafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/turtlesoupy%2Fhaskakafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/turtlesoupy%2Fhaskakafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/turtlesoupy%2Fhaskakafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/turtlesoupy","download_url":"https://codeload.github.com/turtlesoupy/haskakafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245654031,"owners_count":20650796,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-30T01:12:55.989Z","updated_at":"2025-12-11T23:14:48.257Z","avatar_url":"https://github.com/turtlesoupy.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Haskakafka\n\nKafka bindings for Haskell backed by the\n[librdkafka C module](https://github.com/edenhill/librdkafka). It has been tested and fully\nsupports Kafka 0.9.0.1 using librdkafka 0.9.0.99 and higher on Linux and OS X. Haskakafka supports\nboth producers and consumers with optional batch operations.\n\nHackage: http://hackage.haskell.org/package/haskakafka\n\n# Usage\nA quick walkthrough of producers and consumers:\n```Haskell\nimport Haskakafka\n\nimport qualified Data.ByteString.Char8 as C8\n\nexample :: IO ()\nexample = do\n  let\n      -- Optionally, we can configure certain parameters for Kafka\n      kafkaConfig = [(\"socket.timeout.ms\", \"50000\")]\n      topicConfig = [(\"request.timeout.ms\", \"50000\")]\n\n      -- Payloads are just ByteStrings\n      samplePayload = C8.pack \"Hello world\"\n\n\n  -- withKafkaProducer opens a producer connection and gives us\n  -- two objects for subsequent use.\n  withKafkaProducer kafkaConfig topicConfig\n                    \"localhost:9092\" \"test_topic\"\n                    $ \\kafka topic -\u003e do\n\n    -- Produce a single unkeyed message to partition 0\n    let message = KafkaProduceMessage samplePayload\n    _ \u003c- produceMessage topic (KafkaSpecifiedPartition 0) message\n\n    -- Produce a single keyed message\n    let keyMessage = KafkaProduceKeyedMessage (C8.pack \"Key\") samplePayload\n    _ \u003c- produceKeyedMessage topic keyMessage\n\n    -- We can also use the batch API for better performance\n    _ \u003c- produceMessageBatch topic KafkaUnassignedPartition [message, keyMessage]\n\n    putStrLn \"Done producing messages, here was our config: \"\n    dumpConfFromKafka kafka \u003e\u003e= \\d -\u003e putStrLn $ \"Kafka config: \" ++ (show d)\n    dumpConfFromKafkaTopic topic \u003e\u003e= \\d -\u003e putStrLn $ \"Topic config: \" ++ (show d)\n\n\n  -- withKafkaConsumer opens a consumer connection and starts consuming\n  let partition = 0\n  withKafkaConsumer kafkaConfig topicConfig\n                    \"localhost:9092\" \"test_topic\"\n                    partition -- locked to a specific partition for each consumer\n                    KafkaOffsetBeginning -- start reading from beginning (alternatively, use\n                                         -- KafkaOffsetEnd, KafkaOffset or KafkaOffsetStored)\n                    $ \\kafka topic -\u003e do\n    -- Consume a single message at a time\n    let timeoutMs = 1000\n    me \u003c- consumeMessage topic partition timeoutMs\n    case me of\n      (Left err) -\u003e putStrLn $ \"Uh oh, an error! \" ++ (show err)\n      (Right m) -\u003e putStrLn $ \"Woo, payload was \" ++ (C8.unpack $ messagePayload m)\n\n    -- For better performance, consume in batches\n    let maxMessages = 10\n    mes \u003c- consumeMessageBatch topic partition timeoutMs maxMessages\n    case mes of\n      (Left err) -\u003e putStrLn $ \"Something went wrong in batch consume! \" ++ (show err)\n      (Right ms) -\u003e putStrLn $ \"Woohoo, we got \" ++ (show $ length ms) ++ \" messages\"\n\n\n    -- Be a little less noisy\n    setLogLevel kafka KafkaLogCrit\n\n  -- we can also fetch metadata about our Kafka infrastructure\n  let timeoutMs = 1000\n  emd \u003c- fetchBrokerMetadata [] \"localhost:9092\" timeoutMs\n  case emd of\n    (Left err) -\u003e putStrLn $ \"Uh oh, error time: \" ++ (show err)\n    (Right md) -\u003e putStrLn $ \"Kafka metadata: \" ++ (show md)\n```\n\n## Configuration Options\nConfiguration options are set in the call to `withKafkaConsumer` and `withKafkaProducer`. For\nthe full list of supported options, see\n[librdkafka's list](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md).\n\n# High Level Consumers\nHigh level consumers are supported by librdkafka starting from version 0.9.\nHigh-level consumers have the ability to handle more than one partition and even more than one topic.\nScalability and rebalancing are taken care of by librdkafka: once a new consumer in the same\nconsumer group is started the rebalance happens and all consumer share the load.\n\nThis version of Haskakafka adds (experimental) support for high-level consumers,\nhere is how such a consumer can be used in code:\n\n```Haskell\nimport           Haskakafka\nimport           Haskakafka.Consumer\n\nrunConsumerExample :: IO ()\nrunConsumerExample = do\n    res \u003c- runConsumer\n              (ConsumerGroupId \"test_group\")    -- group id is required\n              []                                -- extra kafka conf properties\n              (BrokersString \"localhost:9092\")  -- kafka brokers to connect to\n              [TopicName \"^hl-test*\"]           -- list of topics to consume, supporting regex\n              processMessages                   -- handler to consume messages\n    print $ show res\n\n-- this function is used inside consumer\n-- and it is responsible for polling and handling messages\n-- In this case I will do 10 polls and then return a success\nprocessMessages :: Kafka -\u003e IO (Either KafkaError ())\nprocessMessages kafka = do\n    mapM_ (\\_ -\u003e do\n                   msg1 \u003c- pollMessage kafka 1000\n                   print $ show msg1) [1..10]\n    return $ Right ()\n\n```\n\n# Installation\n\n## Installing librdkafka\n\nAlthough librdkafka is available on many platforms, most of\nthe distribution packages are too old to support haskakafka.\nAs such, we suggest you install from the source:\n\n    git clone https://github.com/edenhill/librdkafka\n    cd librdkafka\n    ./configure\n    make \u0026\u0026 sudo make install\n\nIf the C++ bindings fail for you, just install the C bindings alone.\n\n    cd librdkafka/src\n    make \u0026\u0026 sudo make install\n\nOn Debian and OS X, this will install the shared and static libraries to `/usr/local/lib`.\n\n## Installing Kafka\n\nThe full Kafka guide is at http://kafka.apache.org/documentation.html#quickstart\n\n## Installing Haskakafka\n\nIf you want to use cabal—since haskakafka uses `c2hs` to generate C bindings—you may need to\nexplicitly install `c2hs` somewhere on your path (i.e. outside of a sandbox).\nTo do so, run:\n\n    cabal install c2hs\n\nAfterwards installation should work, so go for\n\n    cabal install haskakafka\n\nThis uses the latest version of Haskakafka from [Hackage](http://hackage.haskell.org/package/haskakafka).\n\n# Testing\n\nHaskakafka ships with a suite of integration tests to verify the library against\na live Kafka instance. To get these setup you must have a broker running\non `localhost:9092` (or overwrite the `HASKAKAFKA_TEST_BROKER` environment variable)\nwith a `haskakafka_tests` topic created (or overwrite the `HASKAKAFKA_TEST_TOPIC`\nenvironment variable).\n\nTo get a broker running, download a [Kafka distribution](http://kafka.apache.org/downloads.html)\nand untar it into a directory. From there, run zookeeper using\n\n    bin/zookeeper-server-start.sh config/zookeeper.properties\n\nand run kafka in a separate window using\n\n    bin/kafka-server-start.sh config/server.properties\n\nWith both Kafka and Zookeeper running, you can run tests through stack:\n\n    stack test\n\nYou can also run tests through cabal:\n\n    cabal install --only-dependencies --enable-tests\n    cabal test --log=/dev/stdout\n\n# Running Examples\n\n    stack build\n    stack exec -- basic --help\n\n```\nbasic example [OPTIONS]\n  Fetch metadata, produce, and consume a message\n\nCommon flags:\n  -b       --brokers=\u003cbrokers\u003e  Comma separated list in format\n                                \u003chostname\u003e:\u003cport\u003e,\u003chostname\u003e:\u003cport\u003e\n  -t       --topic=\u003ctopic\u003e      Topic to fetch / produce\n  -C       --consumer           Consumer mode\n  -P       --producer           Producer mode\n  -L       --list               Metadata list mode\n  -A       --all                Run producer, consumer, and metadata list\n  -p=\u003cnum\u003e                      Partition (-1 for random partitioner when\n                                using producer)\n           --pretty             Pretty print output\n  -?       --help               Display help message\n  -V       --version            Print version information\n```\n\nThe following will produce 11 messages on partition 5 for topic `test_topic`:\n\n    stack exec -- basic -b \"broker1.example.com:9092,broker2.example.com:9092,broker3.example.com:9092\" -t test_topic -p 5 -P\n\nThe following will consume 11 messages on partition 5 for topic `test_topic`:\n\n    stack exec -- basic -b \"broker1.example.com:9092,broker2.example.com:9092,broker3.example.com:9092\" -t test_topic -p 5 -C\n\nThe following will pretty print a list of all brokers and topics:\n\n    stack exec -- basic -b \"broker1.example.com:9092,broker2.example.com:9092,broker3.example.com:9092\" -L --pretty\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fturtlesoupy%2Fhaskakafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fturtlesoupy%2Fhaskakafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fturtlesoupy%2Fhaskakafka/lists"}