{"id":19196458,"url":"https://github.com/jet/fskafka","last_synced_at":"2025-04-07T06:12:31.013Z","repository":{"id":39995749,"uuid":"133712388","full_name":"jet/FsKafka","owner":"jet","description":"Minimal F# wrappers for Confluent.Kafka+librdkafka.redist 1.x","archived":false,"fork":false,"pushed_at":"2022-09-21T09:08:43.000Z","size":478,"stargazers_count":89,"open_issues_count":1,"forks_count":17,"subscribers_count":37,"default_branch":"master","last_synced_at":"2025-03-31T05:05:23.651Z","etag":null,"topics":["dotnet","fsharp","kafka","propulsion"],"latest_commit_sha":null,"homepage":"https://github.com/jet/dotnet-templates","language":"F#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jet.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null}},"created_at":"2018-05-16T19:13:41.000Z","updated_at":"2025-01-25T22:00:24.000Z","dependencies_parsed_at":"2023-01-17T18:46:04.127Z","dependency_job_id":null,"html_url":"https://github.com/jet/FsKafka","commit_stats":null,"previous_names":["jet/jet.confluentkafka.fsharp"],"tags_count":51,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jet%2FFsKafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jet%2FFsKafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jet%2FFsKafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jet%2FFsKafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jet","download_url":"https://codeload.github.com/jet/FsKafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247601449,"owners_count":20964864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dotnet","fsharp","kafka","propulsion"],"created_at":"2024-11-09T12:13:44.941Z","updated_at":"2025-04-07T06:12:30.991Z","avatar_url":"https://github.com/jet.png","language":"F#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FsKafka [![Build Status](https://dev.azure.com/jet-opensource/opensource/_apis/build/status/jet.FsKafka)](https://dev.azure.com/jet-opensource/opensource/_build/latest?definitionId=7) [![release](https://img.shields.io/github/release/jet/FsKafka.svg)](https://github.com/jet/FsKafka/releases) [![NuGet](https://img.shields.io/nuget/v/FsKafka.svg?logo=nuget)](https://www.nuget.org/packages/FsKafka/) [![license](https://img.shields.io/github/license/jet/FsKafka.svg)](LICENSE) ![code size](https://img.shields.io/github/languages/code-size/jet/FsKafka.svg)\n\nF# friendly wrapper for `Confluent.Kafka`, with minimal dependencies or additional abstractions (but see [related repos](#related-repos)). \n\n`FsKafka` wraps `Confluent.Kafka` to provide efficient batched Kafka Producer and Consumer configurations with basic logging instrumentation. [Depends](https://www.fuget.org/packages/FsKafka) on \n[`Confluent.Kafka [1.9.2]`](https://github.com/confluentinc/confluent-kafka-dotnet/blob/master/CHANGELOG.md#192),\n[`librdkafka.redist [1.9.2]`](https://github.com/edenhill/librdkafka/releases/tag/v1.9.2) (pinned to ensure we use a tested pairing),\n`Serilog` (but no specific Serilog sinks, i.e. you configure to emit to `NLog` etc) and\n`Newtonsoft.Json` (used internally to parse Broker-provided Statistics for logging purposes).\n\n## Usage\n\nFsKafka is delivered as a [Nuget package](https://www.nuget.org/packages/FsKafka/) targeting `netstandard2.0` and F# \u003e= 4.5.\n\n```bash\ndotnet add package FsKafka\n```\n\nor for `paket`, use:\n\n```bash\npaket add FsKafka\n```\n\n## Related repos\n\n- See [the Propulsion repo](https://github.com/jet/propulsion) for extended Producers and Consumers.\n- See [the Jet `dotnet new` templates repo](https://github.com/jet/dotnet-templates)'s `proProjector` template (in `-k` mode) for example producer logic using the `BatchedProducer` and the `proConsumer` template for examples of using the `BatchedConsumer` from `FsKafka`, alongside the extended modes in `Propulsion`.\n- See [the Equinox QuickStart](https://github.com/jet/equinox#quickstart) for examples of using this library to project to Kafka from `Equinox.Cosmos` and/or `Equinox.EventStore`.\n\n## CONTRIBUTING\n\nContributions of all sizes are warmly welcomed. See [our contribution guide](CONTRIBUTING.md)\n\n## TEMPLATES\n\nThe best place to start, sample-wise is from the `dotnet new` templates stored [in a dedicated repo](https://github.com/jet/dotnet-templates).\n\n## BUILDING\n\nThe [templates](#templates) are the best way to see how to consume it; these instructions are intended mainly for people looking to make changes.\n\nNB The tests are reliant on a `TEST_KAFKA_BROKER` environment variable pointing to a Broker that has been configured to auto-create ephemeral Kafka Topics as required by the tests (each test run writes to a guid-named topic)\n\n### build, including tests on netcoreapp3.1\n\n```powershell\nexport TEST_KAFKA_BROKER=\"\u003cserver\u003e:9092\"\ndotnet build build.proj -v n\n```\n\n## FAQ\n\n### How do I get rid of all ~~~~the `breaking off polling` ... `resuming polling` spam?\n\n- The `BatchedConsumer` implementation tries to give clear feedback as to when reading is not keeping up, for diagnostic purposes. As of [#32](https://github.com/jet/FsKafka/pull/32), such messages are tagged with the type `FsKafka.Core.InFlightMessageCounter`, and as such can be silenced by including the following in one's `LoggerConfiguration()`: \n\n    `.MinimumLevel.Override(FsKafka.Core.Constants.messageCounterSourceContext, Serilog.Events.LogEventLevel.Warning)`\n\n### What is this, why does it exist, where did it come from, is anyone using it ?\n\nThis code results from building out an end-to-end batteries-included set of libraries and templates as part of the [Equinox](https://github.com/jet/equinox) project.\n\nEquinox places some key constraints on all components and dependencies:-\n\n- batteries-included examples of end-to-end functionality within the Equinox remit; _samples should have clean consistent wiring_\n- pick a well-established base library, try not to add new concepts\n- low dependencies, so it can work in lots of contexts without egregiously forcing you to upgrade things\n- aim to add any resilience features as patches to upstream repos\n- thorough test coverage; integration coverage for core wrapped functionality, unit tests for any non-trivial logic in the wrapper library \n\n## Minimal producer example\n\n```fsharp\n#r \"nuget:FsKafka\"\nopen Confluent.Kafka\nopen FsKafka\n\nlet log = Serilog.LoggerConfiguration().CreateLogger()\n\nlet batching = Batching.Linger (System.TimeSpan.FromMilliseconds 10.)\nlet producerConfig = KafkaProducerConfig.Create(\"MyClientId\", \"kafka:9092\", Acks.All, batching)\nlet producer = KafkaProducer.Create(log, producerConfig, \"MyTopic\")\n   \nlet key = Guid.NewGuid().ToString()\nlet deliveryResult = producer.ProduceAsync(key, \"Hello World!\") |\u003e Async.RunSynchronously\n```\n\n## Minimal batched consumer example\n\n```fsharp\n#r \"nuget:FsKafka\"\nopen Confluent.Kafka\nopen FsKafka\n\nlet log = Serilog.LoggerConfiguration().CreateLogger()\n\nlet handler (messages : ConsumeResult\u003cstring,string\u003e []) = async {\n    for m in messages do\n        printfn \"Received: %s\" m.Message.Value\n} \n\nlet cfg = KafkaConsumerConfig.Create(\"MyClientId\", \"kafka:9092\", [\"MyTopic\"], \"MyGroupId\", AutoOffsetReset.Earliest)\n\nasync {\n    use consumer = BatchedConsumer.Start(log, cfg, handler)\n    return! consumer.AwaitShutdown()\n} |\u003e Async.RunSynchronously\n```\n\n## Minimal batched consumer example with monitor\n\n```fsharp\n#r \"nuget:FsKafka\"\nopen Confluent.Kafka\nopen FsKafka\n\nlet log = Serilog.LoggerConfiguration().CreateLogger()\n\nlet handler (messages : ConsumeResult\u003cstring,string\u003e []) = async {\n    for m in messages do\n        printfn \"Received: %s\" m.Message.Value\n} \n\nlet cfg = KafkaConsumerConfig.Create(\"MyClientId\", \"kafka:9092\", [\"MyTopic\"], \"MyGroupId\", AutoOffsetReset.Earliest)\n\nasync {\n    use consumer = BatchedConsumer.Start(log, cfg, handler)\n    use _ = KafkaMonitor(log).Start(consumer.Inner, cfg.Inner.GroupId)\n    return! consumer.AwaitShutdown()\n} |\u003e Async.RunSynchronously\n```\n\n## Running (and awaiting) a pair of consumers until either throws\n\n```fsharp\n#r \"nuget:FsKafka\"\nopen Confluent.Kafka\nopen FsKafka\n\nlet log = Serilog.LoggerConfiguration().CreateLogger()\n\nlet handler (messages : ConsumeResult\u003cstring,string\u003e []) = async {\n    for m in messages do\n        printfn \"Received: %s\" m.Message.Value\n} \n\nlet config topic = KafkaConsumerConfig.Create(\"MyClientId\", \"kafka:9092\", [topic], \"MyGroupId\", AutoOffsetReset.Earliest)\n\nlet cfg1, cfg2 = config \"MyTopicA\", config \"MyTopicB\"\n\nasync {\n    use consumer1 = BatchedConsumer.Start(log, cfg1, handler)\n    use consumer2 = BatchedConsumer.Start(log, cfg2, handler)\n    use _ = KafkaMonitor(log).Start(consumer1.Inner, cfg1.Inner.GroupId)\n    use _ = KafkaMonitor(log).Start(consumer2.Inner, cfg2.Inner.GroupId)\n    return! Async.Parallel [consumer1.AwaitWithStopOnCancellation(); consumer2.AwaitWithStopOnCancellation()]\n} |\u003e Async.RunSynchronously\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjet%2Ffskafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjet%2Ffskafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjet%2Ffskafka/lists"}