https://github.com/kmcclellan/kafka-di
An extension of Confluent's Kafka client for use with Microsoft.Extensions.DependencyInjection (and friends).
https://github.com/kmcclellan/kafka-di
confluent-kafka dependency-injection ioc kafka logging
Last synced: 5 months ago
JSON representation
An extension of Confluent's Kafka client for use with Microsoft.Extensions.DependencyInjection (and friends).
- Host: GitHub
- URL: https://github.com/kmcclellan/kafka-di
- Owner: kmcclellan
- License: mit
- Created: 2020-06-13T00:51:50.000Z (about 6 years ago)
- Default Branch: main
- Last Pushed: 2026-01-10T15:40:50.000Z (5 months ago)
- Last Synced: 2026-01-14T07:14:18.967Z (5 months ago)
- Topics: confluent-kafka, dependency-injection, ioc, kafka, logging
- Language: C#
- Homepage: https://www.nuget.org/packages/Confluent.Kafka.DependencyInjection
- Size: 176 KB
- Stars: 15
- Watchers: 2
- Forks: 6
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Kafka Dependency Injection
An extension of [Confluent's Kafka client](https://github.com/confluentinc/confluent-kafka-dotnet) for use with `Microsoft.Extensions.DependencyInjection` (and friends).
### Features
* Configure/resolve Kafka clients using the service container.
* Load client config properties using `Microsoft.Extensions.Configuration`.
* Automatically log client events using `Microsoft.Extensions.Logging`.
* Extend a base hosted service to consume/process Kafka messages with `Microsoft.Extensions.Hosting`.
## Installation
Add the NuGet package to your project:
$ dotnet add package Confluent.Kafka.DependencyInjection
## Usage
### Resolving clients
Kafka DI works out-of-the-box after registering services with an `IServiceCollection`.
```c#
services.AddKafkaClient();
services.AddTransient();
```
Inject Kafka clients via constructor.
```c#
public MyService(IProducer producer, IConsumer consumer, IAdminClient adminClient)
{
// Clients are singletons managed by the container.
Producer = producer;
Consumer = consumer;
AdminClient = adminClient;
}
```
### Configuring clients
Client configuration properties are bound to the `Kafka` section of .NET configuration providers, such as `appsettings.json`.
```json
{
"Kafka": {
"Producer": {
"bootstrap.servers": "localhost:9092",
"transactional.id": "example"
},
"Consumer": {
"bootstrap.servers": "localhost:9092",
"group.id": "example"
},
"Admin": {
"bootstrap.servers": "localhost:9092"
}
}
}
```
You can also specify configuration properties using the options pattern.
```c#
// Prepare consumers for manual offset storage.
services.Configure(x => x.EnableAutoOffsetStore = false);
```
Configure serialization by registering the appropriate interface.
```c#
// "Open" generic registrations apply to all key/value types (except built-in types).
services.AddTransient(typeof(IAsyncDeserializer<>), typeof(JsonDeserializer<>));
// Configure schema registry (required by Confluent serializers).
services.AddSingleton(
x => new CachedSchemaRegistryClient(new SchemaRegistryConfig { Url = "localhost:8081" }));
```
For advanced scenarios, implement `IClientBuilderSetup` to customize clients further.
```c#
class MyClientSetup : IClientBuilderSetup
{
public void Apply(ProducerBuilder builder)
{
builder.SetStatisticsHandler(OnStatistics);
}
public void Apply(ConsumerBuilder builder)
{
builder.SetStatisticsHandler(OnStatistics);
}
public void Apply(AdminClientBuilder builder)
{
builder.SetStatisticsHandler(OnStatistics);
}
void OnStatistics(IClient client, string statistics)
{
Console.WriteLine($"New statistics available for {client.Name}");
}
}
```
Register custom setup with services.
```c#
services.AddTransient();
```
### Consumer hosting
Once the client is configured, implement `ConsumerService` to integrate with the .NET host.
```c#
class MyWorker(
IConsumer consumer,
IOptions options,
ILogger logger) :
ConsumerService(consumer, options, logger)
{
protected override ValueTask ProcessAsync(
ConsumeResult result,
CancellationToken cancellationToken)
{
// Process the message.
return ValueTask.CompletedTask;
}
}
```
Register the service with the container.
```c#
builder.Services.AddHostedService();
// Bind consumer hosting configuration.
var hostingOptions = builder.Services.AddOptions()
.BindConfiguration("Kafka:Hosting");
```
Hosted consumer features:
* Configurable parallelism while preserving order for each Kafka message key.
* Parallel-safe offset storage to achieve ["at least once" delivery semantics](https://docs.confluent.io/kafka/design/delivery-semantics.html#consumer-receipt).
```json
{
"Kafka": {
"Consumer": {
"bootstrap.servers": "localhost:9092",
"group.id": "example",
"enable.auto.offset.store": "false"
},
"Hosting": {
"Disabled": false,
"Subscription": [ "my-topic" ],
"MaxDegreeOfParallelism": 10,
"StoreProcessedOffsets": true
}
}
}
```