Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sha3sha3/UE-EasyKafka
EasyKafka is a Kafka/Redpanda client sub-system for unreal engine. It supports producing and consuming records through blueprint and C++.
https://github.com/sha3sha3/UE-EasyKafka
cloud kafka microservice microservices networking redpanda unreal unreal-engine unreal-engine-4 unreal-engine-5 unrealengine
Last synced: 2 months ago
JSON representation
EasyKafka is a Kafka/Redpanda client sub-system for unreal engine. It supports producing and consuming records through blueprint and C++.
- Host: GitHub
- URL: https://github.com/sha3sha3/UE-EasyKafka
- Owner: sha3sha3
- License: other
- Created: 2022-12-11T01:05:08.000Z (almost 2 years ago)
- Default Branch: master
- Last Pushed: 2024-02-20T11:02:58.000Z (8 months ago)
- Last Synced: 2024-07-10T16:02:38.779Z (3 months ago)
- Topics: cloud, kafka, microservice, microservices, networking, redpanda, unreal, unreal-engine, unreal-engine-4, unreal-engine-5, unrealengine
- Language: C++
- Homepage: https://www.unrealengine.com/marketplace/en-US/product/client-for-kafka
- Size: 23.8 MB
- Stars: 32
- Watchers: 2
- Forks: 7
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
- awesome-unreal - UE-EasyKafka - EasyKafka is a Kafka/Redpanda client sub-system for unreal engine. It supports producing and consuming records through blueprint and C++. (Networking)
README
# This project is no longer maintained for some platforms on GitHub, but you may obtain the latest maintained version on [Unreal Marketplace](https://www.unrealengine.com/marketplace/en-US/product/client-for-kafka).
Kafka Client for Unreal Engine 4/5
# EasyKafka subsystem
EasyKafka is a Kafka/Redpanda client sub-system for unreal engine. It supports producing and consuming records through blueprint and C++.
# Supported Platforms
- Windows x86_64
- Hololens 2 (Windows ARM64)
- Linux x86_64
- Linux ARM64
# C++ Modules Link
Link the plugin modules to your project through `.build.cs`:
```cs
CppStandard = CppStandardVersion.Cpp17;//avoid using boost
if(Target.Platform == UnrealTargetPlatform.HoloLens || Target.Platform == UnrealTargetPlatform.Win64)
bUseRTTI = true;
PrivateDependencyModuleNames.AddRange( new string[]
{
"EasyKafka",
"KafkaLib",
"KafkaConsumer",
"KafkaProducer",
"KafkaAdmin"
});
```
# Kafka Consumer Basic Usage
**PAY ATTENTION TO THE BLOCKING METHODS.**
## C++
Create Consumer with default configuration:
```cpp
#include "EasyKafkaSubsystem.h"
TSharedPtr EasyKafka = GEngine->GetEngineSubsystem()->GetEasyKafka();
EasyKafka->GetConsumer()->CreateConsumer(``, ``, ``, (int)EKafkaLogLevel::ERR);
```
Create Consumer with configuration:
```cpp
#include "EasyKafkaSubsystem.h"
TSharedPtr EasyKafka = GEngine->GetEngineSubsystem()->GetEasyKafka();
TMap KafkaConfiguration =
{
{EKafkaConsumerConfig::CLIENT_ID,"34235"},
{EKafkaConsumerConfig::SOCKET_TIMEOUT_MS,"10000"}
};
EasyKafka->GetConsumer()->CreateConsumer(``, ``, ``, KafkaConfiguration, (int)EKafkaLogLevel::ERR);
```
Consume messages:
```cpp
EasyKafka->GetConsumer()->OnNewMessage().AddLambda([](const TArray& Messages)
{
for (FConsumerRecord Message : Messages)
{
UE_LOG(LogTemp, Display, TEXT("New Message %s \n"), *Message.Value);//process messages
}
});
EasyKafka->GetConsumer()->Subscribe(
{
"topic",
"topic1",
"topic2"
});
EasyKafka->GetConsumer()->StartConsuming();
```
**ATTENTION: MAKE SURE TO COMMIT FROM THE CONSUMER RUNNABLE THREAD BEFORE PROCESSING RECORDS IF YOU DISABLED AUTOCOMMIT.**
## Blueprint
# Kafka Producer Basic Usage
**PAY ATTENTION TO THE BLOCKING METHODS.**
## C++
Create Producer with default configuration:
```cpp
#include "EasyKafkaSubsystem.h"
TSharedPtr EasyKafka = GEngine->GetEngineSubsystem()->GetEasyKafka();
EasyKafka->GetProducer()->CreateProducer(``, ``, ``, (int)EKafkaLogLevel::ERR);
```
Create Producer with configuration:
```cpp
#include "EasyKafkaSubsystem.h"
TSharedPtr EasyKafka = GEngine->GetEngineSubsystem()->GetEasyKafka();
TMap KafkaConfiguration =
{
{EKafkaProducerConfig::MESSAGE_TIMEOUT_MS,"5000"},
{EKafkaProducerConfig::REQUEST_TIMEOUT_MS,"5000"}
};
EasyKafka->GetProducer()->CreateProducer(``, ``, ``, KafkaConfiguration, (int)EKafkaLogLevel::ERR);
```
on record produced/failed to produce callback
```cpp
EasyKafka->GetProducer()->OnProduce().AddLambda([](const FProducerCallback& Callback)
{
if (Callback.bError)
{
UE_LOG(LogTemp, Error, TEXT("Error producing recordId: %d \nError Message: %s\n"), Callback.RecordMetadata.RecordId, *Callback.ErrorMessage);
}
else
{
UE_LOG(LogTemp, Display, TEXT("RecordId: %d produced.\n"), Callback.RecordMetadata.RecordId);
}
});
```
produce record async
```cpp
EasyKafka->GetProducer()->ProduceRecord('', '<"RECORD_VALUE>');
/*
More control over your record
Such as headers,Id...
*/
FProducerRecord record;
record.Key = "key";
record.Topic = "topic";
record.Value = "value";
record.Id = 2312;//Unique id to identify this record OnProduce callback;
record.Headers = FRecordHeader(
{
{"KeyOne","ValueOne"},
{"KeyTwo","ValueTwo"}
});
EasyKafka->GetProducer()->ProduceRecord(record);
```
## Blueprint
# Kafka Admin Basic Usage
**ALL THE METHODS ARE BLOCKING, ASYNC TO BE ADDED.**
## C++
Create Admin with default configuration:
```cpp
#include "EasyKafkaSubsystem.h"
TSharedPtr EasyKafka = GEngine->GetEngineSubsystem()->GetEasyKafka();
EasyKafka->GetAdmin()->CreateAdmin(``, ``, ``, (int)EKafkaLogLevel::ERR);
```
Create Admin with configuration:
```cpp
#include "EasyKafkaSubsystem.h"
TSharedPtr EasyKafka = GEngine->GetEngineSubsystem()->GetEasyKafka();
TMap KafkaConfiguration =
{
{EKafkaAdminConfig::SOCKET_TIMEOUT_MS,"10000"}
};
EasyKafka->GetAdmin()->CreateAdmin(``, ``, ``, KafkaConfiguration, (int)EKafkaLogLevel::ERR);
```
Simple Admin request example:
```cpp
const TArray TopicsToDelete = { "Topic1Name", "Topic2Name" };
FAdminRequestResult Result = EasyKafka->GetAdmin()->DeleteTopics(TopicsToDelete);
if (Result.bError)
{
UE_LOG(LogTemp, Error, TEXT("Error deleting topics: %s\n"), *Result.ErrorMessage);
}
```
## Blueprint
## Find it helpful?
Give us a ⭐️!