Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/jarrydk/quarkus-kafka-minecraft
Use Apache Kafka and Quarkus to track players in MineCraft
https://github.com/jarrydk/quarkus-kafka-minecraft
apache-kafka kafka minecraft minecraft-mod minecraft-server quarkus
Last synced: 6 days ago
JSON representation
Use Apache Kafka and Quarkus to track players in MineCraft
- Host: GitHub
- URL: https://github.com/jarrydk/quarkus-kafka-minecraft
- Owner: jarryDk
- License: apache-2.0
- Created: 2022-08-09T07:20:00.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-09-02T12:24:42.000Z (5 months ago)
- Last Synced: 2024-11-20T20:15:34.330Z (2 months ago)
- Topics: apache-kafka, kafka, minecraft, minecraft-mod, minecraft-server, quarkus
- Language: Java
- Homepage:
- Size: 2.66 MB
- Stars: 3
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.adoc
- License: LICENSE
Awesome Lists containing this project
README
= Use Apache Kafka and Quarkus to track players in Minecraft
This project is created to show how to track events including players movement in Minecraft via Apache Kafka and Quarkus.
.Full flow (Minecraft client, Minecraft server via Kafka and Quarkus to the browser)
image::images/quarkus-kafka-minecraft_diagram.png[]
.The page player.html in a browser
image::images/show_player_in_browser.png[]
.Part of the log when Minecraft spins up with the mod `kafkamod` installed
image::images/minecraft_server_start.png[]
The Minecraft mod `kafkamod` might be a bit to chatty, but it is nice doing development.
TIP: Start Kafka before starting the Minecraft server.
Inspiration: https://github.com/holly-cummins/quarkus-minecraft-observability-extension
Spinning up the web-app http://localhost:8080[Use Kafka to track players in MineCraft]
== Requeriments
- https://kafka.apache.org[Apache Kafka]
- https://files.minecraftforge.net/net/minecraftforge/forge/[Minecraft Forge]
.Getting er servers up running
- link:kafka/README.adoc[Get Kafka up running]
- link:minecraft/README.adoc[Get Minecraft up running]
== Minecraft Kafka mod
The `link:minecraft-kafka-mod/[kafkamod]` mod will connect to one or more Apache Kafka brokers on startup. It will create the topics if needed with configuration from `server.properties`.
.kafkamod setting in server.properties
- kafka-mod-player-event-enable | default: true
- kafka-mod-entity-event-enable | default: true
- kafka-mod-server-chat-event-enable | default: true
- kafka-mod-topic-num-partitions | default: 1
- kafka-mod-topic-replication-factor | default: 1
- kafka-mod-kafka-brokers | default: localhost:9092
WARNING: Security - we conect via `PLAINTEXT`
The `link:minecraft-kafka-mod/[kafkamod]` mod use Apache Kafka to send event about
- Chat
- Entities entering or leaving the world
- Entities crafted or picked up by a player
- Player
.Events we react to is
- PlayerEvent.PlayerLoggedInEvent -> Apache Kafka
- PlayerEvent.PlayerLoggedOutEvent -> Apache Kafka
- PlayerEvent.PlayerChangedDimensionEvent -> All Players - client message
- PlayerEvent.PlayerRespawnEvent -> All Players - client message
- PlayerEvent.ItemCraftedEvent -> Apache Kafka
- PlayerEvent.ItemPickupEvent -> Apache Kafka
- net.minecraftforge.event.ServerChatEvent -> Apache Kafka
- EntityJoinLevelEvent -> Apache Kafka
- EntityLeaveLevelEvent -> Apache Kafka
[source,java]
----
@SubscribeEvent
public void onEntityJoinLevelEvent(EntityJoinLevelEvent event) {
EntityRecord entityRecord = new EntityRecord(event);
String key = entityRecord.getName();
KafkaMod.addRecordToTopic(
key,
entityRecord.toJsonNode(),
KafkaProperties.KAFKA_MOD_ENTITY_EVENT);
}
----
WARNING: The methode `addRecordToTopic` will get one record at a time and send it to `Apache Kafka` - we need to have something left on the ToDo list
[source,java]
----
public static void addRecordToTopic(String key, JsonNode record, String topic) {
LOGGER.debug("Producing record: %s\t%s%n", key, record);
getProducer().ifPresent(
producer -> {
producer.send(new ProducerRecord(topic, key, record),
new Callback() {
@Override
public void onCompletion(RecordMetadata m, Exception e) {
if (e != null) {
e.printStackTrace();
} else {
LOGGER.debug(
"Produced record to topic %s partition [%d] @ offset %d%n",
m.topic(),
m.partition(),
m.offset());
}
}
}
);
});
}
----
All entities is able to map to `JsonNode` like
[source,java]
----
public JsonNode toJsonNode() {
return KafkaMod.objectMapper.valueToTree(this);
}
----
=== Build and deploy
Buildin the `link:minecraft-kafka-mod/[kafkamod]` mod is done with
[source,bash]
----
cd minecraft-kafka-mod
./gradlew build
----
Package the `link:minecraft-kafka-mod/[kafkamod]` mod is done with the use of `shadowJar` in `build.gradle`
----
The gradle build use `shadowJar` to inlude all dependencies into our `kafka-1.0.6.jar` file.
Included is `org.apache.kafka:kafka-clients:3.8.0` and `com.fasterxml.jackson.core:jackson-databind:2.17.2`.
----
An alternative to use `shadowJar` is to add the dependencies (five jar-files) manually one my one in the section `-DlegacyClassPath=` in the file
/opt/minecraft/forge/.minecraft_1.21.1-52.0.9/libraries/net/minecraftforge/forge/1.21.1-52.0.9/unix_args.txt
I use link:minecraft-kafka-mod/build_and_deploy.sh[minecraft-kafka-mod/build_and_deploy.sh] to deploy `minecraftforge`. The
script build the binary and deploy to the folder `$FORGE_SERVER_LOCATION/mods/kafka-1.0.6.jar` - no housekeep when
doing a bump of version.
== Quarkus Kafka
The app `link:quarkus-kafka/[quarkus-kafka]` is created to display the data we write/read to/from Apache Kafka in a nice human readable way.
=== ItemStackProcessor
.Subset of configuration needed to get data from Apache Kafka
----
# Location of a Kafka broker (default is broker1.jarry.dk:9192)
kafka.bootstrap.servers=broker1.jarry.dk:9192
# Configure the incoming `kafka-mod-item-stack` Kafka topic
mp.messaging.incoming.kafka-mod-item-stack.topic=kafka-mod-item-stack
mp.messaging.incoming.kafka-mod-item-stack.auto.offset.reset=earliest
----
The class `ItemStackProcessor` gets records from Apache Kafka, extract the player and send it to `players`.
[source,java]
----
import org.eclipse.microprofile.reactive.messaging.Incoming;
import org.eclipse.microprofile.reactive.messaging.Outgoing;
@Incoming("item-stack")
@Outgoing("players")
public Player process(String itemStack) throws InterruptedException {
Player player = null;
try {
JsonNode itemStackObj = objectMapper.readTree(itemStack);
JsonNode playerObj = itemStackObj.get("player");
player = new Player(playerObj);
} catch (Exception e) {
e.printStackTrace();
}
return player;
}
----
NOTE: We do not need to know if the outgoing `players` is internal or external - in this case it is an internal.
=== PlayerResource
The class `PlayerResource` pick up the `Player` and expose it as a `text/event-stream` endpoint for all updates to `players`.
[source,java]
----
import org.eclipse.microprofile.reactive.messaging.Channel;
import io.smallrye.mutiny.Multi;
@Path("/players")
public class PlayerResource {
@Channel("players")
Multi players;
@GET
@Produces(MediaType.SERVER_SENT_EVENTS)
public Multi stream() {
return players;
}
}
----
Starting the app we are now able to use http://localhost:8080/players.html if in `dev` mode to see updates to players.
image::images/show_player_in_browser.png[]
== Apache Kafka
=== Config for a broker1
.Ports
- PLAINTEXT : 9192
- SSL : 9193
- INTERNAL_PLAINTEXT : 9194
Overwrite `inter.broker.listener.name` with `INTERNAL_PLAINTEXT`
.Script to start broker1
----
#!/bin/bash
KAFKA_ZOOKEEPER_CONNECT="zookeeper.jarry.dk:2181"
KAFKA_LISTENERS="PLAINTEXT://:9192,SSL://:9193,INTERNAL_PLAINTEXT://:9194"
KAFKA_LISTENER_SECURITY_PROTOCAL_MAP="PLAINTEXT:PLAINTEXT,INTERNAL_PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL"
KAFKA_INTER_BROKER_LISTENER_NAME="INTERNAL_PLAINTEXT"
KAFKA_SSL_TRUSTSTORE_LOCATION="/opt/apache/kafka/jarry_dk/kafka.server.truststore.jks"
KAFKA_SSL_TRUSTSTORE_PASSWORD="password1234"
KAFKA_SSL_KEYSTORE_LOCATION="/opt/apache/kafka/jarry_dk/broker1.jarry.dk.keystore.jks"
KAFKA_SSL_KEYSTORE_PASSWORD="password1234"
KAFKA_SSL_CLIENT_AUTH="requested"
KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://broker1.jarry.dk:9192,SSL://broker1.jarry.dk:9193,INTERNAL_PLAINTEXT://broker1.jarry.dk:9194"
KAFKA_BROKER_ID="1"
KAFKA_LOG_DIRS="/tmp/kafka-logs-broker-1"
$KAFKA_HOME/bin/kafka-server-start.sh \
$KAFKA_HOME/config/server.properties \
--override zookeeper.connect=$KAFKA_ZOOKEEPER_CONNECT \
--override listeners=$KAFKA_LISTENERS \
--override listener.security.protocol.map=$KAFKA_LISTENER_SECURITY_PROTOCAL_MAP \
--override inter.broker.listener.name=$KAFKA_INTER_BROKER_LISTENER_NAME \
--override ssl.truststore.location=$KAFKA_SSL_TRUSTSTORE_LOCATION \
--override ssl.truststore.password=$KAFKA_SSL_TRUSTSTORE_PASSWORD \
--override ssl.keystore.location=$KAFKA_SSL_KEYSTORE_LOCATION \
--override ssl.keystore.password=$KAFKA_SSL_KEYSTORE_PASSWORD \
--override ssl.client.auth=$KAFKA_SSL_CLIENT_AUTH \
--override advertised.listeners=$KAFKA_ADVERTISED_LISTENERS \
--override broker.id=$KAFKA_BROKER_ID \
--override log.dirs=$KAFKA_LOG_DIRS
----
.Install Kafka
Follow the https://kafka.apache.org/quickstart[Apache Kafka Quarickstart] to install the `zookeeper` and the `server`.
Do the steeps in the folder `/opt/apache/kafka` and you will have `Apache Kafka` installation in the folder `/opt/apache/kafka/kafka_2.13-3.6.1`
.Export KAFKA_HOME
[source,bash]
----
export KAFKA_HOME=/opt/apache/kafka/kafka_2.13-3.8.0
----
.Start Zookeeper
[source,bash]
----
$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.propertie
----
.Start Kafka
[source,bash]
----
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
----
TIP: Add ` | jq` to get the json from the topic in a nice format. How to install https://stedolan.github.io/jq/[jq].
.Consume the kafka-mod-chat topic
[source,bash]
----
$KAFKA_HOME/bin/kafka-console-consumer.sh \
--bootstrap-server broker1.jarry.dk:9192 \
--topic kafka-mod-chat \
--from-beginning | jq
----
.Consume the kafka-mod-item-stack topic
[source,bash]
----
$KAFKA_HOME/bin/kafka-console-consumer.sh \
--bootstrap-server broker1.jarry.dk:9192 \
--topic kafka-mod-item-stack \
--from-beginning | jq
----
.Consume the kafka-mod-entity-event topic
[source,bash]
----
$KAFKA_HOME/bin/kafka-console-consumer.sh \
--bootstrap-server broker1.jarry.dk:9192 \
--topic kafka-mod-entity-event \
--from-beginning | jq
----
.Consume the kafka-mod-player-event topic
[source,bash]
----
$KAFKA_HOME/bin/kafka-console-consumer.sh \
--bootstrap-server broker1.jarry.dk:9192 \
--topic kafka-mod-player-event \
--from-beginning | jq
----
.Create the topic kafka-mod-entity-event (if needed)
[source,bash]
----
$KAFKA_HOME/bin/kafka-topics.sh \
--bootstrap-server broker1.jarry.dk:9192 \
--topic kafka-mod-entity-event \
--create
----
.Create the topic kafka-mod-entity-event - adv. #1 (if needed)
[source,bash]
----
$KAFKA_HOME/bin/kafka-topics.sh \
--bootstrap-server broker1.jarry.dk:9192 \
--topic kafka-mod-entity-event \
--replica-assignment 0:1:2,0:1:2,0:1:2 \
--create
----
.Create the topic kafka-mod-entity-event - adv. #2 (if needed)
[source,bash]
----
$KAFKA_HOME/bin/kafka-topics.sh \
--bootstrap-server broker1.jarry.dk:9192 \
--topic kafka-mod-entity-event \
--replication-factor 2 \
--create
----
.Discribe the topic kafka-mod-entity-event
[source,bash]
----
$KAFKA_HOME/bin/kafka-topics.sh \
--bootstrap-server broker1.jarry.dk:9192 \
--topic kafka-mod-entity-event \
--describe
----
.Modify the topic kafka-mod-entity-event - change partitions
[source,bash]
----
$KAFKA_HOME/bin/kafka-topics.sh \
--bootstrap-server broker1.jarry.dk:9192 \
--topic kafka-mod-entity-event \
--partitions 3 \
--alter
----
NOTE: This can be done with kafka-reassign-partitions.sh too.
.Delete the topic kafka-mod-entity-event
[source,bash]
----
$KAFKA_HOME/bin/kafka-topics.sh \
--bootstrap-server broker1.jarry.dk:9192 \
--topic kafka-mod-entity-event \
--delete
----
.Increasing replication factor for the topic kafka-mod-entity-event
[source,bash]
----
cat > increase-replication-factor.json << EOF
{
"version": 1,
"partitions": [
{
"topic": "kafka-mod-entity-event",
"partition": 0,
"replicas": [
0,
1
],
"replication-factor" : 2
}
]
}
EOF
----
[source,bash]
----
$KAFKA_HOME/bin/kafka-reassign-partitions.sh \
--bootstrap-server broker1.jarry.dk:9192 \
--reassignment-json-file increase-replication-factor.json \
--execute
----
Source : https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor
== Links
- https://files.minecraftforge.net/net/minecraftforge/forge/[Downloads for Minecraft Forge]
- https://docs.minecraftforge.net/en/latest/[MinecraftForge Documentation]
- https://github.com/MinecraftForge/MinecraftForge[MinecraftForge]
- https://nekoyue.github.io/ForgeJavaDocs-NG/javadoc/1.19.1/index.html
- https://quarkus.io/guides/kafka-reactive-getting-started[Getting Started to SmallRye Reactive Messaging with Apache Kafka - Quarkus]
- https://threejs.org
- https://stedolan.github.io/jq/
- https://sequencediagram.org/
=== Kafka links
- https://kafka.apache.org/32/javadoc/index-all.html
- https://kafka.apache.org/32/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html
- https://kafka.apache.org/32/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
- https://github.com/confluentinc/examples/tree/7.2.1-post/clients/cloud/java/src/main/java/io/confluent/examples/clients/cloud
- https://hevodata.com/learn/kafka-replication/
- https://medium.com/@_amanarora/replication-in-kafka-58b39e91b64e
- https://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
- https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor
- https://sleeplessbeastie.eu/2022/01/05/how-to-reassign-kafka-topic-partitions-and-replicas/[How to reassign Kafka topic partitions]
- https://developer.okta.com/blog/2020/04/08/kafka-streams[Secure Kafka Streams with Quarkus and Java]