Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ppatierno/formula1-telemetry-kafka
https://github.com/ppatierno/formula1-telemetry-kafka
apache-camel apache-kafka formula1 grafana influxdb kafka kafka-influxdb kafka-streams telemetry-data telemetry-events udp-kafka
Last synced: 9 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/ppatierno/formula1-telemetry-kafka
- Owner: ppatierno
- License: apache-2.0
- Created: 2020-08-18T12:16:34.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2022-10-09T14:12:33.000Z (about 2 years ago)
- Last Synced: 2024-10-10T20:53:07.208Z (26 days ago)
- Topics: apache-camel, apache-kafka, formula1, grafana, influxdb, kafka, kafka-influxdb, kafka-streams, telemetry-data, telemetry-events, udp-kafka
- Language: Java
- Homepage:
- Size: 15.1 MB
- Stars: 147
- Watchers: 9
- Forks: 29
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/ppatierno/formula1-telemetry-kafka/actions/workflows/build.yml)
[](https://open.vscode.dev/ppatierno/formula1-telemetry-kafka)
# Formula 1 - Telemetry with Apache Kafka
This project aims to use Apache Kafka in order to ingest Formula 1 telemetry data from the F1 202x game (by CodeMasters/EA Sports) running on Microsoft Xbox.
It uses different products, projects and technologies:
* **[F1 202x games (by CodeMasters/EA Sports)](https://www.ea.com/en-gb/games/f1)** for getting the telemetry data via UDP;
* **[Apache Camel](https://camel.apache.org/)** project for routing telemetry events from UDP to Apache Kafka and to InfluxDB;
* **[Apache Kafka](https://kafka.apache.org/)** as the core project for ingesting the telemetry events on different topics;
* **[Kubernetes](https://kubernetes.io/)** for deploying most of the components to run in the cloud;
* **[Strimzi](https://strimzi.io/)** for deploying easily the Apache Kafka on Kubernetes;
* **[InfluxDB](https://www.influxdata.com/)** for storing the telemetry time series as data source for dashboards;
* **[Grafana](https://grafana.com/)** for providing dashboards showing the real time telemetry data;
> The formula 1 telemetry library, used to decode the UDP packets, works with the F1 2020 game format. It's possible to have the project working with newer game editions by selecting the "2020" UDP packet format in the Telemetry settings on the game.
It is possible to select the season for teams, drivers and tracks by setting the `F1_TELEMETRY_SEASON` environment variable when running the different applications.
By default, it uses 2020 season data. For more information see [here](https://github.com/ppatierno/formula1-telemetry#season-selection).
## Overview
Following an overall picture of how these technologies are used together.
A couple of short introduction videos are available [here](https://youtu.be/Re9LOAYZi2A) and [here](https://youtu.be/2efOtyFAZ4s).
You can also find some material from conference presentations:
* Let's code a motorsport telemetry processing pipeline - [slides](https://www.slideshare.net/paolopat/lets-code-a-motorsport-telemetry-processing-pipeline) [demo](https://github.com/ppatierno/formula1-telemetry-kafka) [video](https://www.youtube.com/watch?v=zFmON75w1kE)
* Formula 1 telemetry processing using Kafka Streams - [slides](https://www.slideshare.net/paolopat/formula-1-telemetry-processing-using-kafka-streams) [demo](https://github.com/ppatierno/devconfcz-2021-formula1-kafka-streams) [video](https://www.youtube.com/watch?v=OkXlSb4vfDk)
* Formula 1 telemetry processing using Quarkus and OpenShift Streams for Apache Kafka - [slides](https://www.slideshare.net/paolopat/formula-1-telemetry-processing-using-quarkus-and-openshift-streams-for-apache-kafka) [demo](https://github.com/ppatierno/formula1-telemetry-rhosak) [video](https://events.experiences.redhat.com/widget/redhat/sum22/SessionCatalog22/session/1641400967915001tsYD)
### F1 202x Xbox UDP to Kafka
In order to ingest the telemetry events into Apache Kafka, the Apache Camel project is used with:
* a route getting the raw UDP packets from the F1 202x game (by CodeMasters/EA Sports) on Microsoft Xbox and dispatching these events to three more routes;
* a route just forwarding the raw UDP packets to a corresponding Apache Kafka topic;
* a route to filter only the `EVENT` type raw UDP packets and forwarding them to a corresponding Apache Kafka topic;
* a route to aggregate the raw UDP packets data for producing drivers related data and forwarding them to a corresponding Apache Kafka topic;
### Kafka Streams API
The Kafka Streams API based application shows an example of real-time analytics on the telemetry data.
The application processes the average speed in the last 5 seconds.
It has a source node reading from the topic with drivers related messages, filtering the ones not containing valid telemetry data.
Extract and group the driver's speed by corresponding driver's id using in a tumbling window of 5 seconds.
Next, it sums speeds and counts them in order to process the corresponding average value.
Finally, the sink node writes to the destination topic.
### Kafka to InfluxDB
In order to provide the telemetry data to Grafana dashboards, InfluxDB is used as data source and the telemetry events are stored through Apache Camel with:
* a route getting drivers related data for storing telemetry, motion and car status data;
* a route getting the `EVENT` type raw UDP packets for storing fastest lap and speedtrap events;
## Repository structure
Contains different components for ingesting and handling Formula 1 202x game (by CodeMasters/EA Sports) telemetry data through Apache Kafka.
* **udp-kafka**: Apache Camel application bridging the telemetry packets got on UDP from the Formula 1 202x game to Apache Kafka to different topics:
* _f1-telemetry-packets_ contains the raw `Packet`(s);
* _f1-telemetry-events_ contains only the raw `Packet`(s) of `EVENT` type;
* _f1-telemetry-drivers_ contains the `Driver` messages as result of aggregating `Packet`(s) in the same frame with telemetry data for all drivers;
* **consumer**: Apache Kafka client application consuming `Driver` messages from Apache Kafka;
* **streams-avg-speed**: Apache Kafka Streams API based application getting raw `Driver`(s), processing them in real time to get average speed in a 5 seconds window and writing to Apache Kafka to the _f1-telemetry-streams-avg-speed_ topic;
* **streams-laps**: Apache Kafka Streams API based application getting raw `Driver`(s), processing them in real time to get the best overall times in the three track sectors and writing to Apache Kafka to the _f1-telemetry-best-overall-sector_ topic;
* **common**: common library providing model classes and related Apache Kafka serializer/deserializer;
* **kafka-influxdb**: Apache Camel application writing driver/car's telemetry data to InfluxDB as time series;
* **webui**: A sample Web application showing the race ranking in real time getting `Driver`(s) from Apache Kafka;
* **streams-processing**: Apache Kafka Streams API based application getting raw `Driver`(s) and extracting some telemetry in JSON format;
* **dashboard**: folder containing Grafana dashboard showing driver/car's telemetry data;
* **deployment**: folder containing Kubernetes deployments for all the provided applications;
* **documentation**: folder containing documentation about how to deploy the entire solution;
The Formula 1 2020 game UDP packets specification is [here](https://forums.codemasters.com/topic/54423-f1%C2%AE-2020-udp-specification/).
The library used for decoding the packets is [here](https://github.com/ppatierno/formula1-telemetry).
## Documentation
You can find more information about building, deploying and running the solution in the documentation [here](documentation/README.md)
## Dashboards
The telemetry dashboard shows information like speed, engine (rpm), throttle and brake.
The motion dashboard shows information about the car in motion like the G-force.
The car status dashboard shows information mostly related to the status of each car, for example wings damages and fuel in tank.
The events dashboard shows race events like max speed trap and fastest lap.
The driver dashboard allows to get data for a single driver, selecting one through the corresponding drop down list.
It has a top bar with information about current lap, position, distance and lap times.
It also has some graphs showing specific driver data related to throttle/brake, engine/speed, gear/clutch, brakes and steering.
It also shows specific information about tyres like, for example, the compound, the age laps, wear, surface temperature and damage
The streams dashboard shows data about processed telemetry data through Kafka Streams application.
The first one is about the average speed during the last 5 seconds.
