https://github.com/bakdata/pipeline-atm-fraud
Demo pipeline for ATM fraud detection using streams-bootstrap
https://github.com/bakdata/pipeline-atm-fraud
Last synced: about 1 year ago
JSON representation
Demo pipeline for ATM fraud detection using streams-bootstrap
- Host: GitHub
- URL: https://github.com/bakdata/pipeline-atm-fraud
- Owner: bakdata
- License: mit
- Created: 2024-03-27T15:55:27.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-03-28T13:27:08.000Z (about 2 years ago)
- Last Synced: 2025-04-10T23:48:09.427Z (about 1 year ago)
- Language: Java
- Size: 195 KB
- Stars: 1
- Watchers: 6
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Kafka demo pipeline: ATM fraud detection with streams-bootstrap
> This is an adaption of the example pipeline for ATM fraud detection using [streams-bootstrap](https://github.com/bakdata/streams-bootstrap). The original by Confluent is written in KSQL and can be found in the [ksql-atm-fraud-detection](https://github.com/confluentinc/demo-scene/tree/master/ksql-atm-fraud-detection) repo. Details can be found in their [blogpost](https://www.confluent.io/blog/atm-fraud-detection-apache-kafka-ksql/)
>
> This version differs slightly from the original in that the test accounts are generated using a producer application.
> In addition, we turned the transactionavroproducer application into a producer application, eliminating the need to consume transactions data from an external producer.
> These minor changes allow us to have an automated deployment with no external intervention or manual data production.
## Usage
### Build containers using jib
```shell
gradle jib -Djib.to.image=url-to-container-registry.com/streams-explorer-demo-transactionavroproducer -Djib.container.mainClass=com.bakdata.kafka.TransactionAvroProducer
gradle jib -Djib.to.image=url-to-container-registry.com/streams-explorer-demo-accountproducer -Djib.container.mainClass=com.bakdata.kafka.AccountProducer
gradle jib -Djib.to.image=url-to-container-registry.com/streams-explorer-demo-transactionjoiner -Djib.container.mainClass=com.bakdata.kafka.TransactionJoiner
gradle jib -Djib.to.image=url-to-container-registry.com/streams-explorer-demo-frauddetector -Djib.container.mainClass=com.bakdata.kafka.FraudDetector
gradle jib -Djib.to.image=url-to-container-registry.com/streams-explorer-demo-accountlinker -Djib.container.mainClass=com.bakdata.kafka.AccountLinker
```
### Deploy in Kubernetes
```shell
helm repo add bakdata-common https://raw.githubusercontent.com/bakdata/streams-bootstrap/master/charts/
helm repo update
helm upgrade --debug --install --force --values values-transactionavroproducer.yaml demo-transactionavroproducer bakdata-common/streams-app
helm upgrade --debug --install --force --values values-accountproducer.yaml demo-accountproducer bakdata-common/streams-app
helm upgrade --debug --install --force --values values-transactionjoiner.yaml demo-transactionjoiner bakdata-common/streams-app
helm upgrade --debug --install --force --values values-frauddetector.yaml demo-frauddetector bakdata-common/streams-app
helm upgrade --debug --install --force --values values-accountlinker.yaml demo-accountlinker bakdata-common/streams-app
```
### Generate test accounts
The account producer application publishes accounts data (stored in a `.txt` file) into the expected topic.
### Generate test transactions
To generate our incoming transactions (legitimate or fraudulent) we are using the `transactionavroproducer` application inspired by the [gess](https://github.com/rmoff/gess) tool. This project's data is derived entirely from the same gess project.
The number of incoming transactions is configurable using the variables `REAL_TX` and `ITERATION` in our `values-transactionavroproducer.yaml` file.
The first variable specifies how many legitimate incoming transactions must be produced before producing one fraudulent incoming transaction.
One iteration contains `REAL_TX` legitimate transactions and one fraudulent transaction.