https://github.com/aiven-labs/python-fake-data-producer-for-apache-kafka
The Python fake data producer for Apache Kafka® is a complete demo app allowing you to quickly produce JSON fake streaming datasets and push it to an Apache Kafka topic.
https://github.com/aiven-labs/python-fake-data-producer-for-apache-kafka
aiven aiven-cloud apache-kafka apache-kafka-producer fake-data-generator fake-datasets fake-streaming kafka kafka-producer pyton
Last synced: 2 months ago
JSON representation
The Python fake data producer for Apache Kafka® is a complete demo app allowing you to quickly produce JSON fake streaming datasets and push it to an Apache Kafka topic.
- Host: GitHub
- URL: https://github.com/aiven-labs/python-fake-data-producer-for-apache-kafka
- Owner: Aiven-Labs
- License: apache-2.0
- Created: 2021-01-25T14:25:11.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2023-08-30T09:37:49.000Z (over 1 year ago)
- Last Synced: 2023-08-30T16:10:26.990Z (over 1 year ago)
- Topics: aiven, aiven-cloud, apache-kafka, apache-kafka-producer, fake-data-generator, fake-datasets, fake-streaming, kafka, kafka-producer, pyton
- Language: Python
- Homepage: http://aiven.io
- Size: 209 KB
- Stars: 61
- Watchers: 73
- Forks: 33
- Open Issues: 1
-
Metadata Files:
- Readme: README-gitpod.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Quickstart with Gitpod
This workspace comes with some pre-installed stuff for you :
* Python requirements have already been installed
* avn CLI has already been installed
* jq has benn installed
First make sure to have an Aiven account, otherwise you are just a few clicks away of creating one [here](https://console.aiven.io/signup?utm_source=github&utm_medium=organic&utm_campaign=blog_art&utm_content=post)
Then make sure to get an personal access token, check this [video](https://www.youtube.com/watch?v=64G2QIMYOL4) to learn how to get one.
Open a terminal, you'll need to copy-paste or re-type all the bash commands below.
Now you can login :
```bash
avn user login --token
```
Create a `certs` folder :
```bash
mkdir certs
```
Set your variables :
```bash
KAFKA_INSTANCE_NAME=my-kafka-demo
CLOUD_REGION=aws-eu-south-1
AIVEN_PLAN_NAME=startup-2
DESTINATION_FOLDER_NAME=certs
```
If you haven't yet, create a Aiven for Apache Kafka service :
```bash
avn service create $KAFKA_INSTANCE_NAME \
-t kafka \
--cloud $CLOUD_REGION \
-p $AIVEN_PLAN_NAME \
-c kafka.auto_create_topics_enable=true \
-c kafka_rest=true
```
Retrieve your host and port from the console and set them :
And retrieve the Apache Kafka Service URI with
```bash
KAFKA_HOST=$(avn service get $KAFKA_INSTANCE_NAME --json | jq -r '.service_uri_params.host')
KAFKA_PORT=$(avn service get $KAFKA_INSTANCE_NAME --json | jq -r '.service_uri_params.port')
```
You can wait for the newly created Apache Kafka instance to be ready with :
```bash
avn service wait $KAFKA_INSTANCE_NAME
```
Now get your certificates :
```bash
avn service user-creds-download $KAFKA_INSTANCE_NAME \
-d $DESTINATION_FOLDER_NAME \
--username avnadmin
```
And finally run the demo :
```bash
python main.py \
--security-protocol ssl \
--cert-folder $DESTINATION_FOLDER_NAME\
--host $KAFKA_HOST \
--port $KAFKA_PORT \
--topic-name pizza-orders \
--nr-messages 0 \
--max-waiting-time 2 \
--subject pizza
```
You should see a continuous flow of data being pushed to Apache Kafka, to the topic defined by the `--topic-name` parameter. You can either use the Aiven console, or tools like [kcat](https://docs.aiven.io/docs/products/kafka/howto/kcat) to browse the data.