{"id":17638881,"url":"https://github.com/vishalgattani/quixotic-kafka","last_synced_at":"2026-02-02T16:41:26.414Z","repository":{"id":258338795,"uuid":"873634377","full_name":"vishalgattani/quixotic-kafka","owner":"vishalgattani","description":"Python Stream Processing for Apache Kafka, Spark, Cassandra.","archived":false,"fork":false,"pushed_at":"2024-10-17T17:06:28.000Z","size":40,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-14T07:56:19.630Z","etag":null,"topics":["cassandra","cassandra-database","docker","kafka","kafka-consumer","kafka-producer","kafka-streams","quixstream","ros","ros-noetic","spark-sql","spark-streaming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vishalgattani.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-16T13:48:03.000Z","updated_at":"2024-10-17T17:06:32.000Z","dependencies_parsed_at":"2024-10-18T14:36:24.547Z","dependency_job_id":null,"html_url":"https://github.com/vishalgattani/quixotic-kafka","commit_stats":null,"previous_names":["vishalgattani/quixotic-kafka"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vishalgattani/quixotic-kafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vishalgattani%2Fquixotic-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vishalgattani%2Fquixotic-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vishalgattani%2Fquixotic-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vishalgattani%2Fquixotic-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vishalgattani","download_url":"https://codeload.github.com/vishalgattani/quixotic-kafka/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vishalgattani%2Fquixotic-kafka/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261130365,"owners_count":23113976,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra","cassandra-database","docker","kafka","kafka-consumer","kafka-producer","kafka-streams","quixstream","ros","ros-noetic","spark-sql","spark-streaming"],"created_at":"2024-10-23T04:05:27.230Z","updated_at":"2026-02-02T16:41:21.382Z","avatar_url":"https://github.com/vishalgattani.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# quixotic-kafka\nPython Stream Processing for Apache Kafka.\n\n# Why do I need this?\n\nCurrently, I have a Unity-ROS simulation setup in a docker container. I am wanting to learn how to publish ROS topic data into a structured streaming pipeline by building an architecture that includes a data source (my Unity-ROS simulation) and  to collect data without any loss, analyze them and store results in a database.\n\nI am thinking something along the lines of the following diagram. I shall utilize ROS (Robot Operating System) as a data provider, Kafka as a message queue, Apache Spark as a data processing engine and Apache Cassandra as a database.\n\n```mermaid\nflowchart LR\n    subgraph Data Source\n        ros(Unity-ROS-noetic\u003cbr\u003eSimulation)\n    end\n\n    subgraph Structured Streaming\n        kafka(Kafka) --\u003e spark(Spark)\n        spark --\u003e cass(Cassandra)\n        kafka -. Streaming\u003cbr\u003eDataFrames\u003cbr\u003e(WIP) .-\u003e cass\n    end\n\n    ros --\u003e kafka\n```\n\nAlso, it would be fun to setup a structured streaming pipeline without having to deal with unnecessary headers involved with ROS simulation.\n\nAnother addition would be to use `quixstreams` to setup a streaming pipeline without having to switch the implementation downstream to Spark and bypass it. Essentially, directly utilising Quix Streams' consumer's Streaming DataFrames should enable to solve most of your transformation needs as well as easily sink to Cassandra. A work that is currently in progress.\n\n# Setup\n\nFirstly, I have my ROS simulation up and running in a docker. I would want to publish the rostopic data into Kafka (producer). Then I would setup a way to fetch the data from Kafka (consumer).\n\nSetup kafka console on the host machine using the [Quix CLI](https://quix.io/docs/quix-cli/cli-quickstart.html)\n\n```sh\ncurl -fsSL https://github.com/quixio/quix-cli/raw/main/install.sh | bash\n```\n\nTo verify you have the dependencies installed, run the following command:\n\n```sh\nquix status\n```\nView the output carefully to confirm you have Git and Docker installed:\n```\n✗ Not logged in\n  User:                       ! Not logged in to Quix Cloud\n  Current context:            default (https://portal-api.platform.quix.io)\n  Default environment:        ! Not set\n  SDK Broker configuration:   Local (localhost:19092)\n! Local Pipeline Status:      Not Running\n✓ Local Broker Status:        Running (localhost:19092)\n✓ Local Broker GUI:           Running (http://localhost:8080)\n✓ Docker detected\n✓ Git detected\n  Git Root:                   /home/vishal/mygithub/quixotic-kafka\n```\n\nThe following command will create the right docker compose file such that you have a broker running on localhost:19092 and you do not have to worry about setting up the environment variables and the configuration.\n```sh\nvishal@vishal:~/mygithub/quixotic-kafka$ quix pipeline up\n  Generating 'compose.local.yaml'\n! No deployments found\n✓ Generated 'compose.local.yaml'\n\nExecuting 'docker compose -f compose.local.yaml up --build -d --remove-orphans kafka_broker'\n\n[+] Running 2/2\n ✔ Network quixotic-kafka_default           Created                                                                                                            0.0s\n ✔ Container quixotic-kafka-kafka_broker-1  Started                                                                                                            0.3s\n! No topics specified\n\nExecuting 'docker compose -f compose.local.yaml up --build -d --remove-orphans'\n\n[+] Running 2/2\n ✔ Container quixotic-kafka-console-1       Started                                                                                                            0.3s\n ✔ Container quixotic-kafka-kafka_broker-1  Running                                                                                                            0.0s\n✓ Open http://localhost:8080 to manage your pipeline broker\n```\n\n\n\n## Connectivity issues\n\n1. Firstly, I need my ROS docker container to be able to communicate with kafka.\n\n```sh\nvishal@vishal:~$ catkin-docker run\n55c0ca9f980ef2553d78664ef64659545619c36894a32c3a8b91609c1df4d9bf\nvishal@vishal:~$ docker exec -it 55c0ca9f980ef2553d78664ef64659545619c36894a32c3a8b91609c1df4d9bf /bin/bash\nroot@vishal:/home/vishal# source docker-build/install/setup.bash\nroot@vishal:/home/vishal/kafka-producer# cat setup.sh\napt update \u0026\u0026 apt install python3.8-venv  telnet -y\npip3 install -r requirements.txt\nroot@vishal:/home/vishal/kafka-producer# ./setup.sh\nroot@vishal:/home/vishal# telnet localhost 19092\nTrying 127.0.0.1...\nConnected to localhost.\nEscape character is '^]'.\n\n\nConnection closed by foreign host.\nroot@vishal:/home/vishal# telnet 10.0.0.82 19092\nTrying 10.0.0.82...\nConnected to 10.0.0.82.\nEscape character is '^]'.\n\n\nConnection closed by foreign host.\n```\n\nI am able to connect to the kafka broker! Now, I need to push data into kafka.\n\n2. I need to make sure that my ROS simulation is publishing the data into the topic. Since I have not started a simulation, I won't be able to get any data out.\n\n```sh\nroot@vishal:/home/vishal# rostopic list\nERROR: Unable to communicate with master!\nroot@vishal:/home/vishal/kafka-producer# python3 kafka_producer.py\nHostname: vishal\nIP Address: 10.0.0.82\nUnable to register with master node [http://localhost:11311]: master may not be running yet. Will keep trying.\n```\n\nRunning a ROS simulation with a Clearpath Warthog robot and fetching rostopic data.\n\n```sh\n# assuming i have already started the ros simulation\nroot@vishal:/home/vishal# rostopic list | grep \"/warty/odom\" | wc -l\n4\nroot@vishal:/home/vishal# rostopic list | grep \"/warty/odom\"\n/ground_truth/warty/odom\n/warty/odom\n/warty/odometry/parameter_descriptions\n/warty/odometry/parameter_updates\n```\n\nI can see that I have 4 rostopics related to odometry. I shall use the `/odom` topic to publish data into kafka.\n\n\n3. Starting a ROS simulation to check if I can push data into kafka.\n\n```sh\nroot@vishal:/home/vishal/kafka-producer# rostopic list | wc -l\n337\nroot@vishal:/home/vishal/kafka-producer# python3 kafka_producer.py\nHostname: vishal\nIP Address: 10.0.0.82\nProducing message 2024-10-16 11:56:56.568391 | Message:\n{'id': 0, 'posex': 0.0, 'posey': 0.0, 'posez': 0.0, 'orientx': 0.03372, 'orienty': 0.06719, 'orientz': -0.08519, 'orientw': 0.99352}\nProducing message 2024-10-16 11:56:56.868415 | Message:\n{'id': 1, 'posex': 0.0, 'posey': 0.0, 'posez': 0.0, 'orientx': 0.03369, 'orienty': 0.06719, 'orientz': -0.08519, 'orientw': 0.99353}\nProducing message 2024-10-16 11:56:56.868699 | Message:\n{'id': 2, 'posex': 0.0, 'posey': 0.0, 'posez': 0.0, 'orientx': 0.03359, 'orienty': 0.0672, 'orientz': -0.08518, 'orientw': 0.99353}\nProducing message 2024-10-16 11:56:56.868822 | Message:\n{'id': 3, 'posex': 0.0, 'posey': 0.0, 'posez': 0.0, 'orientx': 0.03359, 'orienty': 0.0672, 'orientz': -0.08518, 'orientw': 0.99353}\nProducing message 2024-10-16 11:56:56.869439 | Message:\n{'id': 4, 'posex': 0.0, 'posey': 0.0, 'posez': 0.0, 'orientx': 0.03363, 'orienty': 0.06719, 'orientz': -0.08518, 'orientw': 0.99353}\nProducing message 2024-10-16 11:56:56.869652 | Message:\n{'id': 5, 'posex': 0.0, 'posey': 0.0, 'posez': 0.0, 'orientx': 0.03368, 'orienty': 0.06718, 'orientz': -0.08519, 'orientw': 0.99353}\nProducing message 2024-10-16 11:56:56.869785 | Message:\n{'id': 6, 'posex': 0.0, 'posey': 0.0, 'posez': 0.0, 'orientx': 0.03366, 'orienty': 0.06718, 'orientz': -0.08519, 'orientw': 0.99353}\nProducing message 2024-10-16 11:56:56.870448 | Message:\n{'id': 7, 'posex': 0.0, 'posey': 0.0, 'posez': 0.0, 'orientx': 0.03374, 'orienty': 0.06718, 'orientz': -0.08519, 'orientw': 0.99352}\nProducing message 2024-10-16 11:56:56.870580 | Message:\n{'id': 8, 'posex': 0.0, 'posey': 0.0, 'posez': 0.0, 'orientx': 0.03362, 'orienty': 0.06718, 'orientz': -0.08518, 'orientw': 0.99353}\nProducing message 2024-10-16 11:56:56.870731 | Message:\n{'id': 9, 'posex': 0.0, 'posey': 0.0, 'posez': 0.0, 'orientx': 0.03364, 'orienty': 0.06718, 'orientz': -0.08519, 'orientw': 0.99353}\nProducing message 2024-10-16 11:56:56.870786 | Message:\n{'id': 10, 'posex': 0.0, 'posey': 0.0, 'posez': 0.0, 'orientx': 0.03368, 'orienty': 0.06718, 'orientz': -0.08519, 'orientw': 0.99353}\n```\n\nAlright, seems like we are getting the data to push into kafka. However, I need to check if kafka received the data. Using `kcat` or formerly known as `kafkacat` I can check if the data is getting pushed to kafka.\n\n```sh\nroot@vishal:/home/vishal/kafka-producer# kafkacat -C -b 10.0.0.82:9092 -t my-topic\n{\"id\": 0, \"posex\": 0.0, \"posey\": 0.0, \"posez\": 0.0, \"orientx\": 0.03372, \"orienty\": 0.06719, \"orientz\": -0.08519, \"orientw\": 0.99352}\n{\"id\": 1, \"posex\": 0.0, \"posey\": 0.0, \"posez\": 0.0, \"orientx\": 0.03369, \"orienty\": 0.06719, \"orientz\": -0.08519, \"orientw\": 0.99353}\n{\"id\": 2, \"posex\": 0.0, \"posey\": 0.0, \"posez\": 0.0, \"orientx\": 0.03359, \"orienty\": 0.0672, \"orientz\": -0.08518, \"orientw\": 0.99353}\n{\"id\": 3, \"posex\": 0.0, \"posey\": 0.0, \"posez\": 0.0, \"orientx\": 0.03359, \"orienty\": 0.0672, \"orientz\": -0.08518, \"orientw\": 0.99353}\n{\"id\": 4, \"posex\": 0.0, \"posey\": 0.0, \"posez\": 0.0, \"orientx\": 0.03363, \"orienty\": 0.06719, \"orientz\": -0.08518, \"orientw\": 0.99353}\n{\"id\": 5, \"posex\": 0.0, \"posey\": 0.0, \"posez\": 0.0, \"orientx\": 0.03368, \"orienty\": 0.06718, \"orientz\": -0.08519, \"orientw\": 0.99353}\n{\"id\": 6, \"posex\": 0.0, \"posey\": 0.0, \"posez\": 0.0, \"orientx\": 0.03366, \"orienty\": 0.06718, \"orientz\": -0.08519, \"orientw\": 0.99353}\n{\"id\": 7, \"posex\": 0.0, \"posey\": 0.0, \"posez\": 0.0, \"orientx\": 0.03374, \"orienty\": 0.06718, \"orientz\": -0.08519, \"orientw\": 0.99352}\n{\"id\": 8, \"posex\": 0.0, \"posey\": 0.0, \"posez\": 0.0, \"orientx\": 0.03362, \"orienty\": 0.06718, \"orientz\": -0.08518, \"orientw\": 0.99353}\n{\"id\": 9, \"posex\": 0.0, \"posey\": 0.0, \"posez\": 0.0, \"orientx\": 0.03364, \"orienty\": 0.06718, \"orientz\": -0.08519, \"orientw\": 0.99353}\n{\"id\": 10, \"posex\": 0.0, \"posey\": 0.0, \"posez\": 0.0, \"orientx\": 0.03368, \"orienty\": 0.06718, \"orientz\": -0.08519, \"orientw\": 0.99353}\n```\n\nAwesome! I am getting somewhere. Since I am publishing to a topic called `my-topic`, I am getting the data stored in kafka. Time to clean-up and use `quixstreams`! I have been wanting to learn this for a while.\n\n4. Drop the `kafka-producer/producer.py` and `kafka-producer/helper.py` files in the docker container running ROS. Run the kafka producer script to start publishing to kafka. Once we have messages in kafka, we can setup a consumer to check if the data is getting pushed to kafka.\n\n6. Preparing Apache Cassandra for use case: Initially, we must create a keyspace and then a topic in it using the given command.\n\n```sh\n# Open the cqlsh and then run the command to create 'ros' keyspace\ncqlsh\u003e CREATE KEYSPACE ros WITH replication = {'class':'SimpleStrategy', 'replication_factor' : 1};\n# Then, run the command to create 'odometry' topic in 'ros'\ncqlsh\u003e create table ros.odometry(\n        id int primary key,\n        posex float,\n        posey float,\n        posez float,\n        orientx float,\n        orienty float,\n        orientz float,\n        orientw float);\n# Check your setup is correct\ncqlsh\u003e DESCRIBE ros\n#and\ncqlsh\u003e DESCRIBE ros.odometry\n```\n\n7. Prepare Apache Spark structured streaming pipeline: Now we setup a spark job to read the data from kafka and write it to cassandra. The following script will read odometry topic from Kafka, (may or may not analyze it) and then writes results to Cassandra.\n  - Create a schema the same as we already defined in Cassandra.\n    ```sh\n    odometrySchema = StructType([\n                  StructField(\"id\",IntegerType(),False),\n                  StructField(\"posex\",FloatType(),False),\n                  StructField(\"posey\",FloatType(),False),\n                  StructField(\"posez\",FloatType(),False),\n                  StructField(\"orientx\",FloatType(),False),\n                  StructField(\"orienty\",FloatType(),False),\n                  StructField(\"orientz\",FloatType(),False),\n                  StructField(\"orientw\",FloatType(),False)\n              ])\n    ```\n  - create a Spark Session using two packages:\n    - spark-cassandra-connector: `com.datastax.spark:spark-cassandra-connector_2.12:3.5.1`\n    - spark-sql-kafka-0-10: `org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.3`\n      ```sh\n      spark = SparkSession \\\n        .builder \\\n        .appName(\"SparkStructuredStreaming\") \\\n        .config(\"spark.jars.packages\", \"com.datastax.spark:spark-cassandra-connector_2.12:3.5.1,org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.3\") \\\n        .getOrCreate()\n      ```\n  - To read Kafka stream, we use readStream() and specify Kafka configurations as the given below:\n    ```sh\n    df = spark \\\n      .readStream \\\n      .format(\"kafka\") \\\n      .option(\"kafka.bootstrap.servers\", \"localhost:9092\") \\\n      .option(\"subscribe\", \"odometry\") \\\n      .option(\"delimeter\",\",\") \\\n      .option(\"startingOffsets\", \"latest\") \\\n      .load()\n    ```\n  - Since Kafka sends data as binary, first we need to convert the binary value to String using selectExpr() as the given below:\n    ```sh\n    df1 = df.selectExpr(\"CAST(value AS STRING)\")\\\n    .select(from_json(col(\"value\"),odometrySchema)\\\n    .alias(\"data\"))\\\n    .select(\"data.*\")\n\n    df1.printSchema()\n    ```\n  - Apache Spark isn’t capable of directly write stream data to Cassandra yet (using writeStream()), we can do it with use foreachBatch() as the given below:\n    ```sh\n    def writeToCassandra(writeDF, _):\n      writeDF.write \\\n        .format(\"org.apache.spark.sql.cassandra\")\\\n        .mode('append')\\\n        .options(table=\"odometry\", keyspace=\"ros\")\\\n        .save()\n    df1.writeStream \\\n        .option(\"spark.cassandra.connection.host\",\"localhost:9042\")\\\n        .foreachBatch(writeToCassandra) \\\n        .outputMode(\"update\") \\\n        .start()\\\n        .awaitTermination()\n    ```\n\n```sh\npython3 kafka2cassandra.py\n```\n\nNow we have our data being pushed into Cassandra.\n\n8. We can check if the data is getting pushed into Cassandra by running the following command:\n\n```sh\ncqlsh\u003e select * from ros.odometry;\n(24 rows)\n id    | orientw | orientx | orienty | orientz  | posex | posey | posez\n-------+---------+---------+---------+----------+-------+-------+-------\n 702413 | 0.64763 |  0.10716 |  -0.0444 |  0.75307 | 41.51074 |  40.37412 | 36.56737\n 704108 | 0.64624 |  0.10403 | -0.04495 |  0.75467 | 41.51074 |  40.37412 | 36.56737\n 524849 | 0.02104 |  0.01246 |  -0.0418 |  0.99883 |  3.75939 | -10.18663 |   5.3459\n 702937 | 0.64724 |   0.1061 | -0.04455 |  0.75355 | 41.51074 |  40.37412 | 36.56737\n 701703 |  0.6479 |  0.10793 | -0.04398 |  0.75275 | 41.51074 |  40.37412 | 36.56737\n 704008 | 0.64633 |  0.10423 |  -0.0449 |  0.75458 | 41.51074 |  40.37412 | 36.56737\n  35355 | 0.99266 |  0.01285 |  0.03103 | -0.11622 |        0 |         0 |        0\n 525183 | 0.02129 |  0.01213 | -0.04196 |  0.99882 |  3.75939 | -10.18663 |   5.3459\n 704175 | 0.64618 |  0.10395 | -0.04495 |  0.75474 | 41.51074 |  40.37412 | 36.56737\n  35388 | 0.99265 |  0.01281 |  0.03102 | -0.11628 |        0 |         0 |        0\n 702498 | 0.64757 |  0.10697 | -0.04442 |  0.75315 | 41.51074 |  40.37412 | 36.56737\n 702450 | 0.64759 |  0.10709 |  -0.0444 |  0.75311 | 41.51074 |  40.37412 | 36.56737\n 701798 | 0.64783 |  0.10813 | -0.04435 |  0.75276 | 41.51074 |  40.37412 | 36.56737\n 701218 | 0.64887 |   0.0904 | -0.02817 |  0.75499 | 41.51074 |  40.37412 | 36.56737\n 525117 | 0.02123 |   0.0122 |   -0.042 |  0.99882 |  3.75939 | -10.18663 |   5.3459\n\n```\n\n\n# Notes\n\nEdit `bashrc` file to add the following:\n\n```sh\nexport SPARK_HOME=/opt/spark/spark-3.5.3-bin-hadoop3/\nexport PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin\nexport PYSPARK_PYTHON=python3\n```\n\n# References\n\n- [Building A Structured Streaming Data Pipeline](https://medium.com/@zekeriyyademirci/apache-spark-structured-streaming-from-kafka-to-cassandra-2f373b699844)\n- [PySpark-Structured-Streaming-ROS-Kafka-ApacheSpark-Cassandra](https://github.com/zekeriyyaa/PySpark-Structured-Streaming-ROS-Kafka-ApacheSpark-Cassandra)\n- [Kafka Consumers in Python - A Walkthrough](https://youtu.be/eCsSAzTy5cE?si=n_RzcimlVcfH0th1)\n- [High Performance Kafka Producers in Python](https://youtu.be/mdhEXg5Pny8?si=mX5_uUKvB5oxIsKp)\n- [Kafka Stream Processing with Python - A Walkthrough](https://youtu.be/5sqegy_EPa0?si=jcpRQYYXaCZDzING)\n- [A Simple Kafka and Python Walkthrough](https://youtu.be/D2NYvGlbK0M?si=IBXmYqvkL5Lojo9D)\n- [Quix Streams](https://quix.io/docs/get-started/welcome.html)\n- [100% Python Stream Processing for Apache Kafka](https://github.com/quixio/quix-streams)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvishalgattani%2Fquixotic-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvishalgattani%2Fquixotic-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvishalgattani%2Fquixotic-kafka/lists"}