{"id":18483628,"url":"https://github.com/ethicalml/kafka-spark-streaming-zeppelin-docker","last_synced_at":"2025-07-22T08:03:00.052Z","repository":{"id":41526955,"uuid":"184081275","full_name":"EthicalML/kafka-spark-streaming-zeppelin-docker","owner":"EthicalML","description":"One click deploy docker-compose with Kafka, Spark Streaming, Zeppelin UI and Monitoring (Grafana + Kafka Manager)","archived":false,"fork":false,"pushed_at":"2021-07-20T21:44:22.000Z","size":1042,"stargazers_count":120,"open_issues_count":4,"forks_count":74,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-09T15:00:49.486Z","etag":null,"topics":["docker","docker-compose","kafka","kafka-spark","kafka-spark-streaming","kafka-zeppelin","spark","spark-kafka","spark-streaming-kafka","spark-zeppelin","streaming","zeppelin"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EthicalML.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-29T13:58:08.000Z","updated_at":"2025-03-02T11:56:57.000Z","dependencies_parsed_at":"2022-09-15T03:02:41.374Z","dependency_job_id":null,"html_url":"https://github.com/EthicalML/kafka-spark-streaming-zeppelin-docker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/EthicalML/kafka-spark-streaming-zeppelin-docker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EthicalML%2Fkafka-spark-streaming-zeppelin-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EthicalML%2Fkafka-spark-streaming-zeppelin-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EthicalML%2Fkafka-spark-streaming-zeppelin-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EthicalML%2Fkafka-spark-streaming-zeppelin-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EthicalML","download_url":"https://codeload.github.com/EthicalML/kafka-spark-streaming-zeppelin-docker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EthicalML%2Fkafka-spark-streaming-zeppelin-docker/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266454598,"owners_count":23931331,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","kafka","kafka-spark","kafka-spark-streaming","kafka-zeppelin","spark","spark-kafka","spark-streaming-kafka","spark-zeppelin","streaming","zeppelin"],"created_at":"2024-11-06T12:36:39.167Z","updated_at":"2025-07-22T08:02:59.948Z","avatar_url":"https://github.com/EthicalML.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"![GitHub](https://img.shields.io/badge/Release-PROD-green.svg)\n![GitHub](https://img.shields.io/badge/Version-0.0.1-lightgrey.svg)\n![GitHub](https://img.shields.io/badge/License-MIT-blue.svg)\n\n# One Click Deploy: Kafka Spark Streaming with Zeppelin UI\n\nThis repository contains a docker-compose stack with Kafka and Spark Streaming, together with monitoring with Kafka Manager and a Grafana Dashboard. The networking is set up so Kafka brokers can be accessed from the host.\n\nIt also comes with a producer-consumer example using a small subset of the [US Census adult income prediction dataset](https://www.kaggle.com/johnolafenwa/us-census-data).\n\n## High level features:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd style=\"width: 50%\"\u003e\n\u003ch2\u003eMonitoring with grafana\u003c/h2\u003e\n\u003cimg src=\"images/grafanakafka.jpg\" alt=\"\"\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003ch2\u003eZeppelin UI\u003c/h2\u003e\n\u003cimg src=\"images/zeppelin-2.jpg\" alt=\"\"\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctd style=\"width: 50%\"\u003e\n\u003ch2\u003eKafka access from host\u003c/h2\u003e\n\u003cimg src=\"images/console.jpg\" alt=\"\"\u003e\n\u003c/td\u003e\n\u003ctd style=\"width: 50%\"\u003e\n\u003ch2\u003eMultiple spark interpreters\u003c/h2\u003e\n\u003cimg src=\"images/sparkui.jpg\" alt=\"\"\u003e\n\u003c/td\u003e\n\u003c/table\u003e\n\n## Detail Summary\n\n| Container | Image | Tag | Accessible | \n|-|-|-|-|\n| zookeeper | wurstmeister/zookeeper | latest | 172.25.0.11:2181 |\n| kafka1 | wurstmeister/kafka | 2.12-2.2.0 | 172.25.0.12:9092 (port 8080 for JMX metrics) |\n| kafka1 | wurstmeister/kafka | 2.12-2.2.0 | 172.25.0.13:9092 (port 8080 for JMX metrics) |\n| kafka_manager | hlebalbau/kafka_manager | 1.3.3.18 | 172.25.0.14:9000 |\n| prometheus | prom/prometheus | v2.8.1 | 172.25.0.15:9090 |\n| grafana | grafana/grafana | 6.1.1 | 172.25.0.16:3000 |\n| zeppelin | apache/zeppelin | 0.8.1 | 172.25.0.19:8080 |\n\n# Quickstart\n\nThe easiest way to understand the setup is by diving into it and interacting with it.\n\n## Running Docker Compose\n\nTo run docker compose simply run the following command in the current folder:\n\n```\ndocker-compose up -d\n```\n\nThis will run deattached. If you want to see the logs, you can run:\n\n```\ndocker-compose logs -f -t --tail=10\n```\n\nTo see the memory and CPU usage (which comes in handy to ensure docker has enough memory) use:\n\n```\ndocker stats\n```\n\n## Accessing the notebook\n\nYou can access the default notebook by going to http://172.25.0.19:8080/#/notebook/2EAB941ZD. Now we can start running the cells.\n\n### 1) Setup\n\n#### Install python-kafka dependency\n\n![](images/zeppelin-1.jpg)\n\n### 2) Producer\n\nWe have an interpreter called %producer.pyspark that we'll be able to run in parallel.\n\n#### Load our example dummy dataset\n\nWe have made available a 1000-row version of the [US Census adult income prediction dataset](https://www.kaggle.com/johnolafenwa/us-census-data).\n\n![](images/zeppelin-2.jpg)\n\n#### Start the stream of rows\n\nWe now take one row at random, and send it using our python-kafka producer. The topic will be created automatically if it doesn't exist (given that `auto.create.topics.enable` is set to true).\n\n![](images/zeppelin-3.jpg)\n\n### 3) Consumer\n\nWe now use the %consumer.pyspark interpreter to run our pyspark job in parallel to the producer.\n\n#### Connect to the stream and print\n\nNow we can run the spark stream job to connect to the topic and listen to data. The job will listen for windows of 2 seconds and will print the ID and \"label\" for all the rows within that window.\n\n![](images/zeppelin-4.jpg)\n\n### 4) Monitor Kafka\n\nWe can now use the kafka manager to dive into the current kafka setup.\n\n#### Setup Kafka Manager\n\nTo set up kafka manager we need to configure it. In order to do this, access http://172.25.0.14:9000/addCluster and fill up the following two fields:\n\n* Cluster name: Kafka\n* Zookeeper hosts: 172.25.0.11:2181\n\nOptionally:\n* You can tick the following;\n    * Enable JMX Polling\n    * Poll consumer information\n\n#### Access the topic information\n\nIf your cluster was named \"Kafka\", then you can go to http://172.25.0.14:9000/clusters/Kafka/topics/default_topic, where you will be able to see the partition offsets. Given that the topic was created automatically, it will have only 1 partition.\n\n![](images/zeppelin-4.jpg)\n\n#### Visualise metrics in Grafana\n\nFinally, you can access the default kafka dashboard in Grafana (username is \"admin\" and password is \"password\") by going to http://172.25.0.16:3000/d/xyAGlzgWz/kafka?orgId=1\n\n![](images/grafanakafka.jpg)\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fethicalml%2Fkafka-spark-streaming-zeppelin-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fethicalml%2Fkafka-spark-streaming-zeppelin-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fethicalml%2Fkafka-spark-streaming-zeppelin-docker/lists"}