{"id":13541294,"url":"https://github.com/trK54Ylmz/kafka-spark-streaming-example","last_synced_at":"2025-04-02T08:31:02.966Z","repository":{"id":77117476,"uuid":"79672869","full_name":"trK54Ylmz/kafka-spark-streaming-example","owner":"trK54Ylmz","description":"Simple examle for Spark Streaming over Kafka topic","archived":false,"fork":false,"pushed_at":"2020-10-13T10:07:14.000Z","size":299,"stargazers_count":106,"open_issues_count":1,"forks_count":78,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-01T09:05:12.458Z","etag":null,"topics":["java","kafka","spark","stream-processing"],"latest_commit_sha":null,"homepage":"https://medium.com/@trK54Ylmz/real-time-dashboard-with-kafka-and-spark-streaming-53fd1f016249","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trK54Ylmz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-01-21T21:05:33.000Z","updated_at":"2025-01-03T21:45:50.000Z","dependencies_parsed_at":"2023-03-12T00:05:38.030Z","dependency_job_id":null,"html_url":"https://github.com/trK54Ylmz/kafka-spark-streaming-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trK54Ylmz%2Fkafka-spark-streaming-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trK54Ylmz%2Fkafka-spark-streaming-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trK54Ylmz%2Fkafka-spark-streaming-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trK54Ylmz%2Fkafka-spark-streaming-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trK54Ylmz","download_url":"https://codeload.github.com/trK54Ylmz/kafka-spark-streaming-example/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246781979,"owners_count":20832948,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java","kafka","spark","stream-processing"],"created_at":"2024-08-01T10:00:43.235Z","updated_at":"2025-04-02T08:31:02.405Z","avatar_url":"https://github.com/trK54Ylmz.png","language":"Java","funding_links":[],"categories":["Uncategorized"],"sub_categories":["Uncategorized"],"readme":"# kafka \u0026middot; spark streaming example\n\n[![Build Status](https://travis-ci.org/trK54Ylmz/kafka-spark-streaming-example.svg?branch=master)](https://travis-ci.org/trK54Ylmz/kafka-spark-streaming-example)\n\nThis is a simple dashboard example on Kafka and Spark Streaming\n\n### Prerequisites\n\nJava 1.8 or newer version required because lambda expression used for few cases\n\n1. Java \u003e= 1.8 (Oracle JDK has been tested)\n2. Maven \u003e= 3\n3. Apache Spark \u003e= 2.0.2\n4. Kafka \u003e= 0.10.1.0\n\n### Installation\n\nFirst of all, clone the git repository,\n\n```bash\n$ git clone git@github.com:trK54Ylmz/kafka-spark-streaming-example.git\n```\n\nafter you need to use Maven for creating uber jar files,\n\n```bash\n$ mvn clean package -DskipTests\n```\n\nuntil that moment we had created jar files and now we'll install Kafka and MySQL,\n\n```bash\n$ wget http://www-us.apache.org/dist/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz\n$ # or wget http://www-eu.apache.org/dist/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz\n$ tar -xf kafka_2.11-0.10.1.0.tgz\n$ cd kafka_2.11-0.10.1.0\n$ nohup ./bin/zookeeper-server-start.sh ./config/zookeeper.properties \u003e /tmp/kafka-zookeeper.out 2\u003e\u00261 \u0026\n$ nohup ./bin/kafka-server-start.sh ./config/server.properties \u003e /tmp/kafka-server.out 2\u003e\u00261 \u0026\n```\n\nKafka is ready we can continue to install MySQL,\n\n```bash\n$ sudo apt-get install mysql-server # for Ubuntu, Debian\n$ sudo yum install mysql-server \u0026\u0026 sudo systemctl start mysqld # for CentOS, RHEL\n$ brew install mysql \u0026\u0026 mysql.server restart # for macOS\n```\n\nand finally create MySQL database and table,\n\n```sql\nCREATE DATABASE IF NOT EXISTS dashboard_test;\n\nUSE dashboard_test;\n\nCREATE TABLE IF NOT EXISTS events (\n market VARCHAR(24) NOT NULL DEFAULT '',\n rate FLOAT DEFAULT NULL,\n dt DATETIME NOT NULL,\n PRIMARY KEY (market, dt)\n)\nENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;\n```\n\n### Usage\n\n1 - Start the Spark streaming service and it'll process events from Kafka topic to MySQL,\n\n```bash\n$ cd kafka-spark-streaming-example\n$ java -Dconfig=./config/common.conf -jar streaming/target/spark-streaming-0.1.jar\n```\n\n2 - Start the Kafka producer and it'll write events to Kafka topic,\n\n```bash\n$ java -Dconfig=./config/common.conf -jar producer/target/kafka-producer-0.1.jar\n```\n\n3 - Start the web server so you can see the dashboard\n\n```bash\n$ java -Dconfig=./config/common.conf -jar web/target/web-0.1.jar\n```\n\n4 - If everything look fine, please enter the dashboard address,\n\n```bash\nopen http://localhost:8080 # default value : 8080\n```\n\n![screen](screen.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FtrK54Ylmz%2Fkafka-spark-streaming-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FtrK54Ylmz%2Fkafka-spark-streaming-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FtrK54Ylmz%2Fkafka-spark-streaming-example/lists"}