{"id":30684337,"url":"https://github.com/chimpler/blog-spark-streaming-log-aggregation","last_synced_at":"2026-03-06T20:02:47.311Z","repository":{"id":18056429,"uuid":"21111823","full_name":"chimpler/blog-spark-streaming-log-aggregation","owner":"chimpler","description":"Example of use of Spark Streaming with Kafka","archived":false,"fork":false,"pushed_at":"2014-07-11T14:03:27.000Z","size":278,"stargazers_count":90,"open_issues_count":2,"forks_count":50,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-03-26T23:58:55.835Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chimpler.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-06-23T02:40:52.000Z","updated_at":"2024-03-26T23:58:55.836Z","dependencies_parsed_at":"2022-08-04T22:15:38.281Z","dependency_job_id":null,"html_url":"https://github.com/chimpler/blog-spark-streaming-log-aggregation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/chimpler/blog-spark-streaming-log-aggregation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chimpler%2Fblog-spark-streaming-log-aggregation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chimpler%2Fblog-spark-streaming-log-aggregation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chimpler%2Fblog-spark-streaming-log-aggregation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chimpler%2Fblog-spark-streaming-log-aggregation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chimpler","download_url":"https://codeload.github.com/chimpler/blog-spark-streaming-log-aggregation/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chimpler%2Fblog-spark-streaming-log-aggregation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273183239,"owners_count":25059814,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-01T20:15:56.148Z","updated_at":"2026-03-06T20:02:42.254Z","avatar_url":"https://github.com/chimpler.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"Simple example consuming an adserver logs stream from Kafka.\n\nMore information on our blog: http://chimpler.wordpress.com/2014/07/01/implementing-a-real-time-data-pipeline-with-spark-streaming/\n\nIn order to run our example, we need to install the followings:\n\n* [Scala 2.10+](http://www.scala-lang.org/)\n* [SBT](http://www.scala-sbt.org/)\n* [Apache Zookeeper](http://zookeeper.apache.org/)\n* [Apache Kafka](http://kafka.apache.org/)\n* [MongoDB](http://www.mongodb.org/)\n\n\nBuilding the examples:\n    \n    $ sbt pack\n\nCreate a topic “adnetwork-topic”:\n    \n    $ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic adnetwork-topic\n    \nStart Zookeeper:\n   \n    $ zookeeper-server-start.sh config/zookeeper.properties\n    \nStart Kafka:\n\n    $ kafka-server-start.sh config/kafka-server1.properties\n\nRun MongoDB:\n    $ sudo mongod\n    \nOn one window, run the aggregator:\n\n    $ target/pack/bin/aggregator\n\nOn the other one, run the adserver log random generator:\n\n    $ target/pack/bin/generator\n    \nYou can also see the messages that are sent using the Kafka console consumer:\n\n    $ kafka-console-consumer.sh --topic adnetwork-topic --zookeeper localhost:2181\n    \nAfter a few seconds, you should see the results in MongoDB:\n\n    $ mongoexport -d adlogdb -c impsPerPubGeo --csv -f date,publisher,geo,imps,uniques,avgBids\n    connected to: 127.0.0.1\n     \n    date,publisher,geo,imps,uniques,avgBids\n    2014-07-01T03:24:39.679Z,\"publisher_4\",\"CA\",3980,3248,0.50062253292876\n    2014-07-01T03:24:39.681Z,\"publisher_4\",\"MI\",3958,3229,0.505213545705667\n    2014-07-01T03:24:39.681Z,\"publisher_1\",\"HI\",3886,3218,0.4984981221446526\n    2014-07-01T03:24:39.681Z,\"publisher_3\",\"CA\",3937,3226,0.5038157362872939\n    2014-07-01T03:24:39.679Z,\"publisher_4\",\"NY\",3894,3200,0.5022389599376207\n    2014-07-01T03:24:39.679Z,\"publisher_2\",\"HI\",3906,3240,0.4988378174961185\n    2014-07-01T03:24:39.679Z,\"publisher_3\",\"HI\",3989,3309,0.4975347625823641\n    2014-07-01T03:24:39.681Z,\"publisher_3\",\"FL\",3957,3167,0.4993339490605483\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchimpler%2Fblog-spark-streaming-log-aggregation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchimpler%2Fblog-spark-streaming-log-aggregation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchimpler%2Fblog-spark-streaming-log-aggregation/lists"}