{"id":20121882,"url":"https://github.com/peregin/iot-spark-rest","last_synced_at":"2026-05-15T07:03:46.739Z","repository":{"id":70392393,"uuid":"101467201","full_name":"peregin/iot-spark-rest","owner":"peregin","description":"Example with kafka, spark, rest service","archived":false,"fork":false,"pushed_at":"2018-11-16T11:32:26.000Z","size":264,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-13T07:26:52.064Z","etag":null,"topics":["iot","kafka","rest","scalatra","spark","streaming"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/peregin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-26T06:21:46.000Z","updated_at":"2023-02-03T09:17:34.000Z","dependencies_parsed_at":"2023-07-14T22:45:57.068Z","dependency_job_id":null,"html_url":"https://github.com/peregin/iot-spark-rest","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peregin%2Fiot-spark-rest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peregin%2Fiot-spark-rest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peregin%2Fiot-spark-rest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peregin%2Fiot-spark-rest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/peregin","download_url":"https://codeload.github.com/peregin/iot-spark-rest/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241565549,"owners_count":19983142,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["iot","kafka","rest","scalatra","spark","streaming"],"created_at":"2024-11-13T19:33:22.985Z","updated_at":"2026-05-15T07:03:46.649Z","avatar_url":"https://github.com/peregin.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IOT data processing with Spark\n\nSimple example for IOT processing.\nIt collects json data from vehicles, it publishes to Kafka, Spark aggregates the data and sends it to a REST service.\n\n```text\n  --------------------        --------        ----------        ---------\n |                    |      |        |      |          |      |         |\n | PublishMessagesApp | ---\u003e |   ZK   | ---\u003e | SparkApp | ---\u003e | RestApp |\n |  VehicleData json  |      | Kafka  |      |          |      | Counter |\n |                    |      |        |      |          |      |         |\n  --------------------        --------        ----------        ---------\n```\n\n## Modules\n\n### iot-data\nThis module is shared with the rest of the modules, it provides the scala model of the ```VehicleData``` and ```SummaryData```.\nAlso provides a utility to convert model to json format and vice versa.\nIn the test resources I was reusing some of my cycling activities recorded with a GPS device to generate test input for Kafka.\n\n### kafka-producer\nIt creates the topic if doesn't exists yet and it will publish ```VehicleData``` in json format to the Kafka broker.\nTo have a fully functional setup I was using spotify/kafka Docker image to run the broker locally on my computer.\nRun it as: ```iot.publish.PublishMessagesApp```\n\n### rest-service\nIt exposes the 'summary' endpoint via a rest service. It logs the incoming messages (```SummaryData``` type) and counts them.\nIf you hit localhost:8888 on the browser, the GET will response with the value of the counter.\nRun it as: ```RestApp```\n\n### spark-processor\nIt reads the input messages from the kafka topic as a stream and it processes it. The results are posted to a\nREST service running locally.\nThe default values (e.g. iot.kafka.topic, iot.rest.endpoint) can be overridden via system properties.\nRun it as: ```iot.spark.SparkApp```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperegin%2Fiot-spark-rest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperegin%2Fiot-spark-rest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperegin%2Fiot-spark-rest/lists"}