{"id":28566618,"url":"https://github.com/orhanbalci/streaming-challenge","last_synced_at":"2026-04-12T12:38:07.225Z","repository":{"id":113617275,"uuid":"188449419","full_name":"orhanbalci/streaming-challenge","owner":"orhanbalci","description":"This project aims to demonstrate log streaming capabilities of scala frameworks such as Spark, Akka-Streaming and Akka-Http ","archived":false,"fork":false,"pushed_at":"2019-06-02T22:35:49.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-10T15:11:34.063Z","etag":null,"topics":["akka","akka-http","akka-streams","docker","docker-compose","kafka","log","scala","spark","streaming"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orhanbalci.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-05-24T15:50:15.000Z","updated_at":"2019-07-26T10:00:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"764dcddd-043d-4925-a6bd-26883192cdee","html_url":"https://github.com/orhanbalci/streaming-challenge","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/orhanbalci/streaming-challenge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orhanbalci%2Fstreaming-challenge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orhanbalci%2Fstreaming-challenge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orhanbalci%2Fstreaming-challenge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orhanbalci%2Fstreaming-challenge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orhanbalci","download_url":"https://codeload.github.com/orhanbalci/streaming-challenge/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orhanbalci%2Fstreaming-challenge/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261495514,"owners_count":23167336,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["akka","akka-http","akka-streams","docker","docker-compose","kafka","log","scala","spark","streaming"],"created_at":"2025-06-10T15:11:24.649Z","updated_at":"2026-04-12T12:38:02.205Z","avatar_url":"https://github.com/orhanbalci.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Log Streaming POC\nThis project aims to demonstrate log streaming capabilities of scala projects such as Spark, Akka-Streaming and Akka-Http\nAll components of the project are dockerized and can be run easily on your local machine.\n\n# Prerequisites\n\n* Install docker on your machine\n* Install docker-compose\n* Install psql or equivalent postgresql db query tool\n* Install kafkacat or equivalent kafka querying tool\n* Install any websocket client\n* Install git client \n\n# How to run the systems\n\n* Get the repository from github. \n```\ngit clone https://github.com/orhanbalci/streaming-challenge\n```\n\n* Run system using \n```\n./up.sh\n```\n\n* You can inspect messages on kafka using kafkacat. There are two topics. First one is logs\nwhich contains json structured raw log messages. Second one is log_totals which includes\n10 minutes windowed log totals per log center. \n```\nkafkacat -C -b localhost -t logs\nkafkacat -C -b localhost -t log_totals\n```\n\n* You can inspect logs written to postgresql db using tsql. Default database name is log_db\nand table name is logs\n```\npsql -h localhost -U postgres log_db\nselect * from logs;\n```\n\n# Components of the system\n\n## log-generator\ngenerates fake log files and writes them into folder ./logsthis folder is mounted on docker host also\n\n## zookeeper \nneeded for kafka. Although kafka runs in single mode zookeeper is needed\n\n## kafka \nkafka message broker\n\n## log-pusher \nwatches foolder changes on logs folder. Reads log reacords and pushes them to kafka on logs topic. This module uses Apache Spark framework structured streaming capabilities\n\n## log-db \npostgresql db. Default db is log_db. Logs are written into logs table\n\n## log-puller \nreads messages from kafka logs topic and records them on postgres db. uses akka-streams and alpakka kafka source plus alpakka slick database sink\n\n## log-transformer \nreads messages from kafka logs topics and groups them according to 10 minutes time window. This module uses spark structured streaming capabilities\n\n## log-http-service \nuses akka-http websocket technology to serve messages from kafka log_totals topic. you can query log totals\nfrom this service using adress ws://localhost:8282/logs\n\n## log-dashboard\ncontains web application with a single page served using nginx. Opens websocket connection to log-gttp-services. \nYou can view web page using http://localhost:8585/log-dashboard.html\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forhanbalci%2Fstreaming-challenge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forhanbalci%2Fstreaming-challenge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forhanbalci%2Fstreaming-challenge/lists"}