{"id":19506777,"url":"https://github.com/calvinlfer/streaming-analytics","last_synced_at":"2026-05-16T10:31:25.185Z","repository":{"id":68736776,"uuid":"122863812","full_name":"calvinlfer/streaming-analytics","owner":"calvinlfer","description":"A streaming data pipeline to perform basic analytics with scalability in mind","archived":false,"fork":false,"pushed_at":"2018-02-25T23:02:37.000Z","size":268,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-05T19:56:45.314Z","etag":null,"topics":["akka-http","akka-streams","cassandra","hyperloglog","kafka","scala"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/calvinlfer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-25T18:31:16.000Z","updated_at":"2021-04-03T06:56:46.000Z","dependencies_parsed_at":"2023-03-14T21:45:42.990Z","dependency_job_id":null,"html_url":"https://github.com/calvinlfer/streaming-analytics","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/calvinlfer/streaming-analytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2Fstreaming-analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2Fstreaming-analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2Fstreaming-analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2Fstreaming-analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/calvinlfer","download_url":"https://codeload.github.com/calvinlfer/streaming-analytics/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2Fstreaming-analytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33098787,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T04:41:52.686Z","status":"ssl_error","status_checked_at":"2026-05-16T04:41:52.009Z","response_time":115,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["akka-http","akka-streams","cassandra","hyperloglog","kafka","scala"],"created_at":"2024-11-10T22:38:17.767Z","updated_at":"2026-05-16T10:31:25.168Z","avatar_url":"https://github.com/calvinlfer.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simple Analytics #\n\nThis system provides simple data analytics for a website. This system is able to answer the following \nquestions:\n\n- How many unique visitors have visited my site per hour?\n- How many clicks per hour?\n- How many impressions per hour?\n\n## Running the system ##\n\nThe system is comprised of three distinct modules:\n\n- __frontend__: provides ingestion and querying endpoints\n- __clicks-and-impressions-stream-processor__: responsible for consuming click and impression events from the journal \nand aggregating them and pushing them to Cassandra\n- __unique-users-stream-processor__: responsible for consuming events from the journal, aggregating them and performing \ncardinality estimation using HyperLogLog and pushing them to Cassandra\n\n### Infrastructure ###\n\nThis system makes use of Cassandra to store analytics and Kafka for firehosing data and as a distribution mechanism for\ndecoupling modules. In order to spin up these dependencies, you can use\n```bash\ndocker-compose up\n```\n\nWhen you want to stop the infrastructure services:\n```bash\ndocker-compose down\n```\n\n__Note:__ Make sure your Docker VM has at least 4GB of RAM allocated as both containers consume quite a bit of memory and will exit with code 137 if you do not provide enough\n\n#### Production-like setup ####\nNow, we will package up the different components of the system to be used in a production-like environment:\n```bash\nsbt universal:packageBin\n```\n\nIn each module's `target` folder, you will find a `universal` folder containing a `zip` artifact.\nFor example, if you want to run the `frontend` component:\n```bash\ncd frontend/target/universal\nunzip frontend-0.1.0-SNAPSHOT.zip\ncd frontend-0.1.0-SNAPSHOT\n./bin/frontend\n```\n\nRun all the different components that make up the system:\n\n- `frontend`\n- `clicks-and-impressions-stream-processor`\n- `unique-users-stream-processor`\n\nEnvironment variables can be found in the `application.conf` of each module should you need to override any configuration.\n\n#### Development setup ####\nIf you are using an IDE like IntelliJ, go to each module and run `Main`. It is set up to talk to the infrastructure running locally on your machine that was created by Docker Compose.\n\n## Notes ##\n\n![](visual.png)\n\nWhat are my queries?\n\n- Unique number of users by timestamp\n- Number of clicks by timestamp\n- Number of impressions by timestamp\n\nNote: to clarify when I say timestamp, I really mean year-month-date-hour\n\nSome key points to note:\nThe GET request provides a timestamp, the period that they are looking at is 1 hour. So they want per-hour statistics \nfor each metric.\n\nSo if the query comes in for `GET /analytics?timestamp=1519152523`\nSo the time asked for is `20/02/2018 6:48 PM` so you would need to report on metrics for `20/02/2018` `6:00PM` to `7:00PM`\n\nLet's say we partitioned data by the hour, and we did at hourly intervals (so 1AM to 2AM, 2AM to 3AM and so on) then \nif someone asked for a query at that particular time (6:48 PM), we would report the 6PM-7PM interval.\n\nStoring clicks and impression data can be done nicely with the help of Cassandra's Counter data-type.\n\nTrying to figure out unique users is definitely more tricky. A naïve approach to doing this could be to store every \nuser that visited the site.\n\nBasically store all the user IDs in a thick partition and look them up to determine if they are already there. \nWhen you want to find out how many unique visitors you have, you do a sum of the rows. This has some pretty serious \ndrawbacks if you get a large number of visitors and something will definitely break since a partition can hold up to a \nmaximum of 2 billion rows. So if you get more than 2 billion, you will blow up the Cassandra partition.\n\nI actually did some research on this and what we are trying to do is estimate data cardinality (i.e. how many unique \nentries of data are present). We don't need to be 100% accurate, we just need a good estimate and this is where \nalgorithms like HyperLogLog and the Linear Probabilistic Counter come into play. They are able to provide you a great \nestimate using only 1/1000 of the space needed by the actual data and giving you error percentage within 10%. The more \nspace you use for these data structures, the lower the error rate.\n\nWe'll take a simple per hour approach of bucketing data. So every hour, we will have a new partition. However, we'll go \nwith skinny partitions. So to figure out the number of unique usernames per hour, we'll make use of __HyperLogLog__ (HLL). \n\nOne problem with the table storing HLLs is that two updates A and B can come at the same time where they both read data \nX and they both change the HLL based on data X resulting in A' and B' and they persist the changes one after the other \nresulting in only one of the updates being persisted (either A' or B') when in fact we should have had the sum of all \nchanges where B sees A' and takes that into account.\n\nDatabases like Riak have the concept of Application Side Conflict Resolution where the database will capture concurrent \nupdates to a key and will present them to the application (aka us) to resolve. In our case, we are dealing with an HLL \nthat has the options to do unions (it's actually a Monoid, take a look at Algebird's implementation) so we could easily \nmerge the HLL representations that the database presents us and solve this ourselves. Note that Cassandra only supports \nLast-Write-Wins semantics so you cannot use Cassandra to do this. In order to get around this you would need a read-write \ntransaction.\n\nHowever, this isn’t good enough because if concurrent updates happen, they are not automatically linearized, they are \nactually denied and you will have to retry them manually. \n\nThe Stream Processor component is implemented using Akka Streams and Akka Streams Kafka. Essentially this component \ntakes a batch of events (bigger the better) and condenses this down to a single event. This makes use of HLL to track \nunique counts because I'm processing the data as it comes in and I don't plan on processing data records for the hour \nall in memory unless I had something like a Spark Cluster. So you still end up having the same tables in Cassandra as \nabove, we just do some optimizations to batch data (to minimize writes) and avoid getting into (as many) concurrency \nproblems as we did before.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcalvinlfer%2Fstreaming-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcalvinlfer%2Fstreaming-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcalvinlfer%2Fstreaming-analytics/lists"}