{"id":22178352,"url":"https://github.com/newfront/odsc-west-streaming-trends","last_synced_at":"2025-08-12T15:18:16.608Z","repository":{"id":138838413,"uuid":"151772088","full_name":"newfront/odsc-west-streaming-trends","owner":"newfront","description":"All Data, Relevant Information, Scripts, and Applications for the Open Data Science Conference (2018)","archived":false,"fork":false,"pushed_at":"2018-11-02T05:08:24.000Z","size":88166,"stargazers_count":11,"open_issues_count":0,"forks_count":13,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-03T08:12:01.080Z","etag":null,"topics":["ml","spark","spark-streaming"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/newfront.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-10-05T20:04:06.000Z","updated_at":"2023-06-09T05:08:14.000Z","dependencies_parsed_at":"2023-03-13T21:30:54.295Z","dependency_job_id":null,"html_url":"https://github.com/newfront/odsc-west-streaming-trends","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/newfront/odsc-west-streaming-trends","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newfront%2Fodsc-west-streaming-trends","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newfront%2Fodsc-west-streaming-trends/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newfront%2Fodsc-west-streaming-trends/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newfront%2Fodsc-west-streaming-trends/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/newfront","download_url":"https://codeload.github.com/newfront/odsc-west-streaming-trends/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newfront%2Fodsc-west-streaming-trends/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270083420,"owners_count":24523927,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ml","spark","spark-streaming"],"created_at":"2024-12-02T08:46:16.240Z","updated_at":"2025-08-12T15:18:16.553Z","avatar_url":"https://github.com/newfront.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"### AUTOMATING TREND DISCOVERY ON STREAMING DATASETS WITH SPARK 2.3\nAll Data, Relevant Information, Scripts, and Applications for the Open Data Science Conference (2018)\n\n#### The Speaker\n**Scott Haines**\nI work at [Twilio](https://www.twilio.com/)\n\nPlaces I exist online\n* [Github - newfront](https://github.com/newfront)\n* [Medium](https://medium.com/@newfrontcreative)\n* [Twitter](https://twitter.com/newfront)\n* [LinkedIn](https://www.linkedin.com/in/scotthaines/)\n\n#### Getting Started\n1. Download and Install Spark (http://spark.apache.org/downloads.html) or https://www.apache.org/dyn/closer.lua/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz \n\n2. Have maven 3 installed. `brew install maven3` if you want to build the Streaming Trend Discovery code\n\n#### Data Set Information\n[Wine Reviews](https://www.kaggle.com/zynicide/wine-reviews) - Thanks to zynicide and kaggle.com for the data set.\n\n### Running the Code Examples\nAll actions should be run from root of the odsc directory\n\n#### Playing with Coffee\n1. `spark-shell -i part2/coffee/basics.scala`\n2. `spark-shell -i part2/coffee/dataframes.scala`\n\n3. Requires 2 terminal windows\n* 3a. `nc -lk 9999`\n* 3b. `spark-shell -i part2/streaming_coffee.scala`\n\n##### Streaming Aggregations on Coffee Ratings\nNow in the terminal window (nc -lk 9999) just copy and paste each of the following lines. nc -lk takes stdin and spark will pick up from that socket connection.\n~~~\nfolgers,1\nfolgers,2,\"gross\"\nritual,5,\"awesome\"\nfour barrel,5,\"great\"\nfour barrel,5,\"great stuff\"\nfour barrel,5,\"really great stuff\"\n~~~\n\nIn the spark streaming coffee terminal you should see the following\n~~~\n-------------------------------------------                                     \nBatch: 0\n-------------------------------------------\n+-------+------+\n|   name|rating|\n+-------+------+\n|folgers|   1.0|\n+-------+------+\n\n-------------------------------------------                                     \nBatch: 1\n-------------------------------------------\n+-------+------+\n|   name|rating|\n+-------+------+\n|folgers|   1.5|\n+-------+------+\n\n-------------------------------------------                                     \nBatch: 2\n-------------------------------------------\n+-------+------------------+\n|   name|            rating|\n+-------+------------------+\n|folgers|2.6666666666666665|\n+-------+------------------+\n~~~\n\n#### Playing with Wine\n1. `cd data/winereviews \u0026\u0026 unzip winemag-csv.zip \u0026\u0026 unzip winemag-json.zip`\n2. `spark-shell -i part2/wine/hello-wine.scala`\n3. `spark-shell -i part2/wine/wine_reviews.scala`\n3. `spark-shell -i part2/wine/wine_reviews_json.scala`\n\n#### Lessons Learned\n* Wine Reviews JSON data is easier and better to work with\n* SparkML **StopWordsRemover** allows us to quickly remove common words from the wine reviews\n* SparkML **FPGrowth** allows us to quickly generate Frequent Item Lists using the [Apriori Algorithm](https://en.wikipedia.org/wiki/Apriori_algorithm)\n* Generating Tasting Notes from Wine can be easy with a little trial and error\n~~~scala\ndef tastingNotes(df: DataFrame): String = {\n  val fpg = new FPGrowth().setItemsCol(\"items\").setMinSupport(0.05).setMinConfidence(0.6)\n  val remover = new StopWordsRemover().setInputCol(\"items\").setOutputCol(\"filteredItems\")\n  \t\t\n  // Cleaning up the Wine Descriptions\n  val descriptions = df.select(col(\"description\")).where(col(\"description\").isNotNull).map { case Row(s:String) =\u003e s.replace(\",\",\"\").replace(\".\",\"\").split(\" \").toSet.toSeq }.toDF(\"items\")\n\n  // remove StopWords\n  val filteredDescriptions = remover.transform(descriptions)\n  val stopWordsFiltered = filteredDescriptions.select(\"filteredItems\").toDF(\"items\")\n\n  val model = fpg.fit(stopWordsFiltered)\n  val freqItems = model.freqItemsets.sort(desc(\"freq\"))\n  val notes = freqItems.select(\"items\").where(col(\"freq\")\u003e400)\n  val topWords = notes.flatMap { case Row(notes: Seq[String]) =\u003e notes }.groupBy(\"value\").count().sort(desc(\"count\"))\n  val tastingNotes = topWords.select(\"value\").collect().map { case Row(s: String) =\u003e s }.toSeq.mkString(\",\")\n  tastingNotes\n}\n~~~\n\nOutput when FPGrowth is run on topN varieties in the corpus of Wine Reviews\n~~~\nWine Variety: Pinot Noir\nTasting Notes: wine,flavors,cherry,fruit,Pinot,acidity,tannins,palate,raspberry,Noir,red,finish,ripe,oak,black,cola,aromas,Drink,spice,rich,dry,silky,texture,light,years,soft,fruits,nose,plum,structure,strawberry,juicy,character,complex,spicy,vanilla,vineyard,touch,new,earthy,cherries,bit,earth,cranberry,berry,fresh,dried,firm,flavor,dark,full,tea,age,sweet,notes,vintage,well,good,tart,drink,raspberries,crisp,shows,smoky,bottling,offers,bright\n\nWine Variety: Bordeaux-style Red Blend\nTasting Notes: wine,tannins,Cabernet,fruit,flavors,Merlot,Sauvignon,blend,ripe,fruits,Franc,acidity,black,Drink,rich,Petit,Verdot,juicy,structure,wood,firm,dry,currant,dark,well,aging,character,aromas,cherry,berry,blackberry,spice,structured,dense,drink,fruity,years,red,soft,still,ready,tannic,2017,fine,solid,2018,fresh,Malbec,Barrel,full,concentrated,age,attractive,texture,finish,sample,shows,balanced,core,balance,palate,oak,fruitiness,Bordeaux,chocolate,smooth,sweet,plum,notes,vintage,good,give,smoky,weight,also\n\nWine Variety: Riesling\nTasting Notes: flavors,palate,finish,Riesling,acidity,dry,wine,lemon,notes,peach,apple,fruit,nose,lime,aromas,fresh,long,sweet,citrus,ripe,Drink,juicy,honey,orange,green,refreshing,off-dry,stone,apricot,mineral,pear,grapefruit,white,zesty,freshness,concentrated,minerality,tangerine,yet,style,tart,crisp,fruity\n\nWine Variety: Sauvignon Blanc\nTasting Notes: flavors,wine,finish,aromas,palate,acidity,Blanc,Sauvignon,green,citrus,fruit,crisp,grapefruit,apple,fresh,lime,nose,texture,ripe,Drink,rich,tropical,dry,white,lemon,fruits,pineapple,clean,notes,melon,peach,drink\n\nWine Variety: Syrah\nTasting Notes: flavors,wine,Syrah,fruit,black,aromas,pepper,finish,tannins,palate,cherry,blackberry,acidity,Drink,meat,rich,berry,dark,oak,spice,chocolate,nose,plum,notes,shows,ripe\n~~~\n\n#### KMeans Clustering in WineReviews\nAfter exploring the data, seeing what is missing, and which values have the least number of distinct values (low cardinality), and after looking at auto-generative Tasting Notes. It came time to look at how to use KMeans clustering of the Wine Reviews dataset.\n\nThe idea was this. Given that Wine Reviews have scores (points), but also have a price, country of origin, and varitey of wine I thought it may be fun to create a range based value called **quality** so the clustering algorithm would have an additional help. Creating a range based value is a simple way of applying dimensional reduction to continious variables. In this case, we create a step based range vs having all values from **80-100** so less effort when clustering.\n\nFirst we removed all NULL values from the dataset. Then applied the `withColumn(\"quality\",...)` to create our range buckets.\n~~~\nval bucketing = wineReviewsJson.where(col(\"price\").isNotNull.and(col(\"points\").isNotNull.and(col(\"country\").isNotNull.and(col(\"variety\").isNotNull)))).withColumn(\"quality\", when(col(\"points\") \u003c 85, 0).when(col(\"points\") \u003c 90, 1).when(col(\"points\") \u003c 95, 2).otherwise(3))\n~~~\n\nThe KMeans Clustering Code\n~~~\ndef kmeansWine(df: DataFrame): DataFrame = {\n\t\tdf.cache()\n\t\t// note: indexers can't handle null values\n\t\tval indexerCountry = new StringIndexer().setInputCol(\"country\").setOutputCol(\"country_index\")\n\t\tval encoderCountry = new OneHotEncoder().setInputCol(\"country_index\").setOutputCol(\"country_encoded\")\n\t\tval indexerVariety = new StringIndexer().setInputCol(\"variety\").setOutputCol(\"variety_index\")\n\t\tval encoderVariety = new OneHotEncoder().setInputCol(\"variety_index\").setOutputCol(\"variety_encoded\")\n\t\tval wineVectorAssembler = new VectorAssembler().setInputCols(Array(\"country_encoded\", \"variety_encoded\", \"price\", \"points\", \"quality\")).setOutputCol(\"features\")\n\t\tval transformerPipeline = new Pipeline().setStages(Array(indexerCountry, encoderCountry, indexerVariety, encoderVariety, wineVectorAssembler))\n\t\tval fittedPipeline = transformerPipeline.fit(df)\n\t\tval Array(trainingData, testData) = df.randomSplit(Array(0.7, 0.3))\n\t\tval transformedTraining = fittedPipeline.transform(trainingData)\n\t\ttransformedTraining.cache()\n\t\tval kmeans = new KMeans().setK(6).setSeed(1L)\n\t\tval kmModel = kmeans.fit(transformedTraining)\n\t\tkmModel.computeCost(transformedTraining)\n\t\tval transformedTest = fittedPipeline.transform(testData)\n\t\ttransformedTest.cache()\n\t\tkmModel.computeCost(transformedTest)\n\t\tkmModel.transform(transformedTest)\n\t}\n~~~\n\nApplying this to WineReviews and looking at the end results\n~~~\nval wineClusters = WineUtils.kmeansWine(bucketing)\n(0 to 5).map { i =\u003e\n    val cluster = wineClusters.select(\"country\",\"price\",\"points\",\"winery\",\"variety\",\"title\",\"features\",\"prediction\").where(col(\"prediction\").equalTo(i))\n\tcluster\n\t\t.groupBy(\"country\")\n\t\t.agg(\n\t\t\tmin(\"price\"),\n\t\t\tavg(\"price\"),\n\t\t\tmax(\"price\"),\n\t\t\tcount(\"*\") as \"total\",\n\t\t\tcountDistinct(\"variety\") as \"varieties\",\n\t\t\tcountDistinct(\"winery\") as \"wineries\",\n\t\t\tlit(i) as \"prediction\"\n\t\t)\n\t\t.sort(desc(\"wineries\"))\n\t\t.show(50, false)\n}\n~~~\n\nLooking at the Clusters\n~~~\n+----------------------+----------+------------------+----------+-----+---------+--------+----------+\n|country               |min(price)|avg(price)        |max(price)|total|varieties|wineries|prediction|\n+----------------------+----------+------------------+----------+-----+---------+--------+----------+\n|US                    |4         |24.869311655032252|41        |11317|176      |3186    |0         |\n|France                |6         |21.427924528301887|41        |3975 |105      |1700    |0         |\n|Italy                 |5         |22.84746778543602 |41        |3337 |130      |1493    |0         |\n|Spain                 |4         |18.59246376811594 |41        |1725 |90       |819     |0         |\n|Argentina             |5         |17.73173076923077 |40        |1040 |46       |329     |0         |\n|Portugal              |6         |17.2031122031122  |40        |1221 |55       |259     |0         |\n|Australia             |5         |20.46313799621928 |40        |529  |48       |244     |0         |\n|Chile                 |6         |16.682076236820762|41        |1233 |45       |216     |0         |\n|New Zealand           |8         |21.768571428571427|41        |350  |22       |171     |0         |\n|Germany               |6         |22.517543859649123|41        |456  |24       |164     |0         |\n|Austria               |9         |23.15336463223787 |41        |639  |35       |158     |0         |\n|South Africa          |5         |19.05763688760807 |41        |347  |30       |154     |0         |\n|Greece                |8         |20.74561403508772 |40        |114  |30       |48      |0         |\n|Israel                |9         |23.06779661016949 |40        |118  |28       |29      |0         |\n|Canada                |12        |26.065217391304348|40        |46   |18       |22      |0         |\n|Hungary               |12        |22.25925925925926 |40        |27   |12       |18      |0         |\n|Mexico                |10        |22.37037037037037 |40        |27   |15       |16      |0         |\n|Bulgaria              |8         |13.5              |30        |44   |14       |15      |0         |\n|Slovenia              |7         |18.96153846153846 |40        |26   |13       |15      |0         |\n|Georgia               |10        |20.193548387096776|40        |31   |8        |14      |0         |\n|Croatia               |13        |21.57894736842105 |38        |19   |11       |14      |0         |\n|Uruguay               |10        |22.0              |40        |26   |9        |14      |0         |\n|Romania               |4         |11.0              |22        |32   |11       |12      |0         |\n|Turkey                |14        |23.8              |40        |20   |10       |9       |0         |\n|Moldova               |8         |15.722222222222221|38        |18   |11       |7       |0         |\n|Brazil                |12        |19.5              |31        |8    |6        |6       |0         |\n|Lebanon               |14        |26.625            |40        |8    |4        |4       |0         |\n|England               |25        |35.4              |40        |5    |2        |4       |0         |\n|Serbia                |15        |23.0              |40        |5    |5        |3       |0         |\n|Macedonia             |15        |15.0              |15        |8    |6        |3       |0         |\n|Cyprus                |11        |16.25             |20        |4    |3        |3       |0         |\n|Peru                  |14        |16.0              |17        |3    |3        |1       |0         |\n|Morocco               |14        |18.0              |25        |6    |3        |1       |0         |\n|India                 |10        |10.666666666666666|12        |3    |2        |1       |0         |\n|Switzerland           |21        |21.0              |21        |1    |1        |1       |0         |\n|Ukraine               |10        |10.0              |10        |1    |1        |1       |0         |\n|Luxembourg            |16        |20.666666666666668|23        |3    |2        |1       |0         |\n|Czech Republic        |15        |16.5              |18        |2    |2        |1       |0         |\n|Bosnia and Herzegovina|13        |13.0              |13        |1    |1        |1       |0         |\n+----------------------+----------+------------------+----------+-----+---------+--------+----------+\n\n/* Empty due to not being in the RandomSplit test set... */\n+-------+----------+----------+----------+-----+---------+--------+----------+  \n|country|min(price)|avg(price)|max(price)|total|varieties|wineries|prediction|\n+-------+----------+----------+----------+-----+---------+--------+----------+\n+-------+----------+----------+----------+-----+---------+--------+----------+\n\n+------------+----------+------------------+----------+-----+---------+--------+----------+\n|country     |min(price)|avg(price)        |max(price)|total|varieties|wineries|prediction|\n+------------+----------+------------------+----------+-----+---------+--------+----------+\n|US          |41        |57.713103756708406|100       |4472 |76       |1491    |2         |\n|Italy       |41        |62.29835831548893 |101       |1401 |63       |671     |2         |\n|France      |41        |62.15113122171946 |101       |1105 |38       |495     |2         |\n|Spain       |42        |62.053497942386834|100       |243  |31       |161     |2         |\n|Portugal    |42        |63.58720930232558 |100       |172  |15       |95      |2         |\n|Argentina   |41        |60.50819672131148 |100       |122  |15       |68      |2         |\n|Austria     |41        |58.583333333333336|100       |156  |23       |60      |2         |\n|Australia   |42        |66.78861788617886 |100       |123  |21       |59      |2         |\n|Germany     |41        |59.94039735099338 |101       |151  |7        |54      |2         |\n|Chile       |42        |67.38095238095238 |100       |84   |17       |52      |2         |\n|New Zealand |41        |58.71153846153846 |90        |52   |5        |29      |2         |\n|South Africa|42        |58.82051282051282 |100       |39   |13       |26      |2         |\n|Israel      |42        |59.58620689655172 |100       |29   |7        |16      |2         |\n|Canada      |45        |67.83333333333333 |95        |24   |10       |13      |2         |\n|England     |42        |61.5625           |95        |16   |3        |12      |2         |\n|Greece      |45        |51.285714285714285|59        |7    |4        |6       |2         |\n|Hungary     |55        |66.66666666666667 |75        |3    |3        |3       |2         |\n|Uruguay     |44        |46.333333333333336|50        |3    |2        |3       |2         |\n|Brazil      |45        |52.5              |60        |2    |2        |2       |2         |\n|Lebanon     |51        |63.0              |75        |3    |1        |2       |2         |\n|Slovenia    |50        |55.0              |60        |2    |2        |2       |2         |\n|Croatia     |57        |62.333333333333336|65        |3    |1        |2       |2         |\n|Turkey      |45        |50.5              |56        |2    |2        |2       |2         |\n|Mexico      |100       |100.0             |100       |1    |1        |1       |2         |\n|Romania     |58        |58.0              |58        |1    |1        |1       |2         |\n|Serbia      |42        |42.0              |42        |1    |1        |1       |2         |\n+------------+----------+------------------+----------+-----+---------+--------+----------+\n\n+------------+----------+------------------+----------+-----+---------+--------+----------+\n|country     |min(price)|avg(price)        |max(price)|total|varieties|wineries|prediction|\n+------------+----------+------------------+----------+-----+---------+--------+----------+\n|US          |102       |143.9725085910653 |250       |291  |18       |171     |3         |\n|France      |102       |140.88181818181818|249       |220  |20       |124     |3         |\n|Italy       |102       |143.47619047619048|252       |168  |16       |110     |3         |\n|Spain       |102       |146.6078431372549 |250       |51   |14       |36      |3         |\n|Portugal    |110       |155.47619047619048|250       |21   |5        |16      |3         |\n|Germany     |103       |149.8846153846154 |250       |26   |3        |15      |3         |\n|Australia   |104       |148.04            |250       |25   |11       |15      |3         |\n|Argentina   |110       |141.46153846153845|230       |26   |7        |13      |3         |\n|Austria     |115       |121.5             |126       |4    |4        |4       |3         |\n|New Zealand |120       |120.0             |120       |3    |1        |3       |3         |\n|Chile       |120       |166.33333333333334|235       |6    |4        |3       |3         |\n|South Africa|102       |120.5             |139       |2    |2        |2       |3         |\n|Hungary     |118       |139.33333333333334|175       |3    |1        |2       |3         |\n|Uruguay     |120       |120.0             |120       |1    |1        |1       |3         |\n|Canada      |120       |120.0             |120       |1    |1        |1       |3         |\n|Switzerland |160       |160.0             |160       |1    |1        |1       |3         |\n+------------+----------+------------------+----------+-----+---------+--------+----------+\n\n+------------+----------+------------------+----------+-----+---------+--------+----------+\n|country     |min(price)|avg(price)        |max(price)|total|varieties|wineries|prediction|\n+------------+----------+------------------+----------+-----+---------+--------+----------+\n|France      |257       |358.10169491525426|550       |59   |5        |31      |4         |\n|Italy       |255       |340.2             |540       |25   |8        |15      |4         |\n|Germany     |279       |365.64285714285717|500       |14   |2        |11      |4         |\n|US          |260       |333.6363636363636 |500       |11   |2        |9       |4         |\n|Portugal    |275       |371.5             |495       |4    |3        |4       |4         |\n|Spain       |300       |375.6666666666667 |500       |6    |3        |4       |4         |\n|Chile       |260       |286.6666666666667 |300       |3    |2        |2       |4         |\n|South Africa|275       |302.5             |330       |2    |2        |2       |4         |\n|Australia   |300       |325.0             |350       |2    |2        |1       |4         |\n|Romania     |320       |320.0             |320       |1    |1        |1       |4         |\n|Hungary     |320       |320.0             |320       |1    |1        |1       |4         |\n+------------+----------+------------------+----------+-----+---------+--------+----------+\n\n+---------+----------+-----------------+----------+-----+---------+--------+----------+\n|country  |min(price)|avg(price)       |max(price)|total|varieties|wineries|prediction|\n+---------+----------+-----------------+----------+-----+---------+--------+----------+\n|France   |569       |784.4285714285714|1500      |14   |5        |12      |5         |\n|Portugal |770       |780.0            |790       |2    |1        |2       |5         |\n|Germany  |775       |775.0            |775       |2    |1        |1       |5         |\n|Australia|780       |780.0            |780       |1    |1        |1       |5         |\n|Italy    |595       |595.0            |595       |1    |1        |1       |5         |\n|Austria  |1100      |1100.0           |1100      |1    |1        |1       |5         |\n+---------+----------+-----------------+----------+-----+---------+--------+----------+\n~~~\n\n#### Spark SQL Tricks\n[Working with Apache Spark DataFrames, Json and the Good Ol StructType](https://medium.com/@newfrontcreative/working-with-apache-spark-dataframes-json-and-the-good-ol-structtype-6291bdcd44bd)\n\n### Streaming Trend Discovery (part3)\nImportant Technologies and Concepts\n\n#### Apache Kafka (reliable pub/sub infrastructure)\n[Docs](https://kafka.apache.org/documentation/#uses)\n\nApache Kafka allows you to reliably read and more importantly (re-read) a stream of time series data. This is important given streaming applications can fail and having to re-run from the last state of an application may require re-reading data in order to pick things back up.\n\n#### Data Sketches\n[DataSketch library docs](https://datasketches.github.io/)\n[Yahoo DataSketches Blog Post](https://yahooeng.tumblr.com/post/135390948446/data-sketches)\n[T-Digest](http://koff.io/posts/using-t-digest/)\n[Monoid Addition via T-Digest](http://erikerlandson.github.io/blog/2016/12/19/converging-monoid-addition-for-t-digest/)\n\nData Sketching has many functions in statistics, and with respect to Percentiles/Quantiles they make it very easy to approximate the actual quantiles data and understand the underlying density (histogram via Probability Density Function) and also understand the shape in terms of Cumulative Density. More importantly to their function in Spark is their *native ability to be distributed and mergable* which means that distributed statistics (monoid / monadic systems idea) becomes as simple as running on your localhost/laptop.\n\n#### Windowing and Watermarking Data\n**Windowing** data is a concept when dealing with TimeSeries Data. A Window is a logical subset of a continuous data stream that begins and ends at specific points in time.\n\n**Watermarking** data is a pattern that has become more popular in the streaming world given that Upstream systems may decide that they needed to replay data in order to fulfill an expectation and unfortunately your data stream may have mixed \"times\" due to the replay. So when you apply a Watermark to your data you are denoting when you would like to reject and ignore **late arriving data**.\n\n#### Simple Trick for Dimensional reduction in your Metric Streams.\n\nSay you have the following two events\n~~~\n  {\n    \"id\": \"uuid1\",\n    \"type\": \"GamePlay\",\n    \"value\": 26.01,\n    \"metric\": \"session_time\",\n    \"dimensions\": {\n      \"country\": \"US\",\n      \"user\": \"id123\",\n      \"game_id\": \"UUID\"\n    }\n  },\n  {\n    \"id\": \"uuid2\",\n    \"type\": \"GamePlay\",\n    \"value\": 10.0,\n    \"metric\": \"game_load_time\",\n    \"dimensions\": {\n      \"country\": \"US\",\n      \"user\": \"id123\",\n      \"game_id\": \"UUID\"\n    }\n  }\n~~~\n\nThe Events (SessionTime and GameLoadTime) are just two different metrics sharing a similar pattern when it comes to the underlying dimensions (categorical features) of said events. Your **hashed dimensions** would end up being the common underlying dimensions that are not **unique** to any one common entity (user etc).\n\nHashing Code\n~~~\ndef generateId(bytes: Array[Byte]): String = {\n  val hf = Hashing.murmur3_128()\n  val hc = hf.hashBytes(bytes)\n  BaseEncoding.base64Url()\n    .omitPadding()\n    .encode(hc.asBytes())\n}\n~~~\n\nResulting in the following logical hash\n~~~\ngenerateId(s\"country=$country:game_id=$game_id\".getBytes) // A2SFbSnmugskJNwhhdLg6w\n~~~\n\n#### How Streaming Discovery Works\n\n##### The Data\nData Structure must be known ahead of time and Dimensions \"shouldn't\" be fully freeform, due to the need to be able to generate a Dimensional Hash. The Hash of the common dimensions creates a latch point or intersection amoungst your metrics, and solves the problem of (what other metrics are experiencing this similar or disimilar behavior). You essentially are creating an ad-hoc sub-graph with respect to Time and Common Dimensions.\n\n##### Mixing Streaming Applications\nGiven that we can read data and re-read data (from kafka) and given that Kafka can host multiple topics, then we use the following approach to automatically detecting trends in \"real-ish time\".\n\n#### Discovery Engine Part 1\nFirst we have our **validated and structured core events** (remember that everything that is enqueued to kafka must first have a valid Data Contract - this is the base hypothesis in order for this to work). This is the initial ingest flow in the discovery architecture. This application is projecting the Core Events into our binary compactible and serializable formats - eg. the `MetricAggregation` primitive and removing the hard work of storing potentially large overhead in memory to preserve the full set of metrics.\n\nThis application can be configured with any variable **Window**, but it is important that this window be small enough so you can **store the aggregate data in memory**. This is the most important phase of the engine given that this must **be correct** and running at all times (otherwise you break rules of Real-Time accountability)\n\nExample of the Streaming Query\n~~~\nval aggregationOutputStream = EventAggregation(config).process(readKafkaStream())(spark)\n  .writeStream\n  .queryName(\"streaming.trend.discovery\")\n  .outputMode(eventProcessor.outputMode)\n  .trigger(Trigger.ProcessingTime(Duration(config.triggerInterval)))\n  .format(\"kafka\")\n  .option(\"topic\", \"spark.summit.call.aggregations\")\n  .options(Map(\n    \"checkpointLocation\" -\u003e config.checkpointPath\n  ))\n  .start()\n~~~\n\nIt is worth noting that this application is setup to use `EventTime` vs `ProcessingTime`. This is very important because EventTime means you trust the timestamp of a given event (or you have added your own **logged_event_ts**). This allows you to do rapid replay and recovery. Consider this.\n\n##### Processing Time\nYou have a streaming application and it is Windowing by 1 hour using processing time. It fails. So it takes 1 hour from the time of failure to recover.\n\n##### Event Time\nYou have a streaming application and it is Windowing by 1 hour using event time. It fails. So you restart it and could be caught up in a matter of seconds (depends on the spark cluster size and capabilities...)\n\n#### Discovery Engine Part 2\nGiven that you have the base pattern (eg. `streaming-trend-discovery` app) for doing Performance Optimized, Windowed, Recoverable, Monitored, Unit Tested streaming aggregation. Then you can easily make a slight update to the code base to handle the ingestion of data from a second Kafka Topic. This application will be dead simple. Here is the gist.\n\n1. You want to collect up to N windowed aggregates to do a delta stream analysis.\n2. Given you are using **EventTime** processing and have say an aggregate window of 5 minutes (from Part 1)\n3. Then you could create a delta stream that take the last 4 windows (eg. 20m) - to do comparison of the data and discover treands in your metric aggregation stream. Given you have a strong hash (**dimensional_hash**) you can use this potentially as a secondary grouping key and then simply window and aggregate etc! \n\nOutput Format from the Example Application\n\n**TimeSeries Rows**\n~~~\n+------+-------------+-------------+---------------+-------+--------------------+--------------------+--------------------+--------------------+\n|metric| window_start|   window_end|window_interval|samples|               stats|           histogram|          dimensions|      dimension_hash|\n+------+-------------+-------------+---------------+-------+--------------------+--------------------+--------------------+--------------------+\n|   pdd|1527966000000|1527966300000|             5m|     11|[0.7, 1.1, 2.2, 3...|[0.0, 0.0, 0.0, 2...|[us, outbound, te...|F-WIKm9w3XXTLzjJb...|\n|   pdd|1527967200000|1527967500000|             5m|      1|[5.9, 5.9, 5.9, 5...|[0.0, 0.0, 0.0, 0...|[us, outbound, te...|F-WIKm9w3XXTLzjJb...|\n|   pdd|1527966000000|1527966300000|             5m|     12|[0.7, 1.7, 2.2, 3...|[0.0, 0.0, 0.0, 2...|[us, outbound, te...|F-WIKm9w3XXTLzjJb...|\n|   pdd|1527966300000|1527966600000|             5m|      6|[1.1, 1.3, 1.9, 2...|[0.0, 0.0, 0.0, 0...|[us, outbound, te...|F-WIKm9w3XXTLzjJb...|\n|   pdd|1527966600000|1527966900000|             5m|      2|[1.9, 1.9, 1.9, 1...|[0.0, 0.0, 0.0, 0...|[us, outbound, te...|F-WIKm9w3XXTLzjJb...|\n|   pdd|1527966000000|1527966300000|             5m|     12|[0.7, 1.7, 2.2, 3...|[0.0, 0.0, 0.0, 2...|[us, outbound, te...|F-WIKm9w3XXTLzjJb...|\n|   pdd|1527968400000|1527968700000|             5m|      1|[10.9, 10.9, 10.9...|[0.0, 0.0, 0.0, 0...|[us, outbound, te...|F-WIKm9w3XXTLzjJb...|\n+------+-------------+-------------+---------------+-------+--------------------+--------------------+--------------------+--------------------+\n~~~\n\n**StructType**\n~~~\n|-- metric: string (nullable = true)\n |-- window_start: long (nullable = true)\n |-- window_end: long (nullable = true)\n |-- window_interval: string (nullable = true)\n |-- samples: integer (nullable = true)\n |-- stats: struct (nullable = true)\n |    |-- min: double (nullable = true)\n |    |-- p25: double (nullable = true)\n |    |-- median: double (nullable = true)\n |    |-- p75: double (nullable = true)\n |    |-- p90: double (nullable = true)\n |    |-- p95: double (nullable = true)\n |    |-- p99: double (nullable = true)\n |    |-- max: double (nullable = true)\n |    |-- mean: double (nullable = true)\n |    |-- sd: double (nullable = true)\n |    |-- variance: double (nullable = true)\n |-- histogram: struct (nullable = true)\n |    |-- bin1: double (nullable = true)\n |    |-- bin2: double (nullable = true)\n |    |-- bin3: double (nullable = true)\n |    |-- bin4: double (nullable = true)\n |    |-- bin5: double (nullable = true)\n |    |-- bin6: double (nullable = true)\n |-- dimensions: struct (nullable = true)\n |    |-- country: string (nullable = true)\n |    |-- direction: string (nullable = true)\n |    |-- carrier: string (nullable = true)\n |    |-- route: string (nullable = true)\n |-- dimension_hash: string (nullable = true)\n~~~\n\nExample of all that is needed (assuming you are using Protobuf and the Encoder from the example application)\n~~~\naggregateWindowStream\n  .withWatermark(\"window_end\", \"5 minutes\")\n  .groupBy(\"metric\", \"dimensional_hash\", window($\"window_end\", \"20 minutes\"))\n  .agg(\n    avg(\"stats.avg\") as \"avg_avg\",\n    avg(\"stats.median\") as \"avg_median\",\n    min(\"stats.p95\") as \"min_p95\",\n    avg(\"stats.p95\") as \"avg_p95\",\n    max(\"stats.p95\") as \"max_p95\",\n    min(\"stats.sd\") as \"min_sd\",\n    avg(\"stats.sd\") as \"avg_sd\",\n    max(\"stats.sd\") as \"max_sd\",\n    min(\"samples\") as \"min_metrics\",\n    avg(\"samples\") as \"avg_metrics\",\n    max(\"samples\") as \"max_metrics\"\n  )\n~~~\n\nThe above would create a stream of aggregates (see Part 1) and then watermark and window these aggregates (further aggregate), and then it is up to you to analyze this further using the tools that are now part of your Spark arsenal.\n\nThanks\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnewfront%2Fodsc-west-streaming-trends","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnewfront%2Fodsc-west-streaming-trends","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnewfront%2Fodsc-west-streaming-trends/lists"}