https://github.com/goatcheesesaladwithpeanutoildressing/scio-demo

Playing w/ Scio
https://github.com/goatcheesesaladwithpeanutoildressing/scio-demo

apache-beam bigquery scio

Last synced: about 1 month ago
JSON representation

Playing w/ Scio

• Host: GitHub
• URL: https://github.com/goatcheesesaladwithpeanutoildressing/scio-demo
• Owner: goatcheesesaladwithpeanutoildressing
• Created: 2019-10-04T19:02:53.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2019-10-09T18:53:04.000Z (over 5 years ago)
• Last Synced: 2025-01-02T15:54:46.507Z (about 1 month ago)
• Topics: apache-beam, bigquery, scio
• Language: Scala
• Homepage:
• Size: 50.8 KB
• Stars: 0
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
## Playing w/ Spotify's Scio

* [X] Ingest [https://www.meetup.com/fr-FR/meetup_api/](https://www.meetup.com/fr-FR/meetup_api/)

* [X] Streaming processing w/ Apache-Beam X Scio

* [X] Query trending topics from BigQuery

```sh

# Playing w/ Spotify's Scio

# Using DirectRunner

# Build

$ sbt pack

# Run

$ target/pack/bin/main

# Just run

sbt run

```

```sql

# (BigQuery) Get top 5 trending topics within the last 5 minutes

SELECT SUM(score) as score, topic_name, timestamp

FROM `project_id.dataset_name.table_name`

WHERE timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 5 MINUTE)

GROUP BY timestamp, topic_name

ORDER BY score DESC 

LIMIT 5

```