https://github.com/pompierninja/scio-demo
Playing w/ Scio
https://github.com/pompierninja/scio-demo
apache-beam bigquery scio
Last synced: about 1 month ago
JSON representation
Playing w/ Scio
- Host: GitHub
- URL: https://github.com/pompierninja/scio-demo
- Owner: pompierninja
- Created: 2019-10-04T19:02:53.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2019-10-09T18:53:04.000Z (over 6 years ago)
- Last Synced: 2026-05-13T19:41:02.969Z (about 1 month ago)
- Topics: apache-beam, bigquery, scio
- Language: Scala
- Homepage:
- Size: 50.8 KB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Playing w/ Spotify's Scio
* [X] Ingest [https://www.meetup.com/fr-FR/meetup_api/](https://www.meetup.com/fr-FR/meetup_api/)
* [X] Streaming processing w/ Apache-Beam X Scio
* [X] Query trending topics from BigQuery
```sh
# Playing w/ Spotify's Scio
# Using DirectRunner
# Build
$ sbt pack
# Run
$ target/pack/bin/main
# Just run
sbt run
```
```sql
# (BigQuery) Get top 5 trending topics within the last 5 minutes
SELECT SUM(score) as score, topic_name, timestamp
FROM `project_id.dataset_name.table_name`
WHERE timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 5 MINUTE)
GROUP BY timestamp, topic_name
ORDER BY score DESC
LIMIT 5
```