Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mattwelke/charter-challenge-for-fair-voting-bot
Bot that web scrapes and logs in BigQuery the donations so far of the Charter Challenge for Fair Voting.
https://github.com/mattwelke/charter-challenge-for-fair-voting-bot
bigquery bot go openwhisk public-data
Last synced: about 1 month ago
JSON representation
Bot that web scrapes and logs in BigQuery the donations so far of the Charter Challenge for Fair Voting.
- Host: GitHub
- URL: https://github.com/mattwelke/charter-challenge-for-fair-voting-bot
- Owner: mattwelke
- License: apache-2.0
- Created: 2022-09-22T02:34:36.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-03T01:30:52.000Z (7 months ago)
- Last Synced: 2024-10-14T02:24:13.068Z (3 months ago)
- Topics: bigquery, bot, go, openwhisk, public-data
- Language: Go
- Homepage:
- Size: 99.6 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# charter-challenge-for-fair-voting-bot
A bot that records the amount donated so far for the Charter Challenge for Fair Voting (https://www.charterchallenge.ca/) into a public BigQuery dataset each hour.
## Querying data
The data is available in a public dataset that is updated hourly. Example query:
```sql
SELECT
*
FROM
`public-datasets-363301.charter_challenge_bot.donation_statuses`
ORDER BY
date DESC
LIMIT
1000
```
Any GCP principal authenticated can query this public dataset.
The data is stored per hour with the string from the website stored as is, but you can use BigQuery SQL to further process the data. For example:
```sql
SELECT
# Truncate timestamps to just their day
DATE(TIMESTAMP_TRUNC(date, DAY)) AS date,
# Convert string from web page to NUMERIC and get a summary
# of how much had been donated by the end of that day. Format
# as currency at the end to display nicely.
CONCAT('$',FORMAT("%'.2f", MAX(CAST(REGEXP_REPLACE(so_far, '[^.0-9 ]', '') AS NUMERIC)))) AS donated_so_far
FROM
`public-datasets-363301.charter_challenge_bot.donation_statuses`
GROUP BY
date
ORDER BY
date DESC
```
You can use BigQuery [window function calls](https://cloud.google.com/bigquery/docs/reference/standard-sql/window-function-calls) to look for significant parts of the data, like during which hours the donations were made, instead of just seeing a running total by querying the raw data:
```sql
WITH
t1 AS (
SELECT
TIMESTAMP_TRUNC(date, HOUR) AS hour,
CAST(REGEXP_REPLACE(so_far, '[^.0-9 ]', '') AS NUMERIC) AS donated_so_far
FROM
`public-datasets-363301.charter_challenge_bot.donation_statuses`
ORDER BY
hour DESC ),
t2 AS (
SELECT
hour,
donated_so_far - LAG(donated_so_far, 1) OVER (ORDER BY hour) AS donated_during_hour
FROM
t1)
SELECT
hour,
CONCAT('$',FORMAT("%'.2f", donated_during_hour)) AS donated_during_hour
FROM
t2
WHERE
donated_during_hour > 0
ORDER BY
hour DESC
```
