Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/faranak-cs/bq-py-app
BigQuery Python Application
https://github.com/faranak-cs/bq-py-app
Last synced: 23 days ago
JSON representation
BigQuery Python Application
- Host: GitHub
- URL: https://github.com/faranak-cs/bq-py-app
- Owner: faranak-cs
- Created: 2024-12-02T18:34:16.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2024-12-10T12:25:18.000Z (26 days ago)
- Last Synced: 2024-12-10T13:28:28.933Z (26 days ago)
- Language: Python
- Size: 9.77 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# BQ PY App
A simple app that connects to GCP and calls the BigQuery API
## Installation
- https://cloud.google.com/sdk/docs/install-sdk
```
gcloud config set compute/region us-west3
```
```
gcloud config set compute/zone us-west3-c
```
```
gcloud config set project PROJECT_ID
```
## Auth commands
```
gcloud auth login
```
```
gcloud auth application-default login
```
## BigQuery commands
### CLI
```
bq load --source_format=NEWLINE_DELIMITED_JSON --autodetect=True PROJECT_ID:DATASET.TABLE_NAME FILE_NAME.json
```
```
bq load --source_format=NEWLINE_DELIMITED_JSON 'PROJECT_ID:DATASET.TABLE_NAME' --autodetect 'gs://BUCKET_NAME/FILE_NAME.json'
```
```
bq show --format=prettyjson PROJECT_ID:DATASET.TABLE_NAME | jq -r '.schema.fields[] .name ' | sort
```
### Query
#### msg-with-reply-using-id
```
SELECT a.id, a.body.content AS Question, b.body.content AS Replies
FROM `PROJECT_ID.DATASET.TABLE_NAME` a , unnest(a.replies) AS b
WHERE a.id = 1725460272017
ORDER BY a.id
```
#### msg-with-reply-merge
```
SELECT a.id, a.body.content AS Question, ARRAY_AGG(b.body.content IGNORE NULLS) AS Reply
FROM `PROJECT_ID.DATASET.TABLE_NAME` a, a.replies AS b
WHERE a.id = 1725460272017
GROUP BY a.id, a.body.content
ORDER BY a.id
```
```
create table PROJECT_ID.DATASET.TABLE_NAME as select * from PROJECT_ID.DATASET.TABLE_NAME where 1 = 2;
```
## JQ commands
```
cat FILE_NAME.json | head -n 1 | jq 'keys'
```
```
cat FILE_NAME.json | head -n 1 | jq '.id'
```
```
cat INPUT_FILE_NAME.json | jq -c '.[] | .' > OUTPUT_FILE_NAME.json
```
```
jq -c 'walk(if type == "object" and has("@odata.type") then del(."@odata.type") else . end)' input.json > output.json
```
## CLI commands
### Disk Usage Human Readable Summary:
```
du -hs FILE_NAME.json
```
### Word Count of Lines:
```
wc -l FILE_NAME.json
```
## Architecture Diagram
