https://github.com/wintermi/bqdo
bqdo is a CLI for executing BigQuery SQL as part of a pipeline.
https://github.com/wintermi/bqdo
bigquery google-cloud google-cloud-platform
Last synced: 16 days ago
JSON representation
bqdo is a CLI for executing BigQuery SQL as part of a pipeline.
- Host: GitHub
- URL: https://github.com/wintermi/bqdo
- Owner: wintermi
- License: apache-2.0
- Created: 2025-08-09T15:18:10.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2025-12-03T06:08:41.000Z (6 months ago)
- Last Synced: 2025-12-06T07:34:17.550Z (6 months ago)
- Topics: bigquery, google-cloud, google-cloud-platform
- Language: Go
- Homepage:
- Size: 48.8 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# bqdo
bqdo is a simple CLI for running BigQuery SQL pipelines from a directory of `.sql` files. It reads a `bqdo.toml` configuration, applies Go `text/template` variables, and executes each SQL file in alphabetical order. It is designed for lightweight, repeatable data workflows.
## Features
- Read `bqdo.toml` from the current directory (or `-c` path)
- Override config values with CLI flags
- Optional service account impersonation
- Recursively discover and run `.sql` files in alphabetical order
- Go `text/template` support using values from the `[vars]` section
- `--dry-run` to validate without executing
## Requirements
- Go `1.24+`
- Optional: `mise` for project tasks (recommended for building)
- Auth to Google Cloud (ADC via `gcloud auth application-default login`, or `GOOGLE_APPLICATION_CREDENTIALS` pointing to a service account JSON)
## Download & Build
Clone the repo and build using `mise` (recommended):
```sh
git clone https://github.com/wintermi/bqdo.git
cd bqdo
mise run build
```
This produces the binary at `bin/bqdo`.
Alternatively, build directly with Go:
```sh
CGO_ENABLED=0 go build -o bin/bqdo ./cmd/bqdo
```
## Quick Start
1) Initialize a new pipeline in your project directory:
```sh
bin/bqdo init
```
This creates a `bqdo.toml`. Edit it to set `directory`, `dataset`, and other values.
2) Add your SQL files under the configured `directory` (e.g., `sql/`). Files will execute in alphabetical order.
3) Run the pipeline:
```sh
bin/bqdo run -c bqdo.toml -p YOUR_PROJECT_ID -d YOUR_DATASET -l US
```
Use `--dry-run` to validate queries without executing:
```sh
bin/bqdo run -c bqdo.toml -p YOUR_PROJECT_ID -d YOUR_DATASET -l US --dry-run
```
To impersonate a service account:
```sh
bin/bqdo run -c bqdo.toml -p YOUR_PROJECT_ID -d YOUR_DATASET -l US \
--impersonate-service-account your-sa@your-project.iam.gserviceaccount.com
```
## Examples
A minimal runnable example is provided in `examples/simple`. See its README for details.
## License
Apache 2.0. See `LICENSE`.