https://github.com/cloudquery/cloudrun-example
A reference for running CloudQuery syncs via Cloud Run
https://github.com/cloudquery/cloudrun-example
Last synced: about 1 month ago
JSON representation
A reference for running CloudQuery syncs via Cloud Run
- Host: GitHub
- URL: https://github.com/cloudquery/cloudrun-example
- Owner: cloudquery
- Created: 2023-05-23T11:22:58.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-04-03T09:42:14.000Z (about 2 years ago)
- Last Synced: 2025-02-25T19:41:51.873Z (over 1 year ago)
- Language: Go
- Size: 4.88 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# CloudQuery Cloud Run Example
This repository contains a small Docker file that starts a Web server on port 8080 for use with Cloud Run. When the server receives a request on the port, it starts a CloudQuery sync using a config file that should be mounted at `/secrets/config.yaml`.
You should also generate a CloudQuery API key and set it as an environment variable `CLOUDQUERY_API_KEY` in the [Cloud Run configuration](https://cloud.google.com/run/docs/configuring/services/secrets#access-secrets). See more information on generating an API key [here](https://docs.cloudquery.io/docs/deployment/generate-api-key).
## Deployment
This guide is still incomplete, but the rough steps are:
1. Create a Docker repository in Google Cloud Artifact Registry.
2. Get the latest version of CloudQuery Container from [ghcr.io/cloudquery/cloudquery](https://ghcr.io/cloudquery/cloudquery). Update the version in the Dockerfile
3. Build the image locally. Replace the `-docker.pkg.dev///:` with the proper values.
```bash
docker build --platform=linux/amd64 -t -docker.pkg.dev///: .
```
Example:
```bash
docker build --platform=linux/amd64 -t europe-north1-docker.pkg.dev/cloudquery-project/cq-repository/cq-image:6.24.2 .
```
4. Upload the image to Artifact Registry:
```bash
docker push europe-north1-docker.pkg.dev/cloudquery-project/cq-repository/cq-image:6.24.2
```
5. Create a cloud run job using the newly pushed image.
- On the `Containers` tab, be sure to set the following container arguments:
```text
--read-only
--add-host=host.docker.internal:host-gateway
--no-log-file
sync
/secrets/config.yaml
```
- The arguments respect the fact that Cloud Run is read-only (except `os.tmpdir()`).
- For a fuller explanation please see [here](https://docs.cloudquery.io/docs/deployment/read-only-containers).
- Make sure to mount your CloudQuery config file at `/secrets/config.yaml` (using Secrets).
- Note that it is possible to combine sources and destinations in a single config file by separating the sections with `---` (see [the docs](https://www.cloudquery.io/docs/core-concepts/configuration))
6. Schedule the job via [Cloud Scheduler](https://cloud.google.com/scheduler).