{"id":48227001,"url":"https://github.com/wan-huiyan/gcp-dataform-rest-api-deploy","last_synced_at":"2026-04-04T19:24:19.956Z","repository":{"id":346390122,"uuid":"1189558242","full_name":"wan-huiyan/gcp-dataform-rest-api-deploy","owner":"wan-huiyan","description":"Claude Code skill: Deploy .sqlx files to Google Cloud Dataform via REST API — full lifecycle with gotcha documentation","archived":false,"fork":false,"pushed_at":"2026-04-02T18:43:51.000Z","size":24,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-03T06:06:13.806Z","etag":null,"topics":["automation","bigquery","ci-cd","claude-code","claude-code-skill","cloud-workflows","dataform","gcp","google-cloud","sql"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wan-huiyan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-23T12:50:35.000Z","updated_at":"2026-04-02T18:43:55.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/wan-huiyan/gcp-dataform-rest-api-deploy","commit_stats":null,"previous_names":["wan-huiyan/gcp-dataform-rest-api-deploy"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/wan-huiyan/gcp-dataform-rest-api-deploy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wan-huiyan%2Fgcp-dataform-rest-api-deploy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wan-huiyan%2Fgcp-dataform-rest-api-deploy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wan-huiyan%2Fgcp-dataform-rest-api-deploy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wan-huiyan%2Fgcp-dataform-rest-api-deploy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wan-huiyan","download_url":"https://codeload.github.com/wan-huiyan/gcp-dataform-rest-api-deploy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wan-huiyan%2Fgcp-dataform-rest-api-deploy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31410416,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T10:20:44.708Z","status":"ssl_error","status_checked_at":"2026-04-04T10:20:06.846Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","bigquery","ci-cd","claude-code","claude-code-skill","cloud-workflows","dataform","gcp","google-cloud","sql"],"created_at":"2026-04-04T19:24:19.880Z","updated_at":"2026-04-04T19:24:19.941Z","avatar_url":"https://github.com/wan-huiyan.png","language":"JavaScript","readme":"# GCP Dataform REST API Deploy\n[![GitHub release](https://img.shields.io/github/v/release/wan-huiyan/gcp-dataform-rest-api-deploy)](https://github.com/wan-huiyan/gcp-dataform-rest-api-deploy/releases) [![Claude Code](https://img.shields.io/badge/Claude_Code-skill-orange)](https://claude.com/claude-code) [![license](https://img.shields.io/github/license/wan-huiyan/gcp-dataform-rest-api-deploy)](LICENSE) [![last commit](https://img.shields.io/github/last-commit/wan-huiyan/gcp-dataform-rest-api-deploy)](https://github.com/wan-huiyan/gcp-dataform-rest-api-deploy/commits)\n\nDeploying Dataform `.sqlx` files via REST API requires base64-encoding, undocumented field names, and a polling loop — six places to get it wrong. This skill handles the full lifecycle without the Dataform CLI.\n\n## Quick Start\n\n```\nYou: Deploy this new .sqlx model to Dataform and run it\nClaude: [writes file to workspace → commits → pushes → compiles → invokes → polls until SUCCEEDED]\n```\n\nOr use the documented bash patterns directly in CI/CD pipelines, Cloud Workflows, or deployment scripts.\n\n## Installation\n\n### Claude Code\n\n```bash\n# Plugin install (recommended)\n/plugin marketplace add wan-huiyan/gcp-dataform-rest-api-deploy\n/plugin install gcp-dataform-rest-api-deploy@wan-huiyan-gcp-dataform-rest-api-deploy\n\n# Git clone (always works)\ngit clone https://github.com/wan-huiyan/gcp-dataform-rest-api-deploy.git ~/.claude/skills/gcp-dataform-rest-api-deploy\n```\n\n### Cursor (2.4+)\n\n```bash\n# Per-project rule (most reliable)\nmkdir -p .cursor/rules\n# Copy SKILL.md content into .cursor/rules/gcp-dataform-rest-api-deploy.mdc with alwaysApply: true\n\n# npx skills CLI\nnpx skills add wan-huiyan/gcp-dataform-rest-api-deploy --global\n\n# Manual global install\ngit clone https://github.com/wan-huiyan/gcp-dataform-rest-api-deploy.git ~/.cursor/skills/gcp-dataform-rest-api-deploy\n```\n\n## What You Get\n\n- **Complete 6-step lifecycle**: writeFile → commit → push → compile → invoke → poll\n- **Failure diagnostics**: Action-level error extraction showing exact SQL errors\n- **6 documented gotchas** that cause silent failures or cryptic errors\n- **Cloud Workflows integration** notes for YAML-based orchestration\n\n## The Problem\n\nThe Dataform REST API (`v1beta1`) has non-obvious patterns that cause frustrating failures:\n\n| What Goes Wrong | Symptom | Root Cause |\n|----------------|---------|------------|\n| 404 on API calls | `Not Found` on valid workspace | Colon-prefixed actions (`:writeFile`) get URL-encoded or appended wrong |\n| Commit fails | `Unknown name \"fileOperations\"` | Wrong field name — API uses `paths`, not `fileOperations` |\n| File write seems to fail | Empty `{}` response | That's actually success — empty response = OK |\n| Invocation runs wrong SQL | Old version executes | Forgot to push workspace before compiling |\n| Region mismatch | Various auth/not-found errors | Dataform region often differs from Cloud Run region |\n| Date parsing errors | `No matching signature for TIMESTAMP_SECONDS` | Column types in BQ don't match what you'd expect from field names |\n\n## How It Works\n\n| Step | API Endpoint | What It Does |\n|------|-------------|--------------|\n| 1. Write | `{workspace}:writeFile` | Upload base64-encoded `.sqlx` to workspace |\n| 2. Commit | `{workspace}:commit` | Commit with author, message, and file paths |\n| 3. Push | `{workspace}:push` | Push workspace changes to default branch |\n| 4. Compile | `{repo}/compilationResults` | Compile from release config |\n| 5. Invoke | `{repo}/workflowInvocations` | Run specific targets |\n| 6. Poll | `GET {invocation}` | Poll `state` until `SUCCEEDED` or `FAILED` |\n\n## Key Gotchas\n\n### 1. Colon-Prefixed Action Endpoints\n```\n# WRONG — colon gets eaten or URL-encoded\n${BASE}/${WS_PATH}writeFile\n\n# RIGHT — colon is part of the path\n${BASE}/${WS_PATH}:writeFile\n```\n\n### 2. Base64 Encoding Required\n`writeFile` expects `contents` as base64, not raw text:\n```bash\nENCODED=$(base64 -i path/to/file.sqlx)\n```\n\n### 3. Commit Uses `paths`, Not `fileOperations`\n```json\n// WRONG\n{\"fileOperations\": {\"definitions/model.sqlx\": {\"writeFile\": {}}}}\n\n// RIGHT\n{\"paths\": [\"definitions/model.sqlx\"], \"author\": {...}, \"commitMessage\": \"...\"}\n```\n\n### 4. Empty `{}` = Success\n`writeFile`, `commit`, and `push` all return empty `{}` on success. Don't treat it as an error.\n\n### 5. Dataform Region != Cloud Run Region\nDataform repos are often in `europe-north1` while Cloud Run services are in `us-central1`. Always check your Dataform repo settings.\n\n### 6. Release Config Full Path Required\n```\nprojects/{project}/locations/{region}/repositories/{repo}/releaseConfigs/{config}\n```\n\n## Checking Failure Details\n\nWhen an invocation fails, get action-level errors:\n```bash\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  \"${BASE}/${INVOCATION_NAME}:query\" | python3 -c \"\nimport sys, json\nfor action in json.load(sys.stdin).get('workflowInvocationActions', []):\n    print(f\\\"{action['target']['name']}: {action['state']}\\\")\n    if action.get('failureReason'):\n        print(f\\\"  Error: {action['failureReason']}\\\")\n\"\n```\n\n## Limitations\n\n- Covers `v1beta1` API only — Google may change endpoints when GA\n- Polling is simple sleep-based; no exponential backoff\n- Does not handle Dataform workspace creation (assumes workspace exists)\n- Does not cover Dataform assertion verification post-invocation\n- Bash examples only — no Python SDK wrapper (yet)\n\n## Dependencies\n\n| Dependency | Required | Purpose |\n|-----------|----------|---------|\n| `gcloud` CLI | Yes | Authentication (`gcloud auth print-access-token`) |\n| `curl` | Yes | HTTP requests to Dataform API |\n| `base64` | Yes | Encoding file contents for writeFile |\n| `python3` | Yes | JSON parsing of API responses |\n| GCP project with Dataform | Yes | Target repository and workspace |\n\n## Version History\n\n| Version | Date | Changes |\n|---------|------|---------|\n| 1.0.0 | 2026-03-23 | Initial release — full lifecycle, 6 gotchas, failure diagnostics |\n\n## License\n\nMIT\n\n---\n\n*Built with [Claude Code](https://claude.ai/claude-code)*\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwan-huiyan%2Fgcp-dataform-rest-api-deploy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwan-huiyan%2Fgcp-dataform-rest-api-deploy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwan-huiyan%2Fgcp-dataform-rest-api-deploy/lists"}