{"id":42482608,"url":"https://github.com/cloudspannerecosystem/spanner-table-copy-pipeline","last_synced_at":"2026-01-28T11:15:41.936Z","repository":{"id":51605997,"uuid":"450433437","full_name":"cloudspannerecosystem/spanner-table-copy-pipeline","owner":"cloudspannerecosystem","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-21T08:14:06.000Z","size":49,"stargazers_count":2,"open_issues_count":12,"forks_count":3,"subscribers_count":34,"default_branch":"main","last_synced_at":"2024-10-21T11:03:17.228Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudspannerecosystem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-01-21T09:40:33.000Z","updated_at":"2024-10-21T08:14:10.000Z","dependencies_parsed_at":"2022-08-22T10:50:36.450Z","dependency_job_id":null,"html_url":"https://github.com/cloudspannerecosystem/spanner-table-copy-pipeline","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/cloudspannerecosystem/spanner-table-copy-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudspannerecosystem%2Fspanner-table-copy-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudspannerecosystem%2Fspanner-table-copy-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudspannerecosystem%2Fspanner-table-copy-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudspannerecosystem%2Fspanner-table-copy-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudspannerecosystem","download_url":"https://codeload.github.com/cloudspannerecosystem/spanner-table-copy-pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudspannerecosystem%2Fspanner-table-copy-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28844861,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-28T10:53:21.605Z","status":"ssl_error","status_checked_at":"2026-01-28T10:53:20.789Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-28T11:15:41.269Z","updated_at":"2026-01-28T11:15:41.931Z","avatar_url":"https://github.com/cloudspannerecosystem.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spanner Table Copy\n\nBeam pipeline to run a query on a Spanner database and write the results to a\nspanner table.\n\nThis pipeline can be used for various functions to transform/update a database,\nwithout needing to be concerned about transaction mutation limitations\n\n* Transforming read data using SQL and writing the results back, e.g. the\n  equivalent of the following pseudo-sql:\n\n```sql\n  INSERT OR UPDATE INTO \u003ctable\u003e (key, value)\n  (\n     SELECT key, value*100 FROM \u003ctable\u003e WHERE \u003ccondition\u003e\n  )\n```\n\n* Point-in-time recovery by reading the database state at a point in the past\n  (within the Spanner database's\n  \u003ca href=\"https://cloud.google.com/spanner/docs/pitr\"\u003eversion retention\n  period\u003c/a\u003e, and then writing it back. e.g. the equivalent of the following\n  pseudo-sql:\n\n```sql\n  INSERT OR UPDATE INTO \u003ctable\u003e (key, value)\n  (\n     SELECT key, value\n     FROM \u003ctable\u003e FOR SYSTEM_TIME AS OF \u003ctimestamp_RFC3339\u003e\n     WHERE \u003ccondition\u003e\n  )\n```\n\n* Copying data from one database to another, e.g. the equivalent of the\n  following pseudo-sql:\n\n```sql\n INSERT OR UPDATE INTO \u003ctable@project2/instance2/database2\u003e (key, value)\n (\n    SELECT key, value FROM \u003ctable@project1/instance1/database1\u003e\n )\n```\n\n## Usage:\n\n### Show help text:\n\n```\nmvn compile exec:java -Dexec.mainClass=com.google.cloud.solutions.SpannerTableCopy\n    -Dexec.args='--help=com.google.cloud.solutions.SpannerTableCopy$SpannerTableCopyOptions'\n```\n\n### Execute the pipeline:\n\nFor example copying a table at a timestamp in the past to a different database:\n\n```\nmvn compile exec:java\n    -Dexec.mainClass=com.google.cloud.solutions.SpannerTableCopy \\\n    -Pdataflow-runner\n    -Dexec.args=\"\n         --runner=DataflowRunner\n\n         --sourceProjectId=SOURCE_PROJECT\n         --sourceInstanceId=SOURCE_INSTANCE\n         --sourceDatabaseId=SOURCE_DATABASE\n\n         --sqlQuery='select * from SOURCE_TABLE'\n         --readTimestamp=2022-01-01T12:00:00Z\n\n         --destinationProjectId=DEST_PROJECT\n         --destinationInstanceId=DEST_INSTANCE\n         --destinationDatabaseId=DEST_DATABASE\n         --destinationTable=DEST_TABLE\n         --writeMode=WRITE_MODE\n\n         --mutationReportFile=gs://BUCKET/PATH/report\n         --failureLogFile=gs://BUCKET/PATH/failures\n    \"\n```\n\nMore examples can be found in the `SpannerTableCopyIntegrationTest`\n\nNote: take care of Bash quoting with `-Dexec.args` and the `--sqlQuery` value.\n\nThe `--dryRun` parameter can be used to create the mutation report file without\nactually writing to the database.\n\n`--writeMode` values correspond to the values of the\n[Mutation.Op enum](https://googleapis.dev/java/google-cloud-spanner/latest/com/google/cloud/spanner/Mutation.Op.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudspannerecosystem%2Fspanner-table-copy-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudspannerecosystem%2Fspanner-table-copy-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudspannerecosystem%2Fspanner-table-copy-pipeline/lists"}