{"id":15060778,"url":"https://github.com/anpandu/ps2bq","last_synced_at":"2026-02-12T12:07:52.845Z","repository":{"id":78904436,"uuid":"243049662","full_name":"anpandu/ps2bq","owner":"anpandu","description":"Stream insert GCP PubSub messages into BigQuery table.","archived":false,"fork":false,"pushed_at":"2020-03-09T15:28:52.000Z","size":23,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-08-19T04:41:35.340Z","etag":null,"topics":["bigquery","golang","pubsub"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anpandu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-25T16:50:49.000Z","updated_at":"2024-06-19T08:08:31.422Z","dependencies_parsed_at":null,"dependency_job_id":"d91c3dcc-8dbc-4084-b247-4804f5f76a0e","html_url":"https://github.com/anpandu/ps2bq","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anpandu%2Fps2bq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anpandu%2Fps2bq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anpandu%2Fps2bq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anpandu%2Fps2bq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anpandu","download_url":"https://codeload.github.com/anpandu/ps2bq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243695589,"owners_count":20332629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","golang","pubsub"],"created_at":"2024-09-24T23:04:24.102Z","updated_at":"2026-02-12T12:07:47.799Z","avatar_url":"https://github.com/anpandu.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ps2bq \u0026middot; [![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/anpandu/ps2bq/blob/master/LICENSE) \n\n## Introduction\n\nPS2BQ is a CLI tool for importing messages from GCP PubSub into BigQuery table.\n\n## Usage\n\n### Run from local\n\n```sh\n# clone repo\ngit clone https://github.com/anpandu/ps2bq\n\n# build binary\ngo mod download\ngo install\n\n# run it\nexport GOOGLE_APPLICATION_CREDENTIALS=~/google-key.json # make sure credential file is set\n$GOPATH/bin/ps2bq run --help\n\n# EXAMPLE\nps2bq run\n    --project=myproject\n    --dataset=mydataset\n    --table=students\n    --topic=t-students\n    --subscription-id=ps2bq-students-20200101\n    --worker=4\n    --message-buffer=1\n```\n\n### Run as Docker container\n\n```sh\n# TBD\n```\n\n### Configs\n\n```\n  -D, --dataset string           BigQuery Dataset\n  -n, --message-buffer int       Number of message to be inserted (default 1)\n  -P, --project string           Google Cloud Platform Project ID\n      --schema string            BigQuery JSON table schema file location (default \"/tmp/schema.json\")\n  -s, --subscription-id string   PubSub Subscription ID\n  -T, --table string             BigQuery Table\n  -t, --topic string             PubSub Topic\n  -w, --worker int               Number of workers (default 4)\n```\n\nPubSub Messages will each inserted as a new row.  \nPubSub Messages received should be JSON Object.  \nMessage example: `{\"id\":123,\"name\":\"Alice\"}`  \nJSON Schema must be provided in order to create table.  \nSee: https://cloud.google.com/bigquery/docs/schemas#creating_a_json_schema_file  \nMessage containing invalid JSON and/or invalid according to table schema will be failed to be inserted.  \n\n## Roadmap\n| Status  | Description |\n|:-------:|:----------- |\n|    ✔    | 1 worker, 1 message inserted |\n|    ✔    | N worker, 1 message inserted each |\n|    ✔    | N worker, N message inserted each (buffered) |\n|    ✘    | Every t seconds, insert all messages in buffer |\n|    ✘    | Dockerfile |\n|    ✘    | Validate message using JSON schema |\n|    ✘    | Create table with partition |\n|    ✘    | Auto-generate subscription ID |\n|    ✘    | go doc |\n|    ✘    | Kubernetes YAMLs |\n|    ✘    | Multiple sink (?) |\n|    ✘    | Multiple source (?) |\n\n\n## License\n\nMIT © [Ananta Pandu](anpandumail@gmail.com)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanpandu%2Fps2bq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanpandu%2Fps2bq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanpandu%2Fps2bq/lists"}