{"id":15555241,"url":"https://github.com/brews/bucket2bq","last_synced_at":"2026-01-24T02:42:45.733Z","repository":{"id":80503033,"uuid":"578795028","full_name":"brews/bucket2bq","owner":"brews","description":"Create an inventory of objects in GCS Bucket with metadata and upload to Big Query","archived":false,"fork":false,"pushed_at":"2024-06-18T10:33:08.000Z","size":58,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-12T11:01:37.689Z","etag":null,"topics":["bigquery","gcp","golang","google-cloud-storage"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brews.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-15T22:38:17.000Z","updated_at":"2024-06-18T10:33:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"50a7135d-9485-4a93-8cbf-7e5ecf3a0bb7","html_url":"https://github.com/brews/bucket2bq","commit_stats":{"total_commits":18,"total_committers":2,"mean_commits":9.0,"dds":"0.38888888888888884","last_synced_commit":"f2504f8875c31ff9badbf6ebf6b0005232debf49"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brews%2Fbucket2bq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brews%2Fbucket2bq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brews%2Fbucket2bq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brews%2Fbucket2bq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brews","download_url":"https://codeload.github.com/brews/bucket2bq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254276473,"owners_count":22043869,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","gcp","golang","google-cloud-storage"],"created_at":"2024-10-02T15:07:30.531Z","updated_at":"2026-01-24T02:42:40.713Z","avatar_url":"https://github.com/brews.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bucket2bq\nCreate an inventory of objects in a single GCS Bucket and upload the inventory to Big Query.\n\nThis small applications discovers all the objects in a Google Cloud Storage bucket and creates an Avro file containing all the objects\nand their attributes. This can be then imported into BigQuery.\n\n### Usage\n\nThe program to create bucket inventory files can be run as an independent program. For example,\n\n```bash\n./bucket2bq -bucket \"name-of-bucket-to-inventory\"\n```\n\nIt has several options:\n\n```bash\n./bucket2bq -help\nGCS Bucket object metadata to BigQuery, version 0.1.0\nUsage of ./bucket2bq:\n  -alsologtostderr\n        log to standard error as well as files\n  -avro_schema string\n        Avro schema (default: use embedded) (default \"embedded\")\n  -bucket string\n        bucket name (default \"bucketname\")\n  -buffer_size int\n        file buffer (default 1000)\n  -concurrency int\n        concurrency (GOMAXPROCS) (default 4)\n  -file string\n        output file name (default \"gcs.avro\")\n  -log_backtrace_at value\n        when logging hits line file:N, emit a stack trace\n  -log_dir string\n        If non-empty, write log files in this directory\n  -logtostderr\n        log to standard error instead of files\n  -stderrthreshold value\n        logs at or above this threshold go to stderr\n  -v value\n        log level for V logs\n  -versions\n        include GCS object versions\n  -vmodule value\n        comma-separated list of pattern=N settings for file-filtered logging\n```\n\nYou can also use the supplied `run.sh` script, which creates the bucket inventory and uploads the inventory to a BigQuery table. This script accepts the following\nenvironment variables as input:\n\n- `BUCKET2BQ_BUCKET`: GCS bucket name to inventory.\n- `BUCKET2BQ_PROJECT`: project ID where the scratch storage bucket and BigQuery dataset resides in\n- `BUCKET2BQ_DATASET`: BigQuery dataset name (eg. `gcs2bq`)\n- `BUCKET2BQ_TABLE`: BigQuery table name (eg. `objects`)\n- `BUCKET2BQ_SCRATCH_BUCKET`: Bucket for storing the temporary Avro file to be loaded into BigQuery (no `gs://` prefix)\n- `BUCKET2BQ_LOCATION`: Location for the bucket and dataset (if they need to be created, eg. `EU`)\n- `BUCKET2BQ_VERSIONS`: Set to non-empty if you want to retrieve object versions as well\n\n### Installing\n\nDocker containers with this application are publicly available at `ghcr.io/brews/bucket2bq`.\n\nYou can also install the binary to create the inventory file on your computer by running:\n\n```bash\ngo install github.com/brews/bucket2bq@latest\n```\n\n### Building\n\nYou can build it either manually, or using the supplied `Dockerfile`:\n\n```bash\ndocker build -t bucket2bq .\n```\n\n## Support\n\nSource code is available online at https://github.com/brews/bucket2gcs. \n\nPlease file bugs in at https://github.com/brews/bucket2bq/issues.\n\nThis software is available under the Apache License, Version 2.0.\n\nThis software is a modification of the \"gcs2bq\" tool, available from https://github.com/GoogleCloudPlatform/professional-services/tree/main/tools/gcs2bq under an Apache-2.0 license.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrews%2Fbucket2bq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrews%2Fbucket2bq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrews%2Fbucket2bq/lists"}