{"id":22123920,"url":"https://github.com/nownabe/go-bqloader","last_synced_at":"2025-08-04T21:42:32.563Z","repository":{"id":36971492,"uuid":"308901311","full_name":"nownabe/go-bqloader","owner":"nownabe","description":"bqloader is a simple ETL framework to load data from Cloud Storage into BigQuery.","archived":false,"fork":false,"pushed_at":"2024-10-23T23:23:45.000Z","size":501,"stargazers_count":21,"open_issues_count":11,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-10-24T12:53:08.459Z","etag":null,"topics":["bigquery","etl","golang","google-cloud","google-cloud-functions","google-cloud-storage"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nownabe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-31T14:36:23.000Z","updated_at":"2024-10-04T18:13:35.000Z","dependencies_parsed_at":"2024-01-18T12:12:14.464Z","dependency_job_id":"8c98306e-b2b7-4689-b112-6f3adb701dce","html_url":"https://github.com/nownabe/go-bqloader","commit_stats":{"total_commits":230,"total_committers":3,"mean_commits":76.66666666666667,"dds":"0.24782608695652175","last_synced_commit":"ce0961a995892720dccf374cae44bd4466a5301d"},"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nownabe%2Fgo-bqloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nownabe%2Fgo-bqloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nownabe%2Fgo-bqloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nownabe%2Fgo-bqloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nownabe","download_url":"https://codeload.github.com/nownabe/go-bqloader/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227590148,"owners_count":17790446,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","etl","golang","google-cloud","google-cloud-functions","google-cloud-storage"],"created_at":"2024-12-01T15:45:25.375Z","updated_at":"2024-12-01T15:45:26.143Z","avatar_url":"https://github.com/nownabe.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"bqloader\n========\n\n[![PkgGoDev](https://pkg.go.dev/badge/go.nownabe.dev/bqloader)](https://pkg.go.dev/go.nownabe.dev/bqloader)\n![Main Branch Workflow](https://github.com/nownabe/go-bqloader/workflows/Main%20Branch%20Workflow/badge.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/nownabe/go-bqloader)](https://goreportcard.com/report/github.com/nownabe/go-bqloader)\n[![codecov](https://codecov.io/gh/nownabe/go-bqloader/branch/main/graph/badge.svg)](https://codecov.io/gh/nownabe/go-bqloader)\n![GitHub](https://img.shields.io/github/license/nownabe/go-bqloader)\n![GitHub tag (latest SemVer)](https://img.shields.io/github/v/tag/nownabe/go-bqloader?sort=semver)\n\nbqloader is a simple ETL framework to load data from Cloud Storage into BigQuery.\n\n## Installation\n\n```bash\ngo get -u go.nownabe.dev/bqloader\n```\n\n## Getting Started with Pre-configured Handlers\n\nSee the [example](https://github.com/nownabe/go-bqloader/tree/main/examples/pre_configured_handlers) to get a full instruction.\n\nTo load some types of CSV formats, you can use pre-configured handlers.\nSee [full list](https://github.com/nownabe/go-bqloader/tree/main/contrib/handlers).\n\n\n```go\npackage myfunc\n\nimport (\n\t\"context\"\n\n\t\"go.nownabe.dev/bqloader\"\n\t\"go.nownabe.dev/bqloader/contrib/handlers\"\n)\n\nvar loader bqloader.BQLoader\n\nfunc init() {\n\tloader, _ = bqloader.New()\n\n\tt := handlers.TableGenerator(os.Getenv(\"BIGQUERY_PROJECT_ID\"), os.Getenv(\"BIGQUERY_DATASET_ID\"))\n\tn := \u0026bqloader.SlackNotifier{\n\t\tToken:   os.Getenv(\"SLACK_TOKEN\"),\n\t\tChannel: os.Getenv(\"SLACK_CHANNEL\"),\n\t}\n\n\thandlers.MustAddHandlers(context.Background(), loader,\n\t\t/*\n\t\t\tThese build handlers to load CSVs, given four arguments:\n\t\t\thandler name, a pattern to file path on Cloud Storage, a BigQuery table and a notifier.\n\t\t*/\n\t\thandlers.SBISumishinNetBankStatement(\"SBI Bank\", `^sbi_bank/`, t(\"sbi_bank\"), n),\n\t\thandlers.SMBCCardStatement(\"SMBC Card\", `^smbc_card/`, t(\"smbc_card\"), n),\n\t)\n}\n\n// BQLoad is the entrypoint for Cloud Functions.\nfunc BQLoad(ctx context.Context, e bqloader.Event) error {\n\treturn loader.Handle(ctx, e)\n}\n```\n\n\n## Getting Started with Custom Handlers\n\n(See [Quickstart example](https://github.com/nownabe/go-bqloader/tree/main/examples/quickstart) to get a full instruction.)\n\nTo load other CSVs, import the package `go.nownabe.dev/bqloader` and write your custom handler.\n\n```go\npackage myfunc\n\nimport (\n\t\"context\"\n\t\"os\"\n\t\"regexp\"\n\t\"strings\"\n\t\"time\"\n\n\t\"golang.org/x/text/encoding/japanese\"\n\t\"golang.org/x/xerrors\"\n\n\t\"go.nownabe.dev/bqloader\"\n)\n\nvar loader bqloader.BQLoader\n\nfunc init() {\n\tloader, _ = bqloader.New()\n\tloader.MustAddHandler(context.Background(), newHandler())\n}\n\nfunc newHandler() *bqloader.Handler {\n\t/*\n\t\tThis projector converts date fields formatted as \"2006/01/02\"\n\t\tat the first column into strings like \"2006-01-02\" that satisfies\n\t\tBigQuery date type.\n\t*/\n\tprojector := func(_ context.Context, r []string) ([]string, error) {\n\t\tt, err := time.Parse(\"2006/01/02\", r[0])\n\t\tif err != nil {\n\t\t\treturn nil, xerrors.Errorf(\"Column 0 cannot parse as a date: %w\", err)\n\t\t}\n\n\t\tr[0] = t.Format(\"2006-01-02\")\n\n\t\treturn r, nil\n\t}\n\n\treturn \u0026bqloader.Handler{\n\t\tName:     \"quickstart\",                         // Handler name used in logs and notifications.\n\t\tPattern:  regexp.MustCompile(\"^example_bank/\"), // This handler processes files matched to this pattern.\n\t\tEncoding: japanese.ShiftJIS,                    // Source file encoding.\n\t\tParser:   bqloader.CSVParser(),                 // Parser parses source file into records.\n\t\tNotifier: \u0026bqloader.SlackNotifier{\n\t\t\tToken:   os.Getenv(\"SLACK_TOKEN\"),\n\t\t\tChannel: os.Getenv(\"SLACK_CHANNEL\"),\n\t\t},\n\t\tProjector:       projector, // Projector transforms each row.\n\t\tSkipLeadingRows: 1,         // Skip header row.\n\n\t\t// Destination.\n\t\tProject: os.Getenv(\"BIGQUERY_PROJECT_ID\"),\n\t\tDataset: os.Getenv(\"BIGQUERY_DATASET_ID\"),\n\t\tTable:   os.Getenv(\"BIGQUERY_TABLE_ID\"),\n\t}\n}\n\n// BQLoad is the entrypoint for Cloud Functions.\nfunc BQLoad(ctx context.Context, e bqloader.Event) error {\n\treturn loader.Handle(ctx, e)\n}\n```\n\n## Diagram\n\n![diagram](https://raw.githubusercontent.com/nownabe/go-bqloader/main/diagram.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnownabe%2Fgo-bqloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnownabe%2Fgo-bqloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnownabe%2Fgo-bqloader/lists"}