{"id":13412235,"url":"https://github.com/viant/bgc","last_synced_at":"2025-07-30T19:44:58.994Z","repository":{"id":57491756,"uuid":"61067172","full_name":"viant/bgc","owner":"viant","description":"Datastore Connectivity for BigQuery in go","archived":false,"fork":false,"pushed_at":"2024-03-17T20:59:48.000Z","size":124,"stargazers_count":22,"open_issues_count":0,"forks_count":8,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-07-31T20:50:04.449Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/viant.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-06-13T20:24:26.000Z","updated_at":"2024-06-20T01:45:14.000Z","dependencies_parsed_at":"2024-06-18T20:02:57.488Z","dependency_job_id":"30cfedc4-e51b-4297-bd34-7e42c4320e7d","html_url":"https://github.com/viant/bgc","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/viant/bgc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viant%2Fbgc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viant%2Fbgc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viant%2Fbgc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viant%2Fbgc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/viant","download_url":"https://codeload.github.com/viant/bgc/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viant%2Fbgc/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267929449,"owners_count":24167453,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:01:22.443Z","updated_at":"2025-07-30T19:44:58.946Z","avatar_url":"https://github.com/viant.png","language":"Go","readme":"# Datastore Connectivity for BigQuery (bgc)\n\n[![Datastore Connectivity library for BigQuery in Go.](https://goreportcard.com/badge/github.com/viant/bgc)](https://goreportcard.com/report/github.com/viant/bgc)\n[![GoDoc](https://godoc.org/github.com/viant/bgc?status.svg)](https://godoc.org/github.com/viant/bgc)\n\nThis library is compatible with Go 1.5+\n\nPlease refer to [`CHANGELOG.md`](CHANGELOG.md) if you encounter breaking changes.\n\n- [Usage](#Usage)\n- [License](#License)\n- [Credits and Acknowledgements](#Credits-and-Acknowledgements)\n\n\n\n\nThis library uses SQL mode and streaming API to insert data as default.\nTo use legacy SQL please use the following /* USE LEGACY SQL */ hint, in this case you will not be able to fetch repeated and nested fields.\n\n\n#### Configuration parameters\n\n###### insertMethod\nTo control insert method just provide config.parameters with the following value:\n    \n    _table_name_.insertMethod = \"load\"\n\nNote that if streaming is used, currently UPDATE and DELETE statements are not supported.\n\n###### insertIdColumn\nFor streaming you can specify which column to use as insertId with the following config.params    \n    \n    _table_name_.insertMethod = \"stream\"\n    _table_name_.insertIdColumn = \"sessionId\"\n\n###### streamBatchCount\n\nstreamBatchCount controls row count in batch (default 9999)\n\n###### insertWaitTimeoutInMs\n\nWhen inserting data data this library checks upto 60 sec if data has been added.\nTo control this behaviour you can set insertWaitTimeoutInMs (default 60 sec)\n\nTo disable this mechanism set: \n     insertWaitTimeoutInMs: -1\n\n###### insertMaxRetires\n\nRetries insert when 503 internal error\n\n###### datasetId \n\nDefault dataset\n\n###### pageSize\n\nDefault 500\n\nThe maximum number of rows of data to return per page of results. \nIn addition to this limit, responses are also limited to 10 MB. \n\n\n## Credentials\n\n1. Google secrets for service account\n\n\na) set GOOGLE_APPLICATION_CREDENTIALS environment variable\n\n\nb) credential can be a name with extension of the JSON secret file placed into ~/.secret/ folder\n\nconfig.yaml\n```yaml\ndriverName: bigquery\ncredentials: bq # place your big query secret json to ~/.secret/bg.json\nparameters:\n  datasetId: myDataset\n```\n\nc) full URL to secret file\n\nconfig.yaml\n```yaml\ndriverName: bigquery\ncredentials: file://tmp/secret/mySecret.json\nparameters:\n  datasetId: myDataset\n```\n\n[Secret file](https://github.com/viant/toolbox/blob/master/cred/config.go) has to specify the following attributes:\n\n````go\ntype Config struct {\n\t//google cloud credential\n\tClientEmail  string `json:\"client_email,omitempty\"`\n\tTokenURL     string `json:\"token_uri,omitempty\"`\n\tPrivateKey   string `json:\"private_key,omitempty\"`\n\tPrivateKeyID string `json:\"private_key_id,omitempty\"`\n\tProjectID  string `json:\"project_id,omitempty\"`\n}\n````\n\n\n2. Private key (pem)\n\n\nconfig.yaml\n```yaml\ndriverName: bigquery\ncredentials: bq # place your big query secret json to ~/.secret/bg.json\nparameters:\n  serviceAccountId: \"***@developer.gserviceaccount.com\"\n  datasetId: MyDataset\n  projectId: spheric-arcadia-98015\n  privateKeyPath: /tmp/secret/bq.pem\n```\n\n\n\n## Usage:\n\nThe following is a very simple example of Reading and Inserting data\n\n\n```go\n\npackage main\n\nimport (\n    \"github.com/viant/bgc\"\n    \"github.com/viant/dsc\"\n    \"time\"\n    \"fmt\"\n    \"log\"\n)\n\n\ntype MostLikedCity struct {\n\tCity      string\n\tVisits    int\n\tSouvenirs []string\n}\n\ntype  Traveler struct {\n\tId            int\n\tName          string\n\tLastVisitTime time.Time\n\tAchievements  []string\n\tMostLikedCity MostLikedCity\n\tVisitedCities []struct {\n\t\tCity   string\n\t\tVisits int\n\t}\n}\n\n\nfunc main() {\n\n    config, err := dsc.NewConfigWithParameters(\"bigquery\", \"\",\n    \t    \"bq\", // google cloud secret placed in ~/.secret/bg.json\n            map[string]string{\n                \"datasetId\":\"MyDataset\",\n            })\n\n    if err != nil {\n        log.Fatal(err)\n    }\n\n\t\t\n    factory := dsc.NewManagerFactory()\n    manager, err := factory.Create(config)\n    if err != nil {\n        log.Fatalf(\"Failed to create manager %v\", err)\n    }\n   \n\n    traveler := Traveler{}\n    success, err := manager.ReadSingle(\u0026traveler, \" SELECT id, name, lastVisitTime, visitedCities, achievements, mostLikedCity FROM travelers WHERE id = ?\", []interface{}{4}, nil)\n    if err != nil {\n        panic(err.Error())\n    }\n\n    travelers :=  make([]Traveler, 0)\n    err:= manager.ReadAll(\u0026interest, \"SELECT iid, name, lastVisitTime, visitedCities, achievements, mostLikedCity\",nil, nil)\n    if err != nil {\n        panic(err.Error())\n    }\n\n   // ...\n\n    inserted, updated, err := manager.PersistAll(\u0026travelers, \"travelers\", nil)\n    if err != nil {\n           panic(err.Error())\n    }\n    // ...\n    \n\n\n\n\n   //Custom reading handler with reading query info type to get CacheHit, TotalRows, TotalBytesProcessed\n\n   var resultInfo = \u0026bgc.QueryResultInfo{}\n   var perf = make(map[string]int)  \n   \terr = manager.ReadAllWithHandler(`SELECT DATE(date), COUNT(*) FROM performance_agg WHERE DATE(date) = ?  GROUP BY 1`, []interface{}{\n   \t\t\"2018-05-03\",\n   \t\tresultInfo,\n   \t}, func(scanner dsc.Scanner) (toContinue bool, err error) {\n   \t        var date string\n   \t        var count int\n   \t        err = scanner.Scan(\u0026date, \u0026count)\n   \t        if err != nil {\n   \t        \treturn false, err\n   \t        }\n   \t        perf[date] = count\n   \t\treturn true, nil\n   \t})\n   \tlog.Printf(\"cache: %v,  rows: %v, bytes: %v\", resultInfo.CacheHit, resultInfo.TotalRows, resultInfo.TotalBytesProcessed)\n\n   \n    dialect := dsc.GetDatastoreDialect(config.DriverName)\n    DDL, err := dialect.ShowCreateTable(manager, \"performance_agg\")\n    fmt.Printf(\"%v %v\\n\", DDL, err)\n   \n}\n```\n\n## GoCover\n\n[![GoCover](https://gocover.io/github.com/viant/bgc)](https://gocover.io/github.com/viant/bgc)\n\n\n\u003ca name=\"License\"\u003e\u003c/a\u003e\n## License\n\nThe source code is made available under the terms of the Apache License, Version 2, as stated in the file `LICENSE`.\n\nIndividual files may be made available under their own specific license,\nall compatible with Apache License, Version 2. Please see individual files for details.\n\n\n\u003ca name=\"Credits-and-Acknowledgements\"\u003e\u003c/a\u003e\n\n##  Credits and Acknowledgements\n\n**Library Author:** Adrian Witas\n\n**Contributors:** Mikhail Berlyant","funding_links":[],"categories":["Database Drivers","数据库驱动程序","数据库驱动`连接和操作数据库工具`","数据库驱动","Generators","數據庫驅動","Data Integration Frameworks","\u003cspan id=\"数据库驱动-database-drivers\"\u003e数据库驱动 Database Drivers\u003c/span\u003e","Relational Databases"],"sub_categories":["Relational Database Drivers","Advanced Console UIs","关系数据库驱动程序","SQL 查询语句构建库","高級控制台界面","\u003cspan id=\"高级控制台用户界面-advanced-console-uis\"\u003e高级控制台用户界面 Advanced Console UIs\u003c/span\u003e","高级控制台界面"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviant%2Fbgc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fviant%2Fbgc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviant%2Fbgc/lists"}