{"id":13721102,"url":"https://github.com/kuzzleio/kdc-bigquery-connector","last_synced_at":"2025-05-07T13:31:43.462Z","repository":{"id":53548126,"uuid":"96310472","full_name":"kuzzleio/kdc-bigquery-connector","owner":"kuzzleio","description":"Plugin forwarding Enterprise Probe Measures to Google BigQuery","archived":true,"fork":false,"pushed_at":"2018-06-14T15:30:34.000Z","size":42,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-08-04T01:14:24.246Z","etag":null,"topics":["google-bigquery","kuzzle","kuzzle-analytics","kuzzle-probe"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kuzzleio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-05T11:08:33.000Z","updated_at":"2024-03-12T12:42:04.000Z","dependencies_parsed_at":"2022-09-09T09:50:55.358Z","dependency_job_id":null,"html_url":"https://github.com/kuzzleio/kdc-bigquery-connector","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuzzleio%2Fkdc-bigquery-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuzzleio%2Fkdc-bigquery-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuzzleio%2Fkdc-bigquery-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kuzzleio%2Fkdc-bigquery-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kuzzleio","download_url":"https://codeload.github.com/kuzzleio/kdc-bigquery-connector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224604761,"owners_count":17339197,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google-bigquery","kuzzle","kuzzle-analytics","kuzzle-probe"],"created_at":"2024-08-03T01:01:12.471Z","updated_at":"2024-11-14T10:31:05.669Z","avatar_url":"https://github.com/kuzzleio.png","language":"JavaScript","funding_links":[],"categories":["Plugins"],"sub_categories":["External services plugins"],"readme":"# KDC BigQuery Connector\n\n\u003e This plugin is currently Work In Progress.\n\nThis plugin enables a Kuzzle Data Collector (see the [Enterprise Probe Plugin](https://github.com/kuzzleio/kuzzle-enterprise-probe)) to forward measures to Google BigQuery, for advanced analytics.\n\nThis is achieved by listening the `plugin-kuzzle-enterprise-probe:saveMeasure` custom event (triggered by the Enterprise Probe Plugin) and by analyzing its payload against the configuration. If measures correspond to probes specified in the configuration, they are formatted and sent to Google BigQuery via the GoogleCloud SDK.\n\nThe tables corresponding to the measures in BigQuery are automatically created if they don't exist.\n\n## Pre-requisites\n\nThis plugin will not initialize unless a valid configuration is provided. Please refer to the [Kuzzle Plugins Reference](http://docs.kuzzle.io/plugins-reference/managing-plugins/#configuring-plugins) to learn how to configure a plugin.\n\nThis plugin relies on the [Enterprise Probe Plugin](https://github.com/kuzzleio/kuzzle-enterprise-probe) and relies on the `plugin-kuzzle-enterprise-probe:saveMeasure` hook to work. The Enterprise Probe Plugin must be installed and its name must not be modified in its `package.json`.\n\n## Configuration\n\nBelow is an example of configuration:\n\n```json\n{\n  \"plugins\": {\n    \"kdc-bigquery-connector\": {\n      \"projectId\": \"your-project-id\",\n      \"dataSet\": \"your-data-set\",\n      \"credentials\": {\n        \"type\": \"service_account\",\n        \"project_id\": \"your-project-id\",\n        \"private_key_id\": \"put-yours-here\",\n        \"private_key\": \"put-yours-here\",\n        \"client_email\": \"put-yours-here\",\n        \"client_id\": \"put-yours-here\",\n        \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n        \"token_uri\": \"https://accounts.google.com/o/oauth2/token\",\n        \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n        \"client_x509_cert_url\": \"\"\n      },\n      \"probes\": {\n        \"probe_watcher_1\": {\n          \"type\": \"watcher\",\n          \"tableName\": \"table_watcher\",\n          \"schema\": {\n            \"fields\": [\n              {\n                \"name\": \"field_1\",\n                \"type\": \"STRING\",\n                \"mode\": \"NULLABLE\"\n              },\n              {\n                \"name\": \"field_2\",\n                \"type\": \"INTEGER\",\n                \"mode\": \"NULLABLE\"\n              }\n            ]\n          }\n        },\n        \"probe_monitor_1\": {\n          \"type\": \"monitor\",\n          \"tableName\": \"table_monitor\",\n          \"hooks\": [\"controller:hook\", \"anotherController:anotherHook\"]\n        },\n        \"probe_counter_1\": {\n          \"type\": \"counter\",\n          \"tableName\": \"table_counter\"\n        }\n      }\n    }\n  }\n}\n```\n\n### projectId\nThe projectId of your BigQuery project. **Must exist before running the plugin**\n\n### dataSet\nThe dataset in you BigQuery project. **Must exist before running the plugin**\n\n### credentials\nThe credentials used to log in the BigQuery service. Please refer to the [Big Query User Manual](https://googlecloudplatform.github.io/google-cloud-node/#/docs/bigquery/0.9.2/guides/authentication).\n\n### probes\nThe probes you are \"listening to\" or, in other words, whose measures must be sent to BigQuery. These probes must be properly configured in the [Enterprise Probe Plugin](https://github.com/kuzzleio/kuzzle-enterprise-probe) and in the [Enterprise Probe Listener Plugin](https://github.com/kuzzleio/kuzzle-enterprise-probe-listener). Each probe must specify the following fields:\n\n* `type` (mandatory): can be `monitor`, `counter`, `watcher`, `sampler`.\n* `table_name` (optional): specifies the name of the table that will contain the measure data. If this field is not provided, the table name will be derived from the name of the probe.\n* `hooks` (mandatory - only for `monitor`): an array of hook names, which corresponds to the hooks monitored by the probe. Hook names are normalized to match valid column names (BigQuery only allows numbers, letters and underscores in column names).\n* `schema` (optional for `monitor` and `counter`, mandatory for `watcher` and `sampler`): the schema to apply to the table if created. Please refer to the [BigQuery Tables Reference](https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#resource) to learn more about the format.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuzzleio%2Fkdc-bigquery-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkuzzleio%2Fkdc-bigquery-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuzzleio%2Fkdc-bigquery-connector/lists"}