{"id":13565442,"url":"https://github.com/wepay/kafka-connect-bigquery","last_synced_at":"2026-01-17T16:30:44.552Z","repository":{"id":41176359,"uuid":"65751573","full_name":"wepay/kafka-connect-bigquery","owner":"wepay","description":"DEPRECATED. PLEASE USE https://github.com/confluentinc/kafka-connect-bigquery. A Kafka Connect BigQuery sink connector","archived":false,"fork":false,"pushed_at":"2024-03-04T20:39:23.000Z","size":792,"stargazers_count":152,"open_issues_count":65,"forks_count":194,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-04-03T22:36:58.395Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wepay.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-08-15T17:34:48.000Z","updated_at":"2025-03-19T03:24:15.000Z","dependencies_parsed_at":"2023-01-25T13:25:12.299Z","dependency_job_id":"aba8d6d4-5534-4f38-99e5-9921fd5657fe","html_url":"https://github.com/wepay/kafka-connect-bigquery","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/wepay/kafka-connect-bigquery","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wepay%2Fkafka-connect-bigquery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wepay%2Fkafka-connect-bigquery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wepay%2Fkafka-connect-bigquery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wepay%2Fkafka-connect-bigquery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wepay","download_url":"https://codeload.github.com/wepay/kafka-connect-bigquery/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wepay%2Fkafka-connect-bigquery/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28511867,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T13:38:16.342Z","status":"ssl_error","status_checked_at":"2026-01-17T13:37:44.060Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:47.060Z","updated_at":"2026-01-17T16:30:44.533Z","avatar_url":"https://github.com/wepay.png","language":"Java","funding_links":[],"categories":["Java","大数据","Data Warehouses"],"sub_categories":["BigQuery"],"readme":"# REPOSITORY IS DEPRECATED. PLEASE USE https://github.com/confluentinc/kafka-connect-bigquery.\n\n# Kafka Connect BigQuery Connector\n\n[![Build Status](https://img.shields.io/travis/wepay/kafka-connect-bigquery.svg?style=flat-square)](https://travis-ci.org/wepay/kafka-connect-bigquery)\n[![Code Coverage](https://img.shields.io/codecov/c/github/wepay/kafka-connect-bigquery.svg?style=flat-square)](https://codecov.io/gh/wepay/kafka-connect-bigquery)\n\nThis is an implementation of a sink connector from [Apache Kafka] to [Google BigQuery], built on top \nof [Apache Kafka Connect]. For a comprehensive list of configuration options, see the [Connector Configuration Wiki].\n\n## Download\n\nThe latest releases are available in the GitHub release tab, or via [tarballs in Maven central](http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22kcbq-connector%22).\n\n## Standalone Quickstart\n\n\u003e **NOTE**: You must have the [Confluent Platform] installed in order to run the example.\n\n### Configuration Basics\n\nFirstly, you need to specify configuration settings for your connector. These can be found in the \n`kcbq-connector/quickstart/properties/connector.properties` file. Look for this section:\n\n```plain\n########################################### Fill me in! ###########################################\n# The name of the BigQuery project to write to\nproject=\n# The name of the BigQuery dataset to write to (leave the '.*=' at the beginning, enter your\n# dataset after it)\ndatasets=.*=\n# The location of a BigQuery service account or user JSON credentials file\n# or service account credentials or user credentials in JSON format (non-escaped JSON blob)\nkeyfile=\n# 'FILE' if keyfile is a credentials file, 'JSON' if it's a credentials JSON\nkeySource=FILE\n```\n\nYou'll need to choose a BigQuery project to write to, a dataset from that project to write to, and\nprovide the location of a JSON key file that can be used to access a BigQuery service account that\ncan write to the project/dataset pair. Once you've decided on these properties, fill them in and\nsave the properties file.\n\nOnce you get more familiar with the connector, you might want to revisit the `connector.properties`\nfile and experiment with tweaking its settings.\n   \n### Building and Extracting a Tarball\n\nIf you haven't already, move into the repository's top-level directory:\n\n```bash\n$ cd /path/to/kafka-connect-bigquery/\n```\n\nBegin by creating a tarball of the connector with the Confluent Schema Retriever included:\n\n```bash\n$ ./gradlew clean distTar\n```\n\nAnd then extract its contents:\n\n```bash\n$ mkdir -p bin/jar/ \u0026\u0026 tar -C bin/jar/ -xf kcbq-confluent/build/distributions/kcbq-confluent-*.tar\n```\n\n### Setting-Up Background Processes\n\nThen move into the `quickstart` directory:\n\n```bash\n$ cd kcbq-connector/quickstart/\n```\n\nAfter that, if your Confluent Platform installation isn't in a sibling directory to the connector, \nspecify its location (and do so before starting each of the subsequent processes in their own \nterminal):\n\n```bash\n$ export CONFLUENT_DIR=/path/to/confluent\n```\n\nThen, initialize the background processes necessary for Kafka Connect (one terminal per script):\n(Taken from http://docs.confluent.io/3.0.0/quickstart.html)\n\n```bash\n$ ./zookeeper.sh\n```\n\n(wait a little while for it to get on its feet)\n\n```bash\n$ ./kafka.sh\n```\n\n(wait a little while for it to get on its feet)\n\n```bash\n$ ./schema-registry.sh\n```\n\n(wait a little while for it to get on its feet)\n\n### Initializing the Avro Console Producer\n\nNext, initialize the Avro Console Producer (also in its own terminal):\n\n```bash\n$ ./avro-console-producer.sh\n```\n\nGive it some data to start off with (type directly into the Avro Console Producer instance):\n\n```json\n{\"f1\":\"Testing the Kafka-BigQuery Connector!\"}\n```\n\n### Running the Connector\n\nFinally, initialize the BigQuery connector (also in its own terminal):\n\n```bash\n$ ./connector.sh\n```\n\n### Piping Data Through the Connector\n\nNow you can enter Avro messages of the schema `{\"f1\": \"$SOME_STRING\"}` into the Avro Console \nProducer instance, and the pipeline instance should write them to BigQuery.\n\nIf you want to get more adventurous, you can experiment with different schemas or topics by \nadjusting flags given to the Avro Console Producer and tweaking the config settings found in the \n`kcbq-connector/quickstart/properties` directory.\n\n## Integration Testing the Connector\n\n\u003e **NOTE**: You must have [Docker] installed and running on your machine in order to run integration\ntests for the connector.\n\nThis all takes place in the `kcbq-connector` directory.\n\n### How Integration Testing Works\n\nIntegration tests run by creating [Docker] instances for [Zookeeper], [Kafka], [Schema Registry], \nand the BigQuery Connector itself, then verifying the results using a [JUnit] test.\n\nThey use schemas and data that can be found in the `test/docker/populate/test_schemas/` directory, \nand rely on a user-provided JSON key file (like in the `quickstart` example) to access BigQuery.\n\nThe project and dataset they write to, as well as the specific JSON key file they use, can be\nspecified by command-line flag, environment variable, or configuration file — the exact details of\neach can be found by running the integration test script with the `-?` flag.\n\n### Data Corruption Concerns\n\nIn order to ensure the validity of each test, any table that will be written to in the course of\nintegration testing is preemptively deleted before the connector is run. This will only be an issue\nif you have any tables in your dataset whose names begin with `kcbq_test_` and match the sanitized\nname of any of the `test_schema` subdirectories. If that is the case, you should probably consider\nwriting to a different project/dataset.\n\nBecause Kafka and Schema Registry are run in Docker, there is no risk that running integration \ntests will corrupt any existing data that is already on your machine, and there is also no need to \nfree up any of your ports that might currently be in use by real instances of the programs that are \nfaked in the process of testing.\n\n### Running the Integration Tests\n\nRunning the series of integration tests is easy:\n\n```bash\n$ test/integrationtest.sh\n```\n\nThis assumes that the project, dataset, and key file have been specified by variable or \nconfiguration file. For more information on how to specify these, run the test script with\nthe `--help` flag.\n\n\u003e **NOTE:** You must have a recent version of [boot2docker], [Docker Machine], [Docker], etc.\ninstalled. Older versions will hang when cleaning containers, and linking doesn't work properly.\n\n### Adding New Integration Tests\n\nAdding an integration test is a little more involved, and consists of two major steps: specifying\nAvro data to be sent to Kafka, and specifying via JUnit test how to verify that such data made \nit to BigQuery as expected.\n\nTo specify input data, you must create a new directory in the `test/resources/test_schemas/`\ndirectory with whatever name you want the Kafka topic of your test to be named, and whatever \nstring you want the name of your test's BigQuery table to be derived from. Then, create two files \nin that directory:\n\n* `schema.json` will contain the Avro schema of the type of data the new test will send\nthrough the connector.\n\n* `data.json` will contain a series of JSON objects, each of which should represent an [Avro] record \nthat matches the specified schema. **Each JSON object must occupy its own line, and each object \ncannot occupy more than one line** (this inconvenience is due to limitations in the Avro \nConsole Producer, and may be addressed in future commits).\n\nTo specify data verification, add a new JUnit test to the file \n`src/integration-test/java/com/wepay/kafka/connect/bigquery/it/BigQueryConnectorIntegrationTest.java`.\nRows that are retrieved from BigQuery in the test are only returned as _Lists_ of _Objects_. The \nnames of their columns are not tracked. Construct a _List_ of the _Objects_ that you expect to be \nstored in the test's BigQuery table, retrieve the actual _List_ of _Objects_ stored via a call to \n`readAllRows()`, and then compare the two via a call to `testRows()`.\n\n\u003e **NOTE**: Because the order of rows is not guaranteed when reading test results from BigQuery, \nyou must include a row number as the first field of any of your test schemas, and every row of test \ndata must have a unique value for its row number (row numbers are one-indexed).\n\n  [Apache Avro]: https://avro.apache.org\n  [Apache Kafka Connect]: http://docs.confluent.io/3.0.0/connect/\n  [Apache Kafka]: http://kafka.apache.org\n  [Apache Maven]: https://maven.apache.org\n  [Avro]: https://avro.apache.org\n  [BigQuery]: https://cloud.google.com/bigquery/\n  [boot2docker]: http://boot2docker.io\n  [Confluent Platform]: http://docs.confluent.io/3.0.0/installation.html\n  [Connector Configuration Wiki]: https://github.com/wepay/kafka-connect-bigquery/wiki/Connector-Configuration\n  [Docker Machine]: https://docs.docker.com/machine/\n  [Docker]: https://www.docker.com\n  [Google BigQuery]: https://cloud.google.com/bigquery/\n  [JUnit]: http://junit.org\n  [Kafka Connect]: http://docs.confluent.io/3.0.0/connect/\n  [Kafka]: http://kafka.apache.org\n  [Maven]: https://maven.apache.org\n  [Schema Registry]: https://github.com/confluentinc/schema-registry\n  [Semantic Versioning]: http://semver.org\n  [Zookeeper]: https://zookeeper.apache.org\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwepay%2Fkafka-connect-bigquery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwepay%2Fkafka-connect-bigquery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwepay%2Fkafka-connect-bigquery/lists"}