{"id":28172502,"url":"https://github.com/memeddev/mysql-to-google-bigquery","last_synced_at":"2025-06-29T13:32:56.552Z","repository":{"id":37547288,"uuid":"71922489","full_name":"MemedDev/mysql-to-google-bigquery","owner":"MemedDev","description":"MySQL to Google BigQuery Sync Tool","archived":false,"fork":false,"pushed_at":"2023-04-19T18:31:33.000Z","size":68,"stargazers_count":62,"open_issues_count":6,"forks_count":26,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-05-15T20:11:32.730Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MemedDev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-25T17:35:44.000Z","updated_at":"2023-10-14T15:27:46.000Z","dependencies_parsed_at":"2025-03-04T18:01:12.549Z","dependency_job_id":"9889e4c5-7cb6-443f-ad61-a63b0fa5e6bd","html_url":"https://github.com/MemedDev/mysql-to-google-bigquery","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/MemedDev/mysql-to-google-bigquery","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MemedDev%2Fmysql-to-google-bigquery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MemedDev%2Fmysql-to-google-bigquery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MemedDev%2Fmysql-to-google-bigquery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MemedDev%2Fmysql-to-google-bigquery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MemedDev","download_url":"https://codeload.github.com/MemedDev/mysql-to-google-bigquery/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MemedDev%2Fmysql-to-google-bigquery/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262600650,"owners_count":23335108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-15T20:10:42.577Z","updated_at":"2025-06-29T13:32:56.527Z","avatar_url":"https://github.com/MemedDev.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"https://cloud.githubusercontent.com/assets/2197005/19776979/f4abd1be-9c54-11e6-9842-212f26e765a5.png\" alt=\"MySQL to Google BigQuery Logo\" /\u003e\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003eMySQL to Google BigQuery Sync Tool\u003c/h1\u003e\n\n## Table of Contents\n\n+ [How it works](#how-it-works)\n+ [Requirements](#requirements)\n+ [Usage](#usage)\n+ [Credits](#credits)\n+ [License](#license)\n\n## How it works\n\nSteps when no order column has been supplied:\n\n+ Count MySQL table rows\n+ Count BigQuery table rows\n+ MySQL rows \u003e BigQuery rows?\n+ Get the rows diff, split in batches of XXXXX rows/batch\n\nSteps when order column has been supplied:\n\n+ Get max value for order column from MySQL table\n+ Get max value for order column from BigQuery table\n+ Max value MySQL \u003e Max value BigQuery?\n+ Delete all rows with order column value = max value BigQuery \nto make sure no duplicate records are being created in BigQuery\n+ Get max value for order column from BigQuery table\n+ Get the rows diff based on new max value BigQuery, \nsplit in batches of XXXXX rows/batch\n\nFinal three steps:\n\n+ Dump MySQL rows to a JSON\n+ Send JSON to BigQuery\n+ Repeat until all batches are sent\n\nTip: Create a cron job for keep syncing the tables using an interval like 15 minutes (respect the Load Jobs [quota policy](https://cloud.google.com/bigquery/quota-policy))\n\n## Requirements\n\nThe following PHP versions are supported:\n\n+ PHP 7\n+ HHVM\n+ PDO Extension with MySQL driver\n\n## Usage\n\nDownload the library using [composer](https://packagist.org/packages/memeddev/mysql-to-google-bigquery):\n\n```bash\n$ composer require memeddev/mysql-to-google-bigquery\n```\n\nNow, define some environment variables or create a `.env` file on the root of the project, replacing the values:\n\n```text\nBQ_PROJECT_ID=bigquery-project-id\nBQ_KEY_FILE=google-service-account-json-key-file.json\nBQ_DATASET=bigquery-dataset-name\n\nDB_DATABASE_NAME=mysql-database-name\nDB_USERNAME=mysql_username\nDB_PASSWORD=mysql_password\nDB_HOST=mysql-host\nDB_PORT=3306\n\nIGNORE_COLUMNS=password,hidden_column,another_column\n```\n\nPS: To create the `Google Service Account JSON Key File`, access [https://console.cloud.google.com/apis/credentials/serviceaccountkey](https://console.cloud.google.com/apis/credentials/serviceaccountkey)\n\nRun:\n\n```bash\nvendor/bin/console sync table-name\n```\n\nIf you want to auto create the table on BigQuery:\n\n```bash\nvendor/bin/console sync table-name --create-table\n```\n\nIf you want to delete (and create) the table on BigQuery for a full dump:\n\n```bash\nvendor/bin/console sync table-name --delete-table\n```\n\n## Credits\n\n:heart: Memed SA ([memed.com.br](https://memed.com.br))\n\n## License\n\nMIT license, see [LICENSE](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmemeddev%2Fmysql-to-google-bigquery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmemeddev%2Fmysql-to-google-bigquery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmemeddev%2Fmysql-to-google-bigquery/lists"}