{"id":15060631,"url":"https://github.com/victorelexpe/bq-schema-sync","last_synced_at":"2026-02-26T00:32:33.536Z","repository":{"id":248037174,"uuid":"827574120","full_name":"victorelexpe/bq-schema-sync","owner":"victorelexpe","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-21T14:42:48.000Z","size":48,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-14T00:38:05.354Z","etag":null,"topics":["bigquery","gcp","google-cloud","python","schema","sync"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/bq-schema-sync","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/victorelexpe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-11T23:39:30.000Z","updated_at":"2024-07-22T07:52:59.000Z","dependencies_parsed_at":"2024-07-19T20:42:23.954Z","dependency_job_id":null,"html_url":"https://github.com/victorelexpe/bq-schema-sync","commit_stats":null,"previous_names":["victorelexpe/bq-schema-sync"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorelexpe%2Fbq-schema-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorelexpe%2Fbq-schema-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorelexpe%2Fbq-schema-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorelexpe%2Fbq-schema-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/victorelexpe","download_url":"https://codeload.github.com/victorelexpe/bq-schema-sync/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243180878,"owners_count":20249384,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","gcp","google-cloud","python","schema","sync"],"created_at":"2024-09-24T23:01:46.891Z","updated_at":"2025-10-24T04:24:00.775Z","avatar_url":"https://github.com/victorelexpe.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bq-schema-sync\n\n`bq-schema-sync` is a Python package designed to help synchronize Google BigQuery table schemas with local schema definitions. It provides tools for comparing schemas, applying changes, validating schema definitions, generating migration scripts, managing schema versions, and enforcing schema validation rules.\n\n## Features\n\n- **Schema Comparison**: Identify differences between local schema definitions and BigQuery table schemas.\n- **Schema Synchronization**: Apply changes to BigQuery table schemas based on local definitions.\n- **Validation**: Ensure that local schema definitions adhere to BigQuery constraints and best practices.\n- **Migration Script Generation**: Generate SQL scripts to manually apply schema changes.\n- **Schema Versioning**: Track schema changes over time, save versions, list versions, and apply specific versions.\n- **Dry Run Mode**: Preview changes without applying them to the BigQuery table.\n- **Schema Validation Rules**: Enforce custom validation rules to ensure schema definitions meet specific criteria.\n\n## Installation\n\nYou can install `bq-schema-sync` using pip:\n\n    pip install bq-schema-sync\n\n## Usage\n\n### Command-Line Interface (CLI)\n\n#### Initialize Configuration\n\nGenerate a template configuration file:\n\n    bq-schema-sync init\n\n#### Compare Schemas\n\nCompare the local schema with the BigQuery table schema:\n\n    bq-schema-sync compare --config config.yaml --dry-run\n\n#### Apply Changes\n\nSync the local schema with the BigQuery table schema:\n\n    bq-schema-sync apply --config config.yaml --dry-run\n\n#### Generate Migration Script\n\nGenerate a SQL script for manual schema migration:\n\n    bq-schema-sync generate-script --config config.yaml --output migration.sql\n\n#### Validate Schema\n\nValidate the local schema against BigQuery constraints and custom validation rules:\n\n    bq-schema-sync validate --config config.yaml\n\n#### Save Schema Version\n\nSave the current schema version with a description:\n\n    bq-schema-sync save-version --config config.yaml --description \"Added new field 'email'\"\n\n#### List Schema Versions\n\nList all saved schema versions:\n\n    bq-schema-sync list-versions --config config.yaml\n\n#### Apply Schema Version\n\nApply a specific schema version:\n\n    bq-schema-sync apply-version --config config.yaml --version 1\n\n### Python API\n\nYou can also use `bq-schema-sync` as a Python module:\n\n    from bq_schema_sync import SchemaSync\n    from google.cloud import bigquery\n\n    # Initialize SchemaSync with configuration details\n    client = bigquery.Client(project='my-gcp-project')\n    schema_sync = SchemaSync(\n        project_id='my-gcp-project',\n        dataset_id='my_dataset',\n        table_id='my_table',\n        schema={'fields': [{'name': 'id', 'type': 'STRING', 'mode': 'REQUIRED', 'description': 'Unique identifier'}]},\n        client=client\n    )\n\n    # Compare schemas\n    differences = schema_sync.compare_schemas()\n    print(\"Schema Differences:\", differences)\n\n    # Apply changes to sync the BigQuery table schema with the local schema\n    schema_sync.apply_changes()\n\n    # Generate migration script\n    schema_sync.generate_migration_script('migration_scripts/update_my_table_schema.sql')\n\n    # Validate the local schema\n    schema_sync.validate_schema()\n\n    # Save the current schema version\n    schema_sync.save_version(\"Initial schema definition\")\n\n    # List all schema versions\n    versions = schema_sync.list_versions()\n    for version in versions:\n        print(f\"Version: {version['version']}, Timestamp: {version['timestamp']}, Description: {version['description']}\")\n\n    # Apply a specific schema version\n    schema_sync.apply_version(1)\n\n## Configuration File\n\nThe configuration file should be in YAML format and include the following details:\n\n    project_id: your-gcp-project-id\n    dataset_id: your-dataset-id\n    table_id: your-table-id\n    schema:\n      fields:\n        - name: id\n          type: STRING\n          mode: REQUIRED\n          description: \"Unique identifier\"\n        - name: created_at\n          type: TIMESTAMP\n          mode: NULLABLE\n          description: \"Record creation timestamp\"\n\n## Example Schema\n\nHere is an example schema file in YAML format (`config.yaml`):\n\n    project_id: your-gcp-project-id\n    dataset_id: your-dataset-id\n    table_id: your-table-id\n    schema:\n      fields:\n        - name: id\n          type: STRING\n          mode: REQUIRED\n          description: \"Unique identifier\"\n        - name: created_at\n          type: TIMESTAMP\n          mode: NULLABLE\n          description: \"Record creation timestamp\"\n\n## Testing\n\nYou can run tests using `unittest`:\n\n    python -m unittest discover tests\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a pull request or open an issue.\n\n## Author\n\nThis project is developed and maintained by Victor Hasim Elexpe Ahamri. You can follow me on Twitter [@victorelexpe](https://twitter.com/victorelexpe) and visit my website [elexpe.dev](https://elexpe.dev).\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictorelexpe%2Fbq-schema-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvictorelexpe%2Fbq-schema-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictorelexpe%2Fbq-schema-sync/lists"}