https://github.com/googlecloudplatform/bigquery-data-lineage

Reference implementation for real-time Data Lineage tracking for BigQuery using Audit Logs, ZetaSQL and Dataflow.
https://github.com/googlecloudplatform/bigquery-data-lineage

bigdata bigquery data-catalog data-governance data-lineage data-management dataflow zetasql

Last synced: 4 months ago
JSON representation

Reference implementation for real-time Data Lineage tracking for BigQuery using Audit Logs, ZetaSQL and Dataflow.

• Host: GitHub
• URL: https://github.com/googlecloudplatform/bigquery-data-lineage
• Owner: GoogleCloudPlatform
• License: apache-2.0
• Created: 2020-06-09T05:52:32.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2024-01-22T17:23:16.000Z (about 1 year ago)
• Last Synced: 2024-01-25T12:09:51.978Z (about 1 year ago)
• Topics: bigdata, bigquery, data-catalog, data-governance, data-lineage, data-management, dataflow, zetasql
• Language: Java
• Homepage: https://cloud.google.com/blog/products/data-analytics/architecting-a-data-lineage-system-for-bigquery
• Size: 405 KB
• Stars: 132
• Watchers: 7
• Forks: 37
• Open Issues: 19
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Authors: AUTHORS

Awesome Lists containing this project

README

        
![build](https://github.com/GoogleCloudPlatform/bigquery-data-lineage/workflows/build/badge.svg)

[![Total alerts](https://img.shields.io/lgtm/alerts/g/GoogleCloudPlatform/bigquery-data-lineage.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/GoogleCloudPlatform/bigquery-data-lineage/alerts/)

[![codecov](https://codecov.io/gh/GoogleCloudPlatform/bigquery-data-lineage/branch/master/graph/badge.svg?token=WWDSB71753)](https://codecov.io/gh/GoogleCloudPlatform/bigquery-data-lineage)

Learn more on [Data lineage for a datawarehouse](https://cloud.google.com/solutions/architecture-concept-data-lineage-systems-in-a-data-warehouse)

Refer instructions for deployment

 * [Data lineage pipeline for BigQuery](documents/bigquery_lineage_pipeline.md)

 * [Automatic policy tag cascading based on data lineage pipeline](documents/cascade_bq_table_acl_pipeline.md)

   

## Disclaimer

**License**: Apache 2.0

This is not an official Google product.