Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/bretamyers/Azure-Synapse-Lakehouse-Sync
https://github.com/bretamyers/Azure-Synapse-Lakehouse-Sync
Last synced: 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/bretamyers/Azure-Synapse-Lakehouse-Sync
- Owner: bretamyers
- Created: 2022-08-23T16:53:12.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-04-08T01:22:36.000Z (almost 2 years ago)
- Last Synced: 2024-06-05T07:32:23.889Z (7 months ago)
- Language: Jupyter Notebook
- Size: 572 KB
- Stars: 18
- Watchers: 2
- Forks: 8
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-synapse - bretamyers/Azure-Synapse-Lakehouse-Sync - Lakehouse sync tool. (Repositories / Tools & Samples)
README
# Azure Synapse Lakehouse Sync
#### Description
Azure Synapse Lakehouse Sync provides an easy solution to synchronizing modeled Gold Zone data from your data lake, to your Synapse Analytics Data Warehouse. Through a series of Databricks notebooks and Synapse Analytics pipelines, it offers a working example of how to continually synchronize your tables.
Additionally, it leverages the new [Change Data Feed](https://docs.delta.io/2.0.0rc1/delta-change-data-feed.html) capabilities in the Delta 2.x format to better track changes to your Gold Zone tables. This allows for significantly easier and more performant extracts of changed data. Best practices are then used to stage, ingest, and store data in the most performant and optimized way within Azure Synapse Dedicated SQL. The synchronization schedule can be configured for whatever interval works best for your environment, whether it's every 10 minutes or daily.
Azure Synapse Lakehouse Sync is designed to be a fully automated, self-healing, and hands-off approach to continually synchronize your data lake with your data warehouse.
https://user-images.githubusercontent.com/16770830/192058619-0c5a4664-0662-4e9b-92f2-dbc093dfea6f.mp4
## Using Azure Synapse Lakehouse Sync
[Self Deployment](/Azure%20Synapse%20Lakehouse%20Sync): Instructions for deploying, configuring, and using Azure Synapse Lakehouse Sync in your own environment.
[Tutorial Environment](/Tutorial%20Environment): Deploys a fully working Azure Synapse Lakehouse Sync tutorial environment in your Azure Subscription. This is a great way to experience how Azure Synapse Lakehouse Sync works end-to-end.