https://github.com/ayeujjawalsingh/databricks
https://github.com/ayeujjawalsingh/databricks
change-data-capture change-data-feed data-engineer data-pipeline data-processing databricks deduplication delta-lake delta-live-tables delta-table medallion-architecture
Last synced: 7 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/ayeujjawalsingh/databricks
- Owner: ayeujjawalsingh
- Created: 2025-06-29T18:07:41.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2025-07-13T19:48:06.000Z (7 months ago)
- Last Synced: 2025-07-13T21:13:07.650Z (7 months ago)
- Topics: change-data-capture, change-data-feed, data-engineer, data-pipeline, data-processing, databricks, deduplication, delta-lake, delta-live-tables, delta-table, medallion-architecture
- Homepage: https://www.linkedin.com/in/ayeujjawalsingh/
- Size: 163 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# 🗺️ Databricks Learning Roadmap
Welcome to the **Databricks Knowledge Base**!
---
## 🚀 Getting Started
### 🔹 Databricks: Introduction and Architecture
- 📄 [Introduction and Architecture](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%3A%20Introduction%20and%20Architecture/Readme.md)
---
## 🧰 Utilities
### 🔹 Databricks Services
- 📄 [Services Overview](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks-Services/Readme.md)
- 📄 [DBFS, DBUTILS, Magic Commands, and Widgets](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks-Services/DBFS%2C%20DBUTILS%2C%20Magic%20Commands%2C%20and%20Widgets/Readme.md)
---
## 🧱 Lakehouse & Delta Lake
### 🔹 Delta Lake
- 📄 [Overview](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Readme.md)
- 📄 [DML (INSERT, UPDATE, DELETE, MERGE)](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/DML(INSERT%2C%20UPDATE%2C%20DELETE%2C%20and%20MERGE)/Readme.md)
- 📄 [Data Warehouse vs Lake vs Lakehouse](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Data%20Warehouse%20vs%20Data%20Lake%20vs%20Data%20Lakehouse/Readme.md)
- 📄 [Delta Table Types and Features](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Delta%20Table%20Types(Managed%20and%20External%20Table)%20(Creation%2C%20Partitioning%2C%20Schema%20Evolution%2C%20Optimization)/Readme.md)
- 📄 [Delta Transaction Logs](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Delta%20Transaction%20Logs/Readme.md)
- 📄 [History](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/History/Readme.md)
- 📄 [Incremental ETL](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Incremental%20ETL%20with%20Delta/Readme.md)
- 📄 [Medallion Lakehouse Architecture](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Medallion%20Lakehouse%20Architecture/Readme.md)
- 📄 [OPTIMIZE and Z-ORDER](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/OPTIMIZE%20and%20Z-ORDER/Readme.md)
- 📄 [Parquet Table to Delta Table](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Parquet%20Table%20to%20Delta%20Table/Readme.md)
- 📄 [Pitfalls of Data Lakes](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Pitfalls%20of%20Data%20Lakes/Readme.md)
- 📄 [Time Travel](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Time-Travel/Readme.md)
- 📄 [Vacuum](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Vacuum/Readme.md)
---
## 🧩 Data Management
### 🔹 Modelling Data Management Solutions
- 📄 [Autoload into Multiplex Bronze](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Autoload%20into%20Multiplex%20Bronze/Readme.md)
- 📄 [Bronze Ingestion Patterns](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Bronze%20Ingestion%20Patterns/Readme.md)
- 📄 [Quality Enforcement](https://github.com/ayeujjawalsingh/Databricks/tree/main/Modelling%20Data%20Management%20Solutions/Quality%20Enforcement/Readme.md)
- 📄 [Slowly Changing Dimensions (SCD)](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Slowly%20Changing%20Dimensions%20(SCD)/Readme.md)
- 📄 [Streaming Deduplication](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Streaming%20Deduplication/Readme.md)
- 📄 [Streaming from Multiplex Bronze](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Streaming%20from%20Multiplex%20Bronze/Reamd.md)
- 📄 [Type 2 SCD](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Type%202%20SCD/Readme.md)
---
## ⚙️ Data Engineering & Processing
### 🔹 Data Processing
- 📄 [Change Data Capture](https://github.com/ayeujjawalsingh/Databricks/blob/main/Data%20Processing/Change%20Data%20Capture/Readme.md)
- 📄 [Materialized Gold Tables](https://github.com/ayeujjawalsingh/Databricks/blob/main/Data%20Processing/Materialized%20Gold%20Tables/Readme.md)
- 📄 [Stream-Stream and Stream-Static Joins](https://github.com/ayeujjawalsingh/Databricks/blob/main/Data%20Processing/Stream-Stream%20and%20Stream-Static%20Joins/Readme.md)
### 🔹 Delta Live Tables (DLT)
- 📄 [Introduction to DLT](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta%20Live%20Table/Introductions/Readme.md)
- 📄 [Define Pipeline, Tables, and Views](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta%20Live%20Table/Define%20Pipeline%2C%20Tables%2C%20and%20Views%20with%20DLT/Readme.md)
- 📄 [Incremental Ingest Using SQL + Auto Loader](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta%20Live%20Table/Ingest%20Raw%20Data%20Incrementally%20Using%20SQL%20%2B%20Auto%20Loader%20in%20DLT/Readme.md)
- 📄 [Test and Deploy DLT](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta%20Live%20Table/Test%20and%20Deploy%20the%20DLT/Readme.md)
---
## 🧠 SQL Analytics
### 🔹 Databricks SQL
- 📄 [Build Visualizations (Dashboards)](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Build%20Visualizations%20(Dashboards)/Readme.md)
- 📄 [Configuring SQL Warehouse](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Configuring%20SQL%20Warehouse/Readme.md)
- 📄 [Creating Queries and Parameterized Queries](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Creating%20Queries%20and%20Parameterized%20Queries/Readme.md)
- 📄 [Databricks SQL - CLI](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Databricks%20SQL%20-%20CLI/Readme.md)
- 📄 [Databricks SQL – Components](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Databricks%20SQL%20%E2%80%93%20Components/Readme.md)
- 📄 [SQL Alerts](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/SQL%20Alerts/Readme.md)
- 📄 [Transfer Ownership](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Transfer%20Ownership/Readme.md)
---
## 🛡️ Data Governance & Security
### 🔹 Data Governance (Manage Permission)
- 📄 [Unity Catalog](https://github.com/ayeujjawalsingh/Databricks/blob/main/Data%20Governance%20(Manage%20Permission)/Unity%20Catalog/Readme.md)
- 📄 [Users, Groups, Access Control, and Delta Sharing](https://github.com/ayeujjawalsingh/Databricks/blob/main/Data%20Governance%20(Manage%20Permission)/Users%2C%20Groups%2C%20Access%20Control%2C%20and%20Delta%20Sharing/Readme.md)
### 🔹 Databricks Security
- 📄 [Security Overview](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks-Security/Readme.md)
---
## 🔧 Tooling & DevOps
### 🔹 Databricks Tooling
- 📄 [Databricks Job Configuration](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20Tooling/Databricks%20Job%20Configuration/Readme.md)
- 📄 [Databricks CLI](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20Tooling/Databricks-CLI/Readme.md)
- 📄 [Orchestration and Scheduling with Multi-Task Jobs](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20Tooling/Orchestration%20and%20Scheduling%20with%20Multi-Task%20Jobs/Readme.md)
- 📄 [REST API](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20Tooling/Rest-API/Readme.md)
### 🔹 Testing and Deployment
- 📄 [Testing & CI/CD](https://github.com/ayeujjawalsingh/Databricks/blob/main/Testing%20and%20Deployment/Readme.md)
---
## 📊 BI & Data Visualization
### 🔹 BI Tools Integration
- 📄 [BI Tools Integration Overview](https://github.com/ayeujjawalsingh/Databricks/blob/main/BI%20Tools%20Integration/Readme.md)
---
## 🧩 Advanced Patterns
### 🔹 High Availability and Disaster Recovery
- 📄 [HA and DR](https://github.com/ayeujjawalsingh/Databricks/blob/main/High%20Availability%20%20and%20Disaster%20Recovery/Readme.md)
---