{"id":29473424,"url":"https://github.com/ayeujjawalsingh/databricks","last_synced_at":"2025-07-14T15:38:50.707Z","repository":{"id":301947733,"uuid":"1010746986","full_name":"ayeujjawalsingh/Databricks","owner":"ayeujjawalsingh","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-13T19:48:06.000Z","size":167,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-13T21:13:07.650Z","etag":null,"topics":["change-data-capture","change-data-feed","data-engineer","data-pipeline","data-processing","databricks","deduplication","delta-lake","delta-live-tables","delta-table","medallion-architecture"],"latest_commit_sha":null,"homepage":"https://www.linkedin.com/in/ayeujjawalsingh/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ayeujjawalsingh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-29T18:07:41.000Z","updated_at":"2025-07-13T19:48:09.000Z","dependencies_parsed_at":"2025-07-13T21:03:23.806Z","dependency_job_id":"280eb60d-6de9-4d9c-80e0-264a15874275","html_url":"https://github.com/ayeujjawalsingh/Databricks","commit_stats":null,"previous_names":["ayeujjawalsingh/databricks"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ayeujjawalsingh/Databricks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ayeujjawalsingh%2FDatabricks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ayeujjawalsingh%2FDatabricks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ayeujjawalsingh%2FDatabricks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ayeujjawalsingh%2FDatabricks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ayeujjawalsingh","download_url":"https://codeload.github.com/ayeujjawalsingh/Databricks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ayeujjawalsingh%2FDatabricks/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265313469,"owners_count":23745190,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["change-data-capture","change-data-feed","data-engineer","data-pipeline","data-processing","databricks","deduplication","delta-lake","delta-live-tables","delta-table","medallion-architecture"],"created_at":"2025-07-14T15:38:46.800Z","updated_at":"2025-07-14T15:38:50.691Z","avatar_url":"https://github.com/ayeujjawalsingh.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# πŸ—ΊοΈ Databricks Learning Roadmap\n\nWelcome to the **Databricks Knowledge Base**!\n\n---\n\n## πŸš€ Getting Started\n\n### πŸ”Ή Databricks: Introduction and Architecture\n- πŸ“„ [Introduction and Architecture](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%3A%20Introduction%20and%20Architecture/Readme.md)\n\n---\n\n## 🧰 Utilities\n\n### πŸ”Ή Databricks Services\n- πŸ“„ [Services Overview](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks-Services/Readme.md)\n- πŸ“„ [DBFS, DBUTILS, Magic Commands, and Widgets](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks-Services/DBFS%2C%20DBUTILS%2C%20Magic%20Commands%2C%20and%20Widgets/Readme.md)\n\n---\n\n## 🧱 Lakehouse \u0026 Delta Lake\n\n### πŸ”Ή Delta Lake\n- πŸ“„ [Overview](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Readme.md)\n- πŸ“„ [DML (INSERT, UPDATE, DELETE, MERGE)](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/DML(INSERT%2C%20UPDATE%2C%20DELETE%2C%20and%20MERGE)/Readme.md)\n- πŸ“„ [Data Warehouse vs Lake vs Lakehouse](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Data%20Warehouse%20vs%20Data%20Lake%20vs%20Data%20Lakehouse/Readme.md)\n- πŸ“„ [Delta Table Types and Features](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Delta%20Table%20Types(Managed%20and%20External%20Table)%20(Creation%2C%20Partitioning%2C%20Schema%20Evolution%2C%20Optimization)/Readme.md)\n- πŸ“„ [Delta Transaction Logs](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Delta%20Transaction%20Logs/Readme.md)\n- πŸ“„ [History](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/History/Readme.md)\n- πŸ“„ [Incremental ETL](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Incremental%20ETL%20with%20Delta/Readme.md)\n- πŸ“„ [Medallion Lakehouse Architecture](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Medallion%20Lakehouse%20Architecture/Readme.md)\n- πŸ“„ [OPTIMIZE and Z-ORDER](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/OPTIMIZE%20and%20Z-ORDER/Readme.md)\n- πŸ“„ [Parquet Table to Delta Table](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Parquet%20Table%20to%20Delta%20Table/Readme.md)\n- πŸ“„ [Pitfalls of Data Lakes](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Pitfalls%20of%20Data%20Lakes/Readme.md)\n- πŸ“„ [Time Travel](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Time-Travel/Readme.md)\n- πŸ“„ [Vacuum](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta-Lake/Vacuum/Readme.md)\n\n---\n\n## 🧩 Data Management\n\n### πŸ”Ή Modelling Data Management Solutions\n- πŸ“„ [Autoload into Multiplex Bronze](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Autoload%20into%20Multiplex%20Bronze/Readme.md)\n- πŸ“„ [Bronze Ingestion Patterns](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Bronze%20Ingestion%20Patterns/Readme.md)\n- πŸ“„ [Quality Enforcement](https://github.com/ayeujjawalsingh/Databricks/tree/main/Modelling%20Data%20Management%20Solutions/Quality%20Enforcement/Readme.md)\n- πŸ“„ [Slowly Changing Dimensions (SCD)](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Slowly%20Changing%20Dimensions%20(SCD)/Readme.md)\n- πŸ“„ [Streaming Deduplication](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Streaming%20Deduplication/Readme.md)\n- πŸ“„ [Streaming from Multiplex Bronze](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Streaming%20from%20Multiplex%20Bronze/Reamd.md)\n- πŸ“„ [Type 2 SCD](https://github.com/ayeujjawalsingh/Databricks/blob/main/Modelling%20Data%20Management%20Solutions/Type%202%20SCD/Readme.md)\n\n---\n\n## βš™οΈ Data Engineering \u0026 Processing\n\n### πŸ”Ή Data Processing\n- πŸ“„ [Change Data Capture](https://github.com/ayeujjawalsingh/Databricks/blob/main/Data%20Processing/Change%20Data%20Capture/Readme.md)\n- πŸ“„ [Materialized Gold Tables](https://github.com/ayeujjawalsingh/Databricks/blob/main/Data%20Processing/Materialized%20Gold%20Tables/Readme.md)\n- πŸ“„ [Stream-Stream and Stream-Static Joins](https://github.com/ayeujjawalsingh/Databricks/blob/main/Data%20Processing/Stream-Stream%20and%20Stream-Static%20Joins/Readme.md)\n\n### πŸ”Ή Delta Live Tables (DLT)\n- πŸ“„ [Introduction to DLT](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta%20Live%20Table/Introductions/Readme.md)\n- πŸ“„ [Define Pipeline, Tables, and Views](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta%20Live%20Table/Define%20Pipeline%2C%20Tables%2C%20and%20Views%20with%20DLT/Readme.md)\n- πŸ“„ [Incremental Ingest Using SQL + Auto Loader](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta%20Live%20Table/Ingest%20Raw%20Data%20Incrementally%20Using%20SQL%20%2B%20Auto%20Loader%20in%20DLT/Readme.md)\n- πŸ“„ [Test and Deploy DLT](https://github.com/ayeujjawalsingh/Databricks/blob/main/Delta%20Live%20Table/Test%20and%20Deploy%20the%20DLT/Readme.md)\n\n---\n\n## 🧠 SQL Analytics\n\n### πŸ”Ή Databricks SQL\n- πŸ“„ [Build Visualizations (Dashboards)](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Build%20Visualizations%20(Dashboards)/Readme.md)\n- πŸ“„ [Configuring SQL Warehouse](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Configuring%20SQL%20Warehouse/Readme.md)\n- πŸ“„ [Creating Queries and Parameterized Queries](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Creating%20Queries%20and%20Parameterized%20Queries/Readme.md)\n- πŸ“„ [Databricks SQL - CLI](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Databricks%20SQL%20-%20CLI/Readme.md)\n- πŸ“„ [Databricks SQL – Components](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Databricks%20SQL%20%E2%80%93%20Components/Readme.md)\n- πŸ“„ [SQL Alerts](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/SQL%20Alerts/Readme.md)\n- πŸ“„ [Transfer Ownership](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20SQL/Transfer%20Ownership/Readme.md)\n\n---\n\n## πŸ›‘οΈ Data Governance \u0026 Security\n\n### πŸ”Ή Data Governance (Manage Permission)\n- πŸ“„ [Unity Catalog](https://github.com/ayeujjawalsingh/Databricks/blob/main/Data%20Governance%20(Manage%20Permission)/Unity%20Catalog/Readme.md)\n- πŸ“„ [Users, Groups, Access Control, and Delta Sharing](https://github.com/ayeujjawalsingh/Databricks/blob/main/Data%20Governance%20(Manage%20Permission)/Users%2C%20Groups%2C%20Access%20Control%2C%20and%20Delta%20Sharing/Readme.md)\n\n### πŸ”Ή Databricks Security\n- πŸ“„ [Security Overview](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks-Security/Readme.md)\n\n---\n\n## πŸ”§ Tooling \u0026 DevOps\n\n### πŸ”Ή Databricks Tooling\n- πŸ“„ [Databricks Job Configuration](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20Tooling/Databricks%20Job%20Configuration/Readme.md)\n- πŸ“„ [Databricks CLI](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20Tooling/Databricks-CLI/Readme.md)\n- πŸ“„ [Orchestration and Scheduling with Multi-Task Jobs](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20Tooling/Orchestration%20and%20Scheduling%20with%20Multi-Task%20Jobs/Readme.md)\n- πŸ“„ [REST API](https://github.com/ayeujjawalsingh/Databricks/blob/main/Databricks%20Tooling/Rest-API/Readme.md)\n\n### πŸ”Ή Testing and Deployment\n- πŸ“„ [Testing \u0026 CI/CD](https://github.com/ayeujjawalsingh/Databricks/blob/main/Testing%20and%20Deployment/Readme.md)\n\n---\n## πŸ“Š BI \u0026 Data Visualization\n\n### πŸ”Ή BI Tools Integration\n- πŸ“„ [BI Tools Integration Overview](https://github.com/ayeujjawalsingh/Databricks/blob/main/BI%20Tools%20Integration/Readme.md)\n\n---\n\n## 🧩 Advanced Patterns\n\n### πŸ”Ή High Availability and Disaster Recovery\n- πŸ“„ [HA and DR](https://github.com/ayeujjawalsingh/Databricks/blob/main/High%20Availability%20%20and%20Disaster%20Recovery/Readme.md)\n\n---\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fayeujjawalsingh%2Fdatabricks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fayeujjawalsingh%2Fdatabricks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fayeujjawalsingh%2Fdatabricks/lists"}