https://github.com/dev-vivekkumarverma/pyspark-databricks

spark, databricks, kafka, batch and stream-processing
https://github.com/dev-vivekkumarverma/pyspark-databricks

airflow batch-processing csv databricks delta-tables distributed-computing etl-pipeline file-formats json kafka medallion-architecture parquet pyspark python3 s3 spark stream-processing unity-catalog watermarking

Last synced: 28 days ago
JSON representation

spark, databricks, kafka, batch and stream-processing

• Host: GitHub
• URL: https://github.com/dev-vivekkumarverma/pyspark-databricks
• Owner: dev-vivekkumarverma
• Created: 2024-09-18T05:09:18.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2025-01-31T11:52:26.000Z (over 1 year ago)
• Last Synced: 2025-10-09T05:35:17.360Z (8 months ago)
• Topics: airflow, batch-processing, csv, databricks, delta-tables, distributed-computing, etl-pipeline, file-formats, json, kafka, medallion-architecture, parquet, pyspark, python3, s3, spark, stream-processing, unity-catalog, watermarking
• Language: Jupyter Notebook
• Homepage:
• Size: 11.4 MB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# pyspark-databricks

`topics covered`

-  pyspark

-  databricks

-  kafka

-  csv, json, parquet, delta, text

-  batch data processing

-  stream-processing

-  s3, ADLS-v2

-  unity catalog

-  watermarking

-  query optimization

-  AQE ( adaptive query execution )