{"id":122730,"url":"https://github.com/benschr/awesome-data-engineering","name":"awesome-data-engineering","description":"A highly curated list of awesome things related to data engineering and development","projects_count":77,"last_synced_at":"2026-07-27T09:00:24.749Z","repository":{"id":298099103,"uuid":"996319410","full_name":"BenSchr/awesome-data-engineering","owner":"BenSchr","description":"A highly curated list of awesome things related to data engineering and development","archived":false,"fork":false,"pushed_at":"2025-07-08T22:12:34.000Z","size":38,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-08T12:02:49.366Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BenSchr.png","metadata":{"files":{"readme":"readme.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-06-04T19:23:40.000Z","updated_at":"2025-07-08T22:12:38.000Z","dependencies_parsed_at":"2025-06-09T12:24:50.475Z","dependency_job_id":"95510f5d-09b9-4c9a-b2c7-d43aaf102245","html_url":"https://github.com/BenSchr/awesome-data-engineering","commit_stats":null,"previous_names":["benschr/awesome-data-engineering"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BenSchr/awesome-data-engineering","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BenSchr%2Fawesome-data-engineering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BenSchr%2Fawesome-data-engineering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BenSchr%2Fawesome-data-engineering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BenSchr%2Fawesome-data-engineering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BenSchr","download_url":"https://codeload.github.com/BenSchr/awesome-data-engineering/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BenSchr%2Fawesome-data-engineering/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35945273,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-27T02:00:06.776Z","response_time":101,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2026-03-03T22:19:10.058Z","updated_at":"2026-07-27T09:00:24.749Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["📰 Blogs","📑 Handbooks \u0026 Guides","📚 Books","🧪 Testing","⚙️ DevOps \u0026 CI/CD","🔐 Data Privacy \u0026 Governance","🔄 Data Plattform Tools","🤖 Agentic Coding","🧱 Databricks","💸 FinOps \u0026 Cost Management","💡 Useful Code Snippets","📊 Reports"],"sub_categories":["🧠 Prompt Engineering \u0026 Memory Bank","📰 Blogs","📚 Repos","💡 Useful links \u0026 snippets"],"readme":"\u003c!-- Favicon for Awesome List --\u003e\n[![Awesome](https://awesome.re/badge-flat.svg)](https://awesome.re)\n\n# Awesome Data Engineering Resources\n\nA highly curated list of useful resources, repositories, blogs, and books for data engineering.\n\n## 📑 Table of Contents\n- [🤖 Agentic Coding](#-agentic-coding)\n  - [📚 Repos](#-repos)\n  - [🧠 Prompt Engineering \u0026 Memory Bank](#prompt-engineering--memory-bank)\n- [📚 Books](#-books)\n- [📰 Blogs](#-blogs)\n- [🔄 Data Plattform Tools](#-data-plattform-tools)\n- [🧱 Databricks](#-databricks)\n  - [📚 Repos](#-repos-1)\n  - [💡 Useful links \u0026 snippets](#-useful-links--snippets)\n  - [📰 Blogs](#-blogs-1)\n- [⚙️ DevOps \u0026 CI/CD](#-devops--cicd)\n- [📑 Handbooks \u0026 Guides](#-handbooks--guides)\n- [🔐 Data Privacy \u0026 Governance](#-data-privacy--governance)\n- [📊 Reports](#-reports)\n- [🧪 Testing](#-testing)\n- [💡 Useful Code Snippets](#-useful-code-snippets)\n- [💸 FinOps \u0026 Cost Management](#-finops--cost-management)\n\n\n## 🤖 Agentic Coding\nAgentic coding patterns, tools, and resources for building AI-driven and autonomous systems.\n\n### 📚 Repos\nOpen-source repositories and frameworks for agentic coding and AI applications.\n- [Dropped](https://github.com/pierceboggan/Dropped) - Open-source iOS project and resource for learning about agentic coding patterns.\n- [Generative AI for Beginners](https://github.com/microsoft/generative-ai-for-beginners) - A 21-lesson course by Microsoft teaching the fundamentals of building Generative AI applications, including hands-on code samples in Python and TypeScript.\n- [GitHub Copilot Vibe Coding Workshop](https://github.com/microsoft/github-copilot-vibe-coding-workshop/tree/main) - Self-paced workshop for building applications using GitHub Copilot Agent Mode, with multi-language samples and containerization.\n- [Graphiti](https://github.com/getzep/graphiti) - Framework for building real-time, temporally-aware knowledge graphs for AI agents, supporting dynamic data integration and hybrid search.\n- [MarkItDown](https://github.com/microsoft/markitdown) - Python tool for converting files and office documents to Markdown, designed for LLM and text analysis pipelines.\n\n- [Awesome Copilot](https://github.com/github/awesome-copilot) - Curated list of resources, tools, and projects related to GitHub Copilot and its ecosystem. A great starting point for exploring Copilot-powered development. 🧑‍💻🤖\n\n### 🧠 Prompt Engineering \u0026 Memory Bank\nGuides and tools for prompt engineering and memory management in LLM and agentic workflows.\n- [Cline Memory Bank Documentation](https://docs.cline.bot/prompting/cline-memory-bank#where-are-the-memory-bank-files-stored%3F) - Official documentation on where and how memory bank files are stored and managed in Cline.\n- [OpenAI Cookbook: GPT-4 Prompting Guide](https://cookbook.openai.com/examples/gpt4-1_prompting_guide) - Practical guide and examples for effective prompting with GPT-4, from the OpenAI Cookbook.\n\n## 📚 Books\nRecommended books on data engineering, architecture, and software best practices.\n- [Building Medallion Architectures](https://www.oreilly.com/library/view/building-medallion-architectures/9781098178826/) - Comprehensive guide to building medallion data architectures.\n- [Deciphering Data Architectures](https://www.oreilly.com/library/view/deciphering-data-architectures/9781098150754/) - Explains modern data architecture patterns and best practices.\n- [Designing Data-Intensive Applications](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/) - In-depth exploration of data systems, scalability, and reliability.\n- [Fundamentals of Data](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/) - Essential concepts and principles for working with data.\n- [The Pragmatic Programmer](https://www.oreilly.com/library/view/the-pragmatic-programmer/9780135956977/) - Classic book on software engineering best practices.\n- [The Staff Engineers Path](https://www.oreilly.com/library/view/the-staff-engineers/9781098118723/) - Book on the role, responsibilities, and career path of staff engineers in modern software organizations.\n\n## 📰 Blogs\nBlogs and publications covering data engineering, analytics, and technology trends.\n- [Confessions of a Data Guy](https://www.confessionsofadataguy.com/) - Blog on real-world data engineering.\n- [Data Engineering Blog (Simon Spaethi)](https://www.ssp.sh/) - Technical blog by Simon Spaethi.\n- [Data Engineering Weekly](https://www.dataengineeringweekly.com) - Weekly blog on data engineering trends and best practices.\n- [DLT Hub Blog](https://dlthub.com/blog) - Blog for DLT Hub, covering data loading and transformation topics.\n- [DuckDB Blog](https://duckdb.org/) - Official blog for DuckDB, an in-process SQL OLAP database management system.\n- [dbt Developer Hub Blog](https://docs.getdbt.com/blog) - Blog for dbt (data build tool) developers, featuring updates, tutorials, and best practices.\n- [Marvelous MLOps - Medium](https://medium.com/marvelous-mlops) - Medium publication focused on MLOps topics and best practices.\n- [MotherDuck Blog](https://motherduck.com/blog/) - Blog for MotherDuck, a DuckDB-based analytics platform.\n- [The GitHub Blog](https://github.blog/) - Official GitHub company blog.\n- [Thoughtworks Insights](https://www.thoughtworks.com/insights) - Thoughtworks' insights and technology trends blog.\n- [Visual Studio Code Blog](https://code.visualstudio.com/) - Official blog for Visual Studio Code, code editing, and development tips.\n- [Overclocking dbt: Discord's Custom Solution in Processing Petabytes of Data](https://discord.com/blog/overclocking-dbt-discords-custom-solution-in-processing-petabytes-of-data) - Deep dive into how Discord scaled dbt to process petabytes of data, including custom solutions for multi-developer workflows, performance, and CI/CD guardrails.\n\n## 🔄 Data Plattform Tools\nOpen-source tools and frameworks for building and managing data platforms.\n- [Apache DataFusion Comet](https://github.com/apache/datafusion-comet) - Query acceleration for Apache DataFusion.\n- [Apache Polaris](https://github.com/apache/polaris) - Snowflake Iceberg Catalog\n- [astronomer/astronomer-cosmos](https://github.com/astronomer/astronomer-cosmos) - Tools and integrations for running dbt projects in Apache Airflow.\n- [dbt-checkpoint](https://github.com/dbt-checkpoint/dbt-checkpoint) - Linting and checks for dbt projects.\n- [dbt-coves](https://github.com/datacoves/dbt-coves) - Workflow automation for dbt projects.\n- [dbt-labs/dbt-core](https://github.com/dbt-labs/dbt-core) - Core dbt framework for analytics engineering.\n- [dlt-hub/dlt](https://github.com/dlt-hub/dlt) - Data loading and transformation library for Python.\n- [duckdb/dbt-duckdb](https://github.com/duckdb/dbt-duckdb) - dbt adapter for DuckDB, enabling analytics engineering workflows on DuckDB databases.\n- [DuckLake](https://github.com/duckdb/ducklake) - Lakehouse implementation for DuckDB.\n- [Koheesio](https://github.com/Nike-Inc/koheesio) - Orchestration framework for building data pipelines.\n- [Lakehouse Engine](https://github.com/adidas/lakehouse-engine) - Lakehouse engine for scalable analytics by Adidas.\n- [VSCode dbt Power User](https://github.com/AltimateAI/vscode-dbt-power-user/blob/master/package.json) - VSCode extension for enhanced dbt development.\n\n## 🧱 Databricks\nResources, tools, and blogs for working with Databricks and the modern data stack.\n\n### 📚 Repos\nOfficial and community Databricks-related repositories and tools.\n- [databricks/cli](https://github.com/databricks/cli) - Official Databricks CLI for automation and scripting.\n- [databricks/databricks-vscode](https://github.com/databricks/databricks-vscode) - Visual Studio Code extension for Databricks development.\n- [databricks/dbt-databricks](https://github.com/databricks/dbt-databricks) - dbt adapter for Databricks, enabling analytics engineering workflows on Databricks.\n- [databrickslabs/discoverx](https://github.com/databrickslabs/discoverx) - Data discovery and cataloging tool for Databricks.\n- [databrickslabs/dqx](https://github.com/databrickslabs/dqx) - Data quality framework for Databricks and Spark.\n- [databricks/terraform-provider-databricks](https://github.com/databricks/terraform-provider-databricks) - Terraform provider for Databricks infrastructure automation.\n- [Terraform Databricks SRA](https://github.com/databricks/terraform-databricks-sra) - Terraform modules and examples for implementing Databricks Security Reference Architecture.\n- [UnityCatalog](https://github.com/unitycatalog/unitycatalog) - Open-source implementation of Unity Catalog for data governance.\n\n### 💡 Useful links \u0026 snippets\nHelpful links, code snippets, and resources for Databricks users.\n- [DBSQL SME Resources](https://github.com/CodyAustinDavis/dbsql_sme/tree/main) - Resources and tools for Databricks SQL subject matter experts.\n- [Databricks Playground](https://github.com/alexott/databricks-playground) - A collection of Databricks notebooks and resources for experimenting and learning.\n- [Retrying dbt Runs in Databricks Workflows](https://community.databricks.com/t5/technical-blog/retrying-dbt-runs-in-databricks-workflows/ba-p/65766) - Databricks blog post on strategies for retrying dbt runs in workflows.\n\n### 📰 Blogs\nBlogs and publications focused on Databricks and its ecosystem.\n- [Databricks AI - Medium](https://medium.com/@AI-on-Databricks) - Medium publication for AI topics on Databricks.\n- [Databricks Blog](https://www.databricks.com) - Official Databricks company blog.\n- [Databricks Community Blog](https://community.databricks.com/t5/technical-blog/bg-p/technical-blog) - Technical articles and updates from the Databricks community.\n- [Databricks DBSQL SME - Medium](https://medium.com/dbsql-sme-engineering) - Medium publication for Databricks SQL SME engineering topics.\n- [Databricks Platform SME - Medium](https://medium.com/databricks-platform-sme) - Medium publication for Databricks Platform subject matter experts.\n- [Databricks SQL SME on Medium](https://medium.com/@databricks_sql_sme) - Medium publication for Databricks SQL SME.\n- [Databricks UC SME - Medium](https://medium.com/databricks-unity-catalog-sme) - Medium publication focused on Databricks Unity Catalog subject matter expertise.\n\n## ⚙️ DevOps \u0026 CI/CD\nDevOps tools, CI/CD automation, and infrastructure resources for data engineering.\n- [Commitizen CLI](https://github.com/commitizen/cz-cli) - Tool for creating conventional commit messages and automating releases.\n- [Copier](https://github.com/copier-org/copier) - Project templating tool for generating and maintaining codebases.\n- [Inspect Docker Images](https://github.com/pythonmonty/inspect-docker-images) - Tool for inspecting Docker images for vulnerabilities and metadata.\n- [VSCode with Podman Desktop](https://podman-desktop.io/blog/2025/05/05/vs-code-with-podman-desktop) - Guide to integrating VSCode with Podman Desktop for container development.\n\n## 📑 Handbooks \u0026 Guides\nComprehensive handbooks, guides, and documentation frameworks for data and engineering teams.\n- [Diátaxis](https://diataxis.fr/) - Framework for organizing technical documentation by user needs, focusing on tutorials, how-to guides, reference, and explanation.\n- [GitLab Enterprise Data Handbook](https://handbook.gitlab.com/handbook/enterprise-data/) - GitLab's official handbook for enterprise data management and governance.\n- [Modern Data Engineering Playbook](https://www.thoughtworks.com/insights/e-books/modern-data-engineering-playbook) - Comprehensive guide on modern data engineering practices and principles.\n- [Kimball Dimensional Modeling Techniques](https://www.kimballgroup.com/wp-content/uploads/2013/08/2013.09-Kimball-Dimensional-Modeling-Techniques11.pdf) - 📊 Classic reference PDF from the Kimball Group summarizing dimensional modeling techniques for data warehousing and business intelligence projects.\n- [Python Patterns Guide](https://python-patterns.guide/) - 🐍 Comprehensive guide to Python programming patterns, including design patterns, idioms, and best practices for writing clean, maintainable Python code.\n\n## 🔐 Data Privacy \u0026 Governance\nResources and tools for data privacy, security, and synthetic data generation.\n- [DataContract CLI](https://github.com/datacontract/datacontract-cli) - CLI tool for managing data contracts in data engineering workflows.\n- [DataHub Project](https://github.com/datahub-project/datahub) - Metadata platform for the modern data stack.\n- [Presidio](https://github.com/microsoft/presidio) - Open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) in text, images, and structured data.\n- [SDV: Synthetic Data Vault](https://github.com/sdv-dev/SDV) - Python library for generating synthetic tabular data using machine learning, with tools for evaluation, anonymization, and quality reporting.\n\n## 📊 Reports\nIndustry reports, playbooks, and technology trend analyses for data engineering.\n- [Looking Glass 2025](https://www.thoughtworks.com/en-de/insights/looking-glass) - Thoughtworks' long-term technology trend report exploring 90+ trends and their business impact, with strategic recommendations.\n\n## 🧪 Testing\nTesting tools, frameworks, and best practices for data and software engineering.\n- [Inline Snapshot](https://github.com/15r10nk/inline-snapshot) - Tool for inline snapshot testing in Python.\n- [Practical Test Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html) - Martin Fowler's article on the test pyramid and testing strategies.\n- [Pytest Basics](https://github.com/The-Compiler/pytest-basics) - Examples and explanations for getting started with pytest.\n- [Test Desiderata](https://testdesiderata.com/) - Philosophical and practical guidance for software testing.\n\n## 💡 Useful Code Snippets\nHandy code snippets and example repositories for data engineering tasks.\n- [Building Medallion Architectures Book Repo](https://github.com/pietheinstrengholt/building-medallion-architectures-book) - The Repo to the book with useful code snippets.\n- [The Hitchhiker's Guide to dbt](https://github.com/jeremyyeo/the-hitchhikers-guide-to-dbt) - A comprehensive guide and resource collection for working with dbt (data build tool).\n\n## 💸 FinOps \u0026 Cost Management\nOpen-source tools and resources for cloud financial operations, cost management, and FinOps best practices.\n- [FinOps Toolkit](https://github.com/microsoft/finops-toolkit) - Microsoft open-source toolkit for automating and extending FinOps capabilities in the Microsoft Cloud, including starter kits, automation scripts, and best practices.\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/benschr%2Fawesome-data-engineering/projects"}