{"id":50779934,"url":"https://github.com/basvdberg/data-engineering-2026","last_synced_at":"2026-06-12T02:30:34.442Z","repository":{"id":359354480,"uuid":"1245604038","full_name":"basvdberg/data-engineering-2026","owner":"basvdberg","description":"GenAI way of working for data engineering in 2026","archived":false,"fork":false,"pushed_at":"2026-05-21T14:14:09.000Z","size":2001,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-21T22:01:45.261Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/basvdberg.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-21T11:30:32.000Z","updated_at":"2026-05-21T14:22:14.000Z","dependencies_parsed_at":"2026-05-21T22:02:20.760Z","dependency_job_id":null,"html_url":"https://github.com/basvdberg/data-engineering-2026","commit_stats":null,"previous_names":["basvdberg/data-engineering-2026"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/basvdberg/data-engineering-2026","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basvdberg%2Fdata-engineering-2026","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basvdberg%2Fdata-engineering-2026/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basvdberg%2Fdata-engineering-2026/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basvdberg%2Fdata-engineering-2026/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/basvdberg","download_url":"https://codeload.github.com/basvdberg/data-engineering-2026/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basvdberg%2Fdata-engineering-2026/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34226629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-12T02:30:33.830Z","updated_at":"2026-06-12T02:30:34.434Z","avatar_url":"https://github.com/basvdberg.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Engineering 2026\n\n## Table of contents\n\n\u003c!-- markdown-toc:start --\u003e\n- [Purpose](#purpose)\n- [Architecture](#architecture)\n- [The role of documentation](#the-role-of-documentation)\n- [New CI/CD process](#new-cicd-process)\n- [Specification of intent](#specification-of-intent)\n  - [DSA for defining data transformations](#dsa-for-defining-data-transformations)\n  - [Design patterns for specifying intent](#design-patterns-for-specifying-intent)\n- [Summary](#summary)\n- [Proof of concept](#proof-of-concept)\n\u003c!-- markdown-toc:end --\u003e\n\n## Purpose\n\nThis project explores how GenAI can improve the data engineering process. GenAI offers a major gain in efficiency and can raise the overall quality of your data solution.\n\nLet's start by describing a new way of working using GenAI. In my case I use the Cursor 3.4 IDE.\n\n## Architecture\n\nThe data engineer steers a three-layer stack. At the top, **specification of intent** captures *what* the solution must do — sourced from generic and data-engineering design patterns. Generative AI translates that declarative intent into **code and configuration**, which runs on **standard tools** in production.\n\n![Gen AI and data engineering architecture](docs/genai-data-engineering-architecture.svg)\n\n**Interactive versions**\n\n| Where | Link |\n|-------|------|\n| Web (shareable) | [genai-data-engineering-architecture.html](docs/genai-data-engineering-architecture.html) — works in any browser; use on [GitHub Pages](https://basvdberg.github.io/data-engineering-2026/genai-data-engineering-architecture.html) for LinkedIn and social sharing |\n| Cursor IDE | [genai-data-engineering-architecture.canvas.tsx](canvases/genai-data-engineering-architecture.canvas.tsx) |\n\nThe web version includes collapsible layers, clickable source boxes, and a light/dark theme. Enable **GitHub Pages** (Settings → Pages → Build from branch `main`, folder `/docs`) to publish the shareable URL.\n\n## The role of documentation\n\nDocumentation captures design decisions and the knowledge needed to understand the data solution and the data itself. That knowledge is exchanged with other data engineers and with users of the solution.\n\nTraditionally, producing and keeping that material current was labor-intensive work that teams postponed while they focused on getting the data solution running. Architects usually documented the starting architecture well, but detailed design decisions and implementation choices were deferred; under delivery pressure, documentation stayed incomplete or out of date.\n\nGenerative AI shifts that balance in two ways:\n\n1. **Easier to produce** — AI can draft and refresh documentation quickly, so maintenance is no longer the bottleneck it was.\n2. **Essential for steering AI** — Up-to-date docs become context for the next change; without them, generated code drifts from intent.\n\n**Before:** Documentation lagged behind delivery: strong at kick-off, thin on detail, and often never finished.\n\n**After:** Start every change by updating documentation, then implement. AI drafts the material quickly so you can review intent before code lands. Documentation and release notes can stay current on each pull request, failed approaches can be recorded for the next session, and prior docs feed the next AI session—documentation becomes part of small CI/CD iterations instead of a separate phase.\n\n## New CI/CD process\n\nThe CI/CD cycle for a data engineer shifts in two places. **Design** still comes first. What used to be hand-written code becomes generated code, and documentation moves from an afterthought to the input that drives generation.\n\n![Old vs new CI/CD workflow for data engineers](docs/ci-cd-old-vs-new.png)\n\n## Specification of intent\n\nSo documentation is the gasoline that fires the GenAI motor. One of the pitfalls when using GenAI is ambiguity or under-specification of your intent: when you vaguely describe what needs to be done, you give GenAI the freedom to generate many different implementations. What GenAI needs is a well-defined specification of your intent in a declarative language—you describe *what* needs to be done, and the code generator produces the code in any desired software stack.\n\n![One declarative specification driving code generation across multiple stacks](docs/open-standard-drives-generation.png)\n\n### DSA for defining data transformations\n\nFortunately there is a well-defined open-source standard to describe source-to-target mappings, data objects, and connections: the [Data Solution Automation (DSA) metadata schema](https://github.com/data-solution-automation-engine/data-warehouse-automation-metadata-schema). The standard expresses data transformations in readable JSON files. [Agnostic Data Labs (ADL)](https://docs.agnosticdatalabs.com/docs/) is a free companion tool to visualize those files. This tool can also generate code using templates, but one of my questions is whether to do this using such a tool or via genAI. \n\n### Design patterns for specifying intent\n\nIn [*Data Engine Thinking*](https://dataenginethinking.com/), Roelant Vos and Dirk Lerner describe generic design patterns for building a data solution. Those patterns are a strong way to specify intent: they are technology-agnostic and encode industry best practices. I have also been working on my own collection of design patterns here [data-engineering-design-patterns](https://github.com/basvdberg/data-engineering-design-patterns).\n\n## Summary\n\nGenAI demands well-written documentation and a clear, unambiguous specification of intent (*what* needs to be done). Documentation becomes a first-class step in the CI/CD process. Open standards expressed in a declarative language, together with explicit design patterns, are a natural fit for GenAI-driven generation.\n\n## Proof of concept\n\nThe [data-solution-2026](https://github.com/basvdberg/data-solution-2026) repository is the proof of concept that applies this way of working to a concrete staging-layer ingestion pipeline.\n\n## Project structure\n\n\u003c!-- markdown-project-structure:start --\u003e\n- [Data Engineering 2026](readme.md)\n  - Canvases\n  - Docs\n    - [LinkedIn post](docs/linkedin-post.md)\n- Related repositories\n  - [Data Engineering Design Patterns](https://github.com/basvdberg/data-engineering-design-patterns) — Design pattern catalogue\n  - [Data Solution 2026](https://github.com/basvdberg/data-solution-2026) — Data solution proof of concept\n\u003c!-- markdown-project-structure:end --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbasvdberg%2Fdata-engineering-2026","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbasvdberg%2Fdata-engineering-2026","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbasvdberg%2Fdata-engineering-2026/lists"}