{"id":50779935,"url":"https://github.com/basvdberg/data-solution-2026","last_synced_at":"2026-06-12T02:30:34.620Z","repository":{"id":360443067,"uuid":"1236781998","full_name":"basvdberg/data-solution-2026","owner":"basvdberg","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-11T11:33:25.000Z","size":7423,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-11T12:17:25.846Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/basvdberg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-12T15:09:32.000Z","updated_at":"2026-06-11T11:33:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/basvdberg/data-solution-2026","commit_stats":null,"previous_names":["basvdberg/data-solution-2026"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/basvdberg/data-solution-2026","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basvdberg%2Fdata-solution-2026","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basvdberg%2Fdata-solution-2026/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basvdberg%2Fdata-solution-2026/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basvdberg%2Fdata-solution-2026/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/basvdberg","download_url":"https://codeload.github.com/basvdberg/data-solution-2026/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basvdberg%2Fdata-solution-2026/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34226629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-12T02:30:33.925Z","updated_at":"2026-06-12T02:30:34.612Z","avatar_url":"https://github.com/basvdberg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Solution 2026\n\n## Table of contents\n\n\u003c!-- markdown-toc:start --\u003e\n- [Purpose](#purpose)\n- [Proof of concept](#proof-of-concept)\n- [📈 Progress](#progress)\n- [Documentation](#documentation)\n- [Design patterns changed](#design-patterns-changed)\n  - [Created](#created)\n  - [Modified (pre-existing)](#modified-pre-existing)\n\u003c!-- markdown-toc:end --\u003e\n\n## Purpose\n\nProof of concept for the GenAI way of working as described in [Data Engineering 2026](https://github.com/basvdberg/data-engineering-2026).\n\nIntent is specified with [data-engineering-design-patterns](https://github.com/basvdberg/data-engineering-design-patterns) as building blocks. Implementation code and orchestration are generated and refined with Cursor. Data transformations are specified with meta data. \n\n## Proof of concept\n\nA **staging layer** for daily mean temperature across the Netherlands.\n  \n| Aspect | Choice |\n|--------|--------|\n| Source | [Open-Meteo](https://open-meteo.com/) — public API, no key, continuously updated models |\n| Metadata | [DSA](https://github.com/data-solution-automation-engine/data-warehouse-automation-metadata-schema) connections, data objects, and mappings |\n| Extraction | GenAI-generated Python poller and extractor (`code/extractor_and_poller/`) |\n| Runtime metadata | PostgreSQL table `poller` (markers and poll history) |\n| Orchestration | Apache Airflow and Apache Kafka (`code/` DAGs; hosted on local NAS for this PoC) |\n\n## 📈 Progress\n\n**June 1, 2026** — follow-up to the May 19 LinkedIn post on data engineering with Gen-AI.\n\n**Done**\n\n- Infrastructure on the local server: Airflow and Kafka deployed and tuned via GenAI over SSH.\n- Design patterns created and modified to specify intent. See [design patterns changed](#design-patterns-changed).\n- After discovering that the KNMI API was actually quite old and not providing any recent data updates, we switched to a new public data API, which was relatively easy because of the new way of working.\n- Open-Meteo extractor and poller implemented.\n\n**Next**\n\n- Event-based orchestration for source-to-staging: poller publishes change events on Kafka; Airflow reacts and triggers extract (see [event-based orchestration plan](doc/design/event-based-orchestration-plan.md)).\n\n**Lessons so far** ([full notes](lessons-learned-part1.md)).\n\n## Documentation\n\n| Topic | Document |\n|-------|----------|\n| Run locally | [Getting started](getting-started.md) |\n| Generated runtime (DAGs) | [code/](code/readme.md) |\n| Docker (Airflow, Kafka) | [Infrastructure](infra/readme.md) |\n| Architecture and flow | [Architecture](doc/design/architecture.md) |\n| DSA layout in Git | [Meta data design](doc/design/meta-data-design.md) |\n| Event orchestration | [Event-based orchestration plan](doc/design/event-based-orchestration-plan.md) |\n| Kafka topics | [Kafka topic naming](doc/design/kafka-topic-naming.md) |\n| Implementation checklist | [Implementation plan](doc/implementation-plan.md) |\n| Observations | [Lessons learned](lessons-learned-part1.md) |\n| LinkedIn (lessons learned) | [LinkedIn post (part 3)](docs/linkedin-post-part3.md) |\n\n## Design patterns changed\n\nSince the [May 19 LinkedIn post](https://github.com/basvdberg/data-engineering-2026), the [data-engineering-design-patterns](https://github.com/basvdberg/data-engineering-design-patterns) catalogue grew to support this PoC. Patterns live under `design-patterns/data-engineering/` and `design-patterns/generic/` (reorganized 1 June 2026).\n\n### Created\n\n| Pattern | Summary |\n|--------|---------|\n| [Separate what and how](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/generic/separate-what-and-how.md) | Declarative specification (*what*) vs imperative implementation (*how*); DSA in Git, tools and code for execution. |\n| [Data extractor](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/data-engineering/data-extractor.md) | Reads a source via protocol and format adapter; writes the target with integrity and observability. |\n| [Data object poller](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/data-engineering/data-object-poller.md) | Detects marker changes on a schedule; publishes events without full payload reads. |\n| [Data object](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/data-engineering/data-object.md) | Atomic unit: identity, location, schema, classification (table, file, API dataset). |\n| [Data object container](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/data-engineering/data-object-container.md) | Hierarchical scope (database, folder, layer) with inherited properties; groups objects, no payload. |\n| [Data solution layer](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/data-engineering/data-solution-layer.md) | Staging → raw → integrated → presentation; unidirectional flow. |\n| [Simplicity](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/generic/simplicity.md) | Prefer the simpler correct design; keep the system boundary small. |\n| [Functional decomposition](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/generic/functional-decomposition.md) | Decompose by responsibility; express variation in configuration. |\n| [Data object tree](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/data-engineering/data-object-tree.md) | Parent–child hierarchy and discovery of objects in a solution. |\n| [Data object tree property inheritance](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/data-engineering/data-object-tree-property-inheritance.md) | Properties on tree nodes; child overrides parent, else inherit from ancestor. |\n\n**Removed:** *Object property tree* — split into **Data object tree** and **Data object tree property inheritance**. [Prefer simple decomposition](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/generic/prefer-simple-decomposition.md) is now a stub pointing to **Simplicity** and **Functional decomposition**.\n\n### Modified (pre-existing)\n\n| Pattern | Summary |\n|--------|---------|\n| [Data solution](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/data-engineering/data-solution.md) | Linked to new data-engineering patterns and PoC vocabulary. |\n| [Event-based orchestration](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/data-engineering/event-based-orchestration.md) | Aligned with poller events, extractor tasks, and Kafka/Airflow flow used here. |\n| [Historic bitemporal table](https://github.com/basvdberg/data-engineering-design-patterns/blob/main/design-patterns/data-engineering/historic-bitemporal-table.md) | Cross-links and structure refresh alongside the catalogue. |\n\n## Project structure\n\n\u003c!-- markdown-project-structure:start --\u003e\n- [Data Solution 2026](readme.md)\n  - Code\n    - Airflow\n      - Dags\n      - Plugins\n    - Extractor_And_Poller\n      - Common\n      - Controller\n      - Extract\n      - Openmeteo\n        - Extractor\n        - Poller\n      - Poller\n      - Tests\n    - Postgres\n      - Migrations\n  - Connection\n  - Data\n    - Staging\n      - Openmeteo\n        - Daily_Temperature\n  - Data Object\n    - Source\n      - Openmeteo\n    - Staging\n      - Openmeteo\n  - Data Object Mapping\n    - Staging\n      - Openmeteo\n  - Doc\n    - Data Solution\n      - Data Object Mapping\n    - Design\n      - [Architecture](doc/design/architecture.md)\n      - [CI/CD workflow (main only + server pull deploy)](doc/design/ci-cd.md)\n      - [Event-based orchestration plan (single data object)](doc/design/event-based-orchestration-plan.md)\n      - [Kafka topic naming](doc/design/kafka-topic-naming.md)\n      - [Meta data design](doc/design/meta-data-design.md)\n    - Operation\n      - Incident\n        - [INC-001 — NAS non-interactive SSH environment](doc/operation/incident/inc-001-nas-ssh-environment.md)\n        - [INC-002 — Airflow standalone infra instability](doc/operation/incident/inc-002-airflow-infra-stability.md)\n        - [INC-003 — Agent rediscovery and false-done verification](doc/operation/incident/inc-003-agent-process-gaps.md)\n        - [INC-004 — Airflow PYTHONPATH drift (dag_run_guard import)](doc/operation/incident/inc-004-airflow-pythonpath-drift.md)\n        - [INC-\u003cNNN\u003e — \u003cshort title\u003e](doc/operation/incident/incident-template.md)\n      - [Issue categories](doc/operation/issue-category.md)\n    - [Implementation plan (Open-Meteo → event orchestration)](doc/implementation-plan.md)\n  - Docs\n    - [LinkedIn post (part 3)](docs/linkedin-post-part3.md)\n    - [Linkedin Post Part3V2](docs/linkedin-post-part3v2.md)\n  - Infra\n    - Airflow\n      - Dags\n    - Kafka\n    - Postgres\n  - Release\n    - 2026\n      - 06\n        - 02\n          - V2026.06.02.1\n            - [Notes](release/2026/06/02/v2026.06.02.1/notes.md)\n          - V2026.06.02.2\n            - [Release v2026.06.02.2](release/2026/06/02/v2026.06.02.2/notes.md)\n        - 03\n          - V2026.06.03.1\n            - [Release v2026.06.03.1](release/2026/06/03/v2026.06.03.1/notes.md)\n          - V2026.06.03.2\n            - [Release v2026.06.03.2](release/2026/06/03/v2026.06.03.2/notes.md)\n          - V2026.06.03.3\n            - [Release v2026.06.03.3](release/2026/06/03/v2026.06.03.3/notes.md)\n          - V2026.06.03.4\n            - [Release v2026.06.03.4](release/2026/06/03/v2026.06.03.4/notes.md)\n            - [Retrospective](release/2026/06/03/v2026.06.03.4/retrospective.md)\n        - 04\n          - V2026.06.04.1\n            - [Notes](release/2026/06/04/v2026.06.04.1/notes.md)\n        - 05\n          - V2026.06.05.6\n            - [Notes](release/2026/06/05/v2026.06.05.6/notes.md)\n            - [Retrospective](release/2026/06/05/v2026.06.05.6/retrospective.md)\n        - 08\n          - V2026.06.08.1\n            - [Notes](release/2026/06/08/v2026.06.08.1/notes.md)\n            - [Retrospective](release/2026/06/08/v2026.06.08.1/retrospective.md)\n          - V2026.06.08.2\n            - [Notes](release/2026/06/08/v2026.06.08.2/notes.md)\n            - [Retrospective](release/2026/06/08/v2026.06.08.2/retrospective.md)\n        - 09\n          - V2026.06.09.1\n            - [Notes](release/2026/06/09/v2026.06.09.1/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.1/retrospective.md)\n          - V2026.06.09.10\n            - [Notes](release/2026/06/09/v2026.06.09.10/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.10/retrospective.md)\n          - V2026.06.09.11\n            - [Notes](release/2026/06/09/v2026.06.09.11/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.11/retrospective.md)\n          - V2026.06.09.12\n            - [Notes](release/2026/06/09/v2026.06.09.12/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.12/retrospective.md)\n          - V2026.06.09.13\n            - [Notes](release/2026/06/09/v2026.06.09.13/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.13/retrospective.md)\n          - V2026.06.09.14\n            - [Notes](release/2026/06/09/v2026.06.09.14/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.14/retrospective.md)\n          - V2026.06.09.15\n            - [Notes](release/2026/06/09/v2026.06.09.15/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.15/retrospective.md)\n          - V2026.06.09.16\n            - [Notes](release/2026/06/09/v2026.06.09.16/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.16/retrospective.md)\n          - V2026.06.09.17\n            - [Notes](release/2026/06/09/v2026.06.09.17/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.17/retrospective.md)\n          - V2026.06.09.2\n            - [Notes](release/2026/06/09/v2026.06.09.2/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.2/retrospective.md)\n          - V2026.06.09.3\n            - [Notes](release/2026/06/09/v2026.06.09.3/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.3/retrospective.md)\n          - V2026.06.09.4\n            - [Notes](release/2026/06/09/v2026.06.09.4/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.4/retrospective.md)\n          - V2026.06.09.5\n            - [Notes](release/2026/06/09/v2026.06.09.5/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.5/retrospective.md)\n          - V2026.06.09.6\n            - [Notes](release/2026/06/09/v2026.06.09.6/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.6/retrospective.md)\n          - V2026.06.09.7\n            - [Notes](release/2026/06/09/v2026.06.09.7/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.7/retrospective.md)\n          - V2026.06.09.8\n            - [Notes](release/2026/06/09/v2026.06.09.8/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.8/retrospective.md)\n          - V2026.06.09.9\n            - [Notes](release/2026/06/09/v2026.06.09.9/notes.md)\n            - [Retrospective](release/2026/06/09/v2026.06.09.9/retrospective.md)\n        - 11\n          - V2026.06.11.1\n            - [Notes](release/2026/06/11/v2026.06.11.1/notes.md)\n            - [Retrospective](release/2026/06/11/v2026.06.11.1/retrospective.md)\n          - V2026.06.11.2\n            - [Notes](release/2026/06/11/v2026.06.11.2/notes.md)\n            - [Retrospective](release/2026/06/11/v2026.06.11.2/retrospective.md)\n          - V2026.06.11.3\n            - [Notes](release/2026/06/11/v2026.06.11.3/notes.md)\n            - [Retrospective](release/2026/06/11/v2026.06.11.3/retrospective.md)\n          - V2026.06.11.4\n            - [Notes](release/2026/06/11/v2026.06.11.4/notes.md)\n            - [Retrospective](release/2026/06/11/v2026.06.11.4/retrospective.md)\n          - V2026.06.11.5\n            - [Notes](release/2026/06/11/v2026.06.11.5/notes.md)\n            - [Retrospective](release/2026/06/11/v2026.06.11.5/retrospective.md)\n          - V2026.06.11.6\n            - [Notes](release/2026/06/11/v2026.06.11.6/notes.md)\n            - [Retrospective](release/2026/06/11/v2026.06.11.6/retrospective.md)\n          - V2026.06.11.7\n            - [Notes](release/2026/06/11/v2026.06.11.7/notes.md)\n            - [Retrospective](release/2026/06/11/v2026.06.11.7/retrospective.md)\n    - [Release \u003cversion\u003e](release/release-notes-template.md)\n    - [Retrospective — \u003cversion\u003e](release/retrospective-template.md)\n  - Setting\n  - Template\n  - [Getting started](getting-started.md)\n  - [Lessons learned](lessons-learned-part1.md)\n  - [Lessons learned (part 2)](lessons-learned-part2.md)\n- Related repositories\n  - [Data Engineering 2026](https://github.com/basvdberg/data-engineering-2026) — Course and learning materials\n  - [Data Engineering Design Patterns](https://github.com/basvdberg/data-engineering-design-patterns) — Design pattern catalogue\n\u003c!-- markdown-project-structure:end --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbasvdberg%2Fdata-solution-2026","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbasvdberg%2Fdata-solution-2026","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbasvdberg%2Fdata-solution-2026/lists"}