{"id":24881819,"url":"https://github.com/realdatadriven/etlx","last_synced_at":"2026-04-30T22:01:19.341Z","repository":{"id":265916038,"uuid":"896859349","full_name":"realdatadriven/etlx","owner":"realdatadriven","description":"ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.","archived":false,"fork":false,"pushed_at":"2026-04-22T13:39:13.000Z","size":13814,"stargazers_count":41,"open_issues_count":2,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-04-22T15:27:46.221Z","etag":null,"topics":["data-engineering","data-lake","data-lakehouse","data-quality","data-quality-checks","data-quality-monitoring","data-science","duckdb","elt","elt-pipeline","etl","etl-elt-pipelines","etl-pipeline","object-storage","relational-databases","report","report-automation","s3","s3-storage"],"latest_commit_sha":null,"homepage":"https://realdatadriven.github.io/etlxdocs/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/realdatadriven.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-01T13:35:38.000Z","updated_at":"2026-04-22T13:39:18.000Z","dependencies_parsed_at":"2026-01-17T21:05:14.307Z","dependency_job_id":"cbe02db5-b08c-40a1-be4f-c3c2ba66f69c","html_url":"https://github.com/realdatadriven/etlx","commit_stats":null,"previous_names":["realdatadriven/etlx"],"tags_count":143,"template":false,"template_full_name":null,"purl":"pkg:github/realdatadriven/etlx","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/realdatadriven%2Fetlx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/realdatadriven%2Fetlx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/realdatadriven%2Fetlx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/realdatadriven%2Fetlx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/realdatadriven","download_url":"https://codeload.github.com/realdatadriven/etlx/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/realdatadriven%2Fetlx/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32478162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"ssl_error","status_checked_at":"2026-04-30T13:12:06.837Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","data-lake","data-lakehouse","data-quality","data-quality-checks","data-quality-monitoring","data-science","duckdb","elt","elt-pipeline","etl","etl-elt-pipelines","etl-pipeline","object-storage","relational-databases","report","report-automation","s3","s3-storage"],"created_at":"2025-02-01T12:13:44.504Z","updated_at":"2026-04-30T22:01:19.327Z","avatar_url":"https://github.com/realdatadriven.png","language":"Go","funding_links":[],"categories":["Tools Powered by DuckDB","Go"],"sub_categories":[],"readme":"# ETLX\n\n**ETLX** is an **open-source, SQL-first data workflow engine** and an **evolving specification** for building **self-documenting data pipelines**.\n\nPipelines are defined using **structured Markdown**, which serves simultaneously as:\n\n* executable configuration\n* human-readable documentation\n* governance and audit artifacts source\n\nETLX pipelines can be **executed**, **versioned**, and **rendered as documentation** — making the workflow itself the source of truth.\n\nIt combines:\n\n* **Declarative pipelines**\n* **Executable documentation**\n* **Multi-engine SQL execution**\n* **Built-in observability**\n\nPowered by [**DuckDB**](https://duckdb.org), but **not locked to it**.\n\n---\n\n## ✨ What Makes ETLX Different?\n\n- ✔ Pipelines are written in **Markdown + YAML + SQL**\n- ✔ The pipeline **is the documentation**\n- ✔ Runs on **DuckDB, PostgreSQL, SQLite, MySQL, SQL Server, ODBC**\n- ✔ One specification for **ETL / ELT / Reverse ETL, data quality, report generation and automation, scripts execution, ...**\n- ✔ Fully **auditable \u0026 reproducible** by design\n- ✔ Available as a **CLI and embeddable Go library**\n\n\u003e ETLX is not just a runtime — it also meant to be a **specification for declarative data workflows**, where **all logic is explicit, inspectable, and versionable**.\n\n---\n\n## 🚀 Quick Example - pipeline.md\n\n````md\n# INPUTS\n```yaml\nname: INPUTS\ndescription: this defines a ETL / ELT / Reverse ETL block where every level two block with proper metadata (yaml) is treated as a step in the workflow\nruns_as: ETL # the runs_as defines how the block shoud be treated\nactive: true # active if missing the is consider active, if false this block and all its child are ignored\n```\n\n## SALES\n```yaml\nname: SALES\ntable: sales\nload_conn: \"duckdb:\" # Opens a DuckDB in-memory instance\nload_before:\n    - ATTACH 'postgres:@PG_CON' AS SRC (TYPE POSTGRES) #  Ataches data source as SRC in this case postgres OLTP DB, but could be any DBMS with a connecter / scanner\n    - ATTACH 'ducklake:@DL_CON' AS TGT (DATA_PATH 's3://my-lakehouse_bucket...', ENCRYPTED) # Attaches target DB, TGT in this case a ducklake, prefirable, but again could be any DMBMS\nload_validation: # Basic validation, normally used to check updates, avoid data duplication and unnessessary extractions (for more advanced conditional check use \u003cstep\u003e_condition)\n  - type: throw_if_empty # The processes will fail and be logged as such if the query returns empty\n    sql: FROM SRC.\u003ctable\u003e WHERE date_field = '{YYYY-MM-DD}' LIMIT 10 # The query that is executed\n    msg: \"The given date ({YYYY-MM-DD}) is not avaliable in the source!\" # The message to be logged\n    active: true\n  - type: throw_if_not_empty # Fails if query return any row\n    sql: FROM TGT.\u003ctable\u003e WHERE date_field = '{YYYY-MM-DD}' LIMIT 10\n    msg: \"The date {YYYY-MM-DD} is already imported in the target, check to avoid duplications, or clean this period first!\"\n    active: true\nload_sql: load_sales_data # Extracts from source and load on target in a sigle query thanks to duckdb capability of attaching different DBMS\nload_on_err_match_patt: '(?i)table.+with.+name.+(\\w+).+does.+not.+exist' # In case the load data query throws an error because the table is not created yet, in runs the sql in load_on_err_match_sql\nload_on_err_match_sql: create_sales_table_instead # this sql only runs in case the load data fails and the error matchs the pattern in load_on_err_match_patt\nload_after:\n    - DETACH SRC # detaches the source DB\n    - DETACH TGT # detaches the target DB\n```\n\n\u003c!-- INSERT --\u003e\n```sql load_sales_data\nINSERT INTO TGT.\u003ctable\u003e BY NAME\nSELECT *\nFROM SRC.\u003ctable\u003e\nWHERE date_field = '{YYYY-MM-DD}'\n```\n\n\u003c!-- CREATE --\u003e\n```sql\n-- create_sales_table_instead\nCREATE TABLE TGT.\u003ctable\u003e AS\nSELECT *\nFROM SRC.\u003ctable\u003e\n```\n...\n````\n\u003e @PG_CON, @DL_CON are connection strings defined in the environment or in the `.env` file.\n\nRun it:\n\n```bash\netlx --config pipeline.md\n```\n\n---\n\n## 📘 Documentation\n\n👉 **Full documentation, concepts, and examples**\n[https://realdatadriven.github.io/etlxdocs](https://realdatadriven.github.io/etlxdocs)\n\nIncludes:\n\n* Quickstart\n* Core concepts\n* Specification reference\n* Advanced examples\n* Go API usage\n* Logging \u0026 observability\n* Multi-engine execution\n\n---\n\n## 🧠 Philosophy\n\nETLX embraces:\n\n* **SQL as the transformation language**\n* **Markdown as the contract**\n* **Metadata as a first-class citizen**\n* **Transparency over magic**\n\nNo hidden state.\nNo proprietary DSL.\nNo opaque execution model.\n\n---\n\n## 🤝 Contributing\n\nETLX is community-driven.\n\n👉 Contribution guide:\n[https://realdatadriven.github.io/etlxdocs/docs/contributing/](https://realdatadriven.github.io/etlxdocs/docs/contributing/)\n\n---\n\n## 📜 License\n\nApache License 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frealdatadriven%2Fetlx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frealdatadriven%2Fetlx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frealdatadriven%2Fetlx/lists"}