{"id":21514864,"url":"https://github.com/getindata/dp-framework","last_synced_at":"2026-03-19T20:41:51.275Z","repository":{"id":212561492,"uuid":"730764328","full_name":"getindata/dp-framework","owner":"getindata","description":null,"archived":false,"fork":false,"pushed_at":"2023-12-18T11:46:17.000Z","size":2032,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-01-24T02:30:25.351Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/getindata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-12T16:12:12.000Z","updated_at":"2023-12-15T07:52:49.000Z","dependencies_parsed_at":"2025-01-24T02:29:40.861Z","dependency_job_id":"cbd99204-2704-4462-bf6e-83405b547be2","html_url":"https://github.com/getindata/dp-framework","commit_stats":null,"previous_names":["getindata/dp-framework"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fdp-framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fdp-framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fdp-framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fdp-framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/getindata","download_url":"https://codeload.github.com/getindata/dp-framework/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244066190,"owners_count":20392407,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T23:53:19.285Z","updated_at":"2026-01-04T07:04:42.025Z","avatar_url":"https://github.com/getindata.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# DP Framework\n\n## About\nFrom low-value inputs into high-value outputs - data value chain describes the full data lifecycle from collection to analysis and usage… and it’s not all about data transformation. \nAn open-source dbt-based **DP Framework** has a goal to support the whole process in the spirit of data democratization, in a portable way to many of infrastructure choices and clouds.\n\n![dp_framework_logo.png](images/dp_framework_logo.png)\n\nKey characteristics of DP Framework:\n* Single unified integration layer to stop \"reinventing the wheel\".\n* Readiness for diverse environments - flexibility in component selection to use them interchangeably\n* Ability to work on any environment and with any data storage.\n* Standardization, simplification and unification across projects (through templating)\n* Hides complexity form Analytical Engineers by grouping most of the interactions with a data platform into one user interface\n\n## Components\n\n### data-pipelines-CLI \u0026 Project Template Factory\n\n`data-pipelines-CLI`: [Project on GitHub](https://github.com/getindata/data-pipelines-cli) ([documentation](https://data-pipelines-cli.readthedocs.io/en/latest/index.html))\n![img.png](images/data-pipelines-cli-diagram.png)\n\ndata-pipelines-CLI:\n* Building and managing data pipelines\n* Interaction with the whole data environment\n* Abstraction layer hiding complexity from the end user\n* Handling deployments and publications, automation support\n\nProject Template Factory:\n* Defining standardized templates for your organization’s data pipelines\n* Differentiating config for different environments\n* Creating projects out of templates with a handy cookie cutter\n\n![project-template-factory.png](images/project-template-factory.png)\n\n### dbt-airflow-factory\n\n`dbt-airflow-factory`: [Project on GitHub](https://github.com/getindata/dbt-airflow-factory/) ([documentation](https://dbt-airflow-factory.readthedocs.io/en/latest/index.html))\n\n* parses dbt manifest files and builds orchestrator (Apache Airflow, GCP Workflows, Databricks Workflows) jobs\n* highly customizable, pluggable runtime \n* DAG is built on-the-fly - without materialization\n* task grouping, hiding ephemeral models, etc.\n* sends DAG failure notification to Slack or Microsoft Teams channel\n\n![dbt_to_DAG.png](images/dbt_to_DAG.png)\n\n![img_1.png](images/dag_failure_notifications.png)\n\n## Workshops\n\nAs GetInData we delivered a number of **workshops** on how to deploy dbt pipelines on production using best engineering practices with **DP Framework**.\n\n* [Data Mass 2023](https://summit.datamass.io/build-reliable-data-pipelines-using-modern-data-stack-in-the-cloud-2023/)\n* [BigDataTechWarsaw 2023](https://bigdatatechwarsaw.eu/build-data-pipelines-stack-in-the-cloud/)\n* [Data Mass 2022](https://summit.datamass.io/datamass-gdansk-summit-2022-archive/#agenda)\n* [BigDataTecWarsaw 2022](https://bigdatatechwarsaw.eu/modern-data-pipelines-with-dbt/)\nhttps://github.com/getindata/gid-mdp-workshop\n\nShort demo of our Modern Data Platform with DP Framework:\n[![Watch Modern Data Platform with DP Framework demo](https://img.youtube.com/vi/ECpRKXH243k/0.jpg)](https://www.youtube.com/watch?v=ECpRKXH243k)\n\n## Tutorials\nFirst Steps With DP Framework: [GitHub](https://github.com/getindata/first-steps-with-data-pipelines) \n\n\n## Blog posts \u0026 whitepapers\n\nList of our publications on data platform architectures leveraging **DP Framework**:\n\n* Modern Data Platform - the what's, why's and how's? Demystifying the buzzword [link](https://getindata.com/blog/modern-data-platform-what-why-how-demystifying-buzzword)\n\n* Announcing the GetInData Modern Data Platform -  a self-service solution for Analytics Engineers [link](https://getindata.com/blog/announcing-getindata-modern-data-platform-self-service-solution-analytics-engineers)\n\n* GetInData Modern Data Platform - features \u0026 tools [link](https://getindata.com/blog/getindata-modern-data-platform-features-tools)\n\n* How we built a Modern Data Platform in 4 months for Volt.io, a FinTech scale-up. [link](https://getindata.com/blog/how-we-built-modern-data-platform-for-fintech-scale-up/)\n\n## Presentations\n\nPresentations on various conferences about **DP Framework**: \n* [Providing end-to-end data value chain with open-source dbt-based DP Framework](https://godatafest.com/broadcasts/providing-end-to-end-data-value-chain-open-source-dbt-based-dp-framework/) (GoDataFest, Amsterdam, 2023)\n* [Data Platform - a modern one. A new stack that promotes self-service with well-known best DataOps practices](https://bigdatatechwarsaw.eu/agenda-2023/) (Big Data Tech Warsaw, Warsaw, 2023)\n* [Data Platform - what does it take to be called a modern one? A new stack with well-known best practices](https://dssconf.pl/en/#agenda-section) (Data Science Summit, Warsaw, 2022)\n\n## Contributions\n\nAll the components of DP Framework are open-source. Pull requests are welcome. Please check out detailed contribution instructions on particular project's repository.\n\n\n## Contact us\n[Contact us](https://getindata.com/estimate) \u0026 sign up for DP Framework demo! \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetindata%2Fdp-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgetindata%2Fdp-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetindata%2Fdp-framework/lists"}