{"id":22152142,"url":"https://github.com/findinpath/dbt_jaffle_shop_data_mesh","last_synced_at":"2026-03-19T21:58:06.871Z","repository":{"id":112790860,"uuid":"408224982","full_name":"findinpath/dbt_jaffle_shop_data_mesh","owner":"findinpath","description":"A self-contained dbt project for showcasing the capabilities of the dbt-trino adapter","archived":false,"fork":false,"pushed_at":"2021-09-19T20:04:18.000Z","size":853,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-06T21:40:50.333Z","etag":null,"topics":["data-mesh","dbt","dbt-trino"],"latest_commit_sha":null,"homepage":"","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/findinpath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-19T20:02:50.000Z","updated_at":"2023-11-24T21:50:58.000Z","dependencies_parsed_at":"2023-09-15T04:01:36.939Z","dependency_job_id":null,"html_url":"https://github.com/findinpath/dbt_jaffle_shop_data_mesh","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/findinpath/dbt_jaffle_shop_data_mesh","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fdbt_jaffle_shop_data_mesh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fdbt_jaffle_shop_data_mesh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fdbt_jaffle_shop_data_mesh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fdbt_jaffle_shop_data_mesh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/findinpath","download_url":"https://codeload.github.com/findinpath/dbt_jaffle_shop_data_mesh/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fdbt_jaffle_shop_data_mesh/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28911821,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T08:15:08.179Z","status":"ssl_error","status_checked_at":"2026-01-30T08:14:31.507Z","response_time":66,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-mesh","dbt","dbt-trino"],"created_at":"2024-12-02T00:47:24.617Z","updated_at":"2026-01-30T11:20:08.887Z","avatar_url":"https://github.com/findinpath.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"`jaffle_shop` data mesh\n==============================\n\nThis project has the purpose of showcasing [dbt-trino](https://github.com/findinpath/dbt-trino) dbt adapter functionality\nin the context of the `jaffle_shop` `dbt` project in order to showcase how `jaffle_shop` project would look like in a \n_data mesh_ context.\n\nThis showcase is inspired by the following two projects:\n\n- [jaffle_shop](https://github.com/dbt-labs/jaffle_shop/) which is showcasing the functionality \n  of [dbt](https://www.getdbt.com/) data warehouse transformation  tool.\n- [trino-dbt-demo](https://github.com/victorcouste/trino-dbt-demo) which showcases how to use `dbt` in a scenario where the \n  data to be transformed is found in several databases. See also the associated \n  [blog](https://medium.com/geekculture/trino-dbt-a-match-in-sql-heaven-1df2a3d12b5e) for this project. \n\n\n## Introduction\n\nOne frequently asked question in the context of using `dbt` tool is: \n\n\u003e Can I connect my dbt project to two databases?\n\n(see the answered [question](https://docs.getdbt.com/faqs/connecting-to-two-dbs-not-allowed) on the dbt website).\n\n**tldr;** `dbt` stands for transformation as in `T` within `ELT`  pipelines, it doesn't move data from source to a warehouse.\n\nThe creators of the `dbt` tool however have added support for handling such scenarios via \n[dbt-presto](https://github.com/dbt-labs/dbt-presto) plugin.\n\n[Trino](https://trino.io/) is a [fork](https://trino.io/blog/2020/12/27/announcing-trino.html) of the popular \npresto high performance, distributed SQL query engine for big data.\nThis SQL query engine offers a helping hand in performing SQL queries on top of a myriad of data sources.\nTrino supports talking to the common relational databases (Postgres, MySQL) and also to data sources\nthat don't support SQL (AWS S3, Apache Kafka, Apache Cassandra, etc.). Feel free to check\nthe list of supported Trino [connectors](https://trino.io/docs/current/connector.html) for more details.\n\nBy using Trino, there can be queried data from fully separated databases. This makes Trino \n_the analytics engine for data mesh_ (quote from [Starburst Data](https://www.starburst.io/) website).\n\n![dbt-trino-architecture](img/dbt-trino-architecture.png)\n\n_dbt Trino Architecture Image taken from [Trino Community Broadcast](https://trino.io/episodes/21.html)_\n\n\n## jaffle shop data mesh\n\n[Data Mesh](https://www.thoughtworks.com/radar/techniques/data-mesh) is a paradigm to the data engineering domain\nwhich provides an alternative to the common recipe of using a centralized, monolithic data warehouse.\n\nThe principles on which this paradigm is being founded are quoted below:\n\n- domain-oriented decentralization of data ownership and architecture \n- domain-oriented data served as a product \n- self-serve data infrastructure as a platform to enable autonomous, domain-oriented data teams \n- federated governance to enable ecosystems and interoperability. \n\nIn the context of the project [jaffle_shop](https://github.com/dbt-labs/jaffle_shop/) there are being used\nthe domains:\n\n- customers\n- orders (customers make orders)\n- payments (each completed order has a corresponding payment)\n\n![jaffle shop entity relationship diagram](img/jaffle_shop_erd.png)\n\nThis project provides the answer to the questions answered by the project [jaffle_shop](https://github.com/dbt-labs/jaffle_shop/) \n(which was operating in the centralized data warehouse context) in a _data mesh_ decentralized data warehouse context \nwhere each domain (customer, order, payment) is being stored in a separate database.\nThe insights gained from the `dbt` transformations will be also saved in a separate database.\n\n![jaffle shop data mesh](img/data_mesh.png)\n\n\n## Local setup\n\nCreate a docker network used for test purposes:\n\n```bash\ndocker network create trino_jaffle_shop_network\n```\n\n\nSpin up the [docker](https://www.docker.com/) environment:\n\n```bash\ndocker-compose -f ./docker/docker-compose.yaml up -d\n```\n\nthere will be started the following containers:\n\n- `docker_paymentsdb_1`: contains the payments data in a Postgres database\n- `docker_ordersdb_1`: contains the orders data in a Postgres database\n- `docker_customersdb_1`: contains the customers data in a Postgres database\n- `docker_trino_1`: Trino SQL query engine which maps corresponding catalogs for the three databases mentioned previously and also an `insights` catalog of type `memory` to store the output of `dbt` transformations\n\n\nThe insights gained from analyzing the data mesh, are stored for the purpose of this showcase only on memory, but obviously \nTrino offers the possibility to persist the results of the transformations to a persistent storage as well.\n\n\nOnce the containers are spinned up, there can be made interactions with Trino via Trino CLI:\n\n```bash\ndocker exec -it docker_trino_1 /usr/bin/trino\n```\n\nThe catalogs shown should correspond to the databases belonging to the jaffle shop domains:\n\n```sql\ntrino\u003e show catalogs;\nCatalog   \n-------------\n customersdb \n insights    \n ordersdb    \n paymentsdb  \n system      \n(5 rows)\n```\n\nHere are some basic queries performed on each of the database to make sure that\neverything is running as expected:\n\n```sql\ntrino\u003e select count(*) from customersdb.public.customers;\n _col0 \n-------\n   100 \n(1 row)\n\nQuery 20210703_204341_00001_pdadk, FINISHED, 1 node\nSplits: 17 total, 17 done (100.00%)\n0.32 [1 rows, 0B] [3 rows/s, 0B/s]\n\ntrino\u003e select count(*) from ordersdb.public.orders;\n _col0 \n-------\n    99 \n(1 row)\n\nQuery 20210703_204405_00002_pdadk, FINISHED, 1 node\nSplits: 17 total, 17 done (100.00%)\n0.22 [1 rows, 0B] [4 rows/s, 0B/s]\n\ntrino\u003e select count(*) from paymentsdb.public.payments;\n _col0 \n-------\n   113 \n(1 row)\n\nQuery 20210703_204420_00003_pdadk, FINISHED, 1 node\nSplits: 17 total, 17 done (100.00%)\n0.22 [1 rows, 0B] [4 rows/s, 0B/s]\n```\n\nNow that the access to each of the databases from Trino has been verified, \nthere can be built transformations of the Trino virtual data warehouse \nvia [dbt-trino](https://github.com/findinpath/dbt-trino) adapter.\n\nThis showcase comes with an adapted version of [jaffle_shop](https://github.com/dbt-labs/jaffle_shop/)\nwhich allows performing the `dbt` transformations in a docker container, so there is\nno more need to install and configure `dbt` locally.\n\n\nBuild the `dbt-trino-jaffle-shop` docker image:\n\n\n```\n./docker/dbt/build.sh\n```\n\nOnce the container is built, run the `dbt` transformations from inside a container of the previously built docker image:\n\n```\n./docker/insights.sh\n\nRunning with dbt=0.20.0\nFound 8 models, 20 tests, 0 snapshots, 0 analyses, 146 macros, 0 operations, 0 seed files, 3 sources, 0 exposures\n\n21:41:00 | Concurrency: 1 threads (target='dev')\n21:41:00 | \n21:41:01 | 1 of 8 START view model default.stg_orders........................... [RUN]\n21:41:01 | 1 of 8 OK created view model default.stg_orders...................... [OK in 0.10s]\n21:41:01 | 2 of 8 START view model default.stg_payments......................... [RUN]\n21:41:01 | 2 of 8 OK created view model default.stg_payments.................... [OK in 0.08s]\n21:41:01 | 3 of 8 START view model default.stg_customers........................ [RUN]\n21:41:01 | 3 of 8 OK created view model default.stg_customers................... [OK in 0.07s]\n21:41:01 | 4 of 8 START table model default.customer_orders..................... [RUN]\n21:41:01 | 4 of 8 OK created table model default.customer_orders................ [OK in 0.52s]\n21:41:01 | 5 of 8 START table model default.customer_payments................... [RUN]\n21:41:02 | 5 of 8 OK created table model default.customer_payments.............. [OK in 0.48s]\n21:41:02 | 6 of 8 START table model default.order_payments...................... [RUN]\n21:41:02 | 6 of 8 OK created table model default.order_payments................. [OK in 0.35s]\n21:41:02 | 7 of 8 START table model default.dim_customers....................... [RUN]\n21:41:02 | 7 of 8 OK created table model default.dim_customers.................. [OK in 0.32s]\n21:41:02 | 8 of 8 START table model default.fct_orders.......................... [RUN]\n21:41:03 | 8 of 8 OK created table model default.fct_orders..................... [OK in 0.26s]\n21:41:03 | \n21:41:03 | Finished running 3 view models, 5 table models in 2.89s.\n\nCompleted successfully\n\nDone. PASS=8 WARN=0 ERROR=0 SKIP=0 TOTAL=8\n\n```\n\nOnce the `dbt` transformations complete successfully, there can be verified the content of the `insights` Trino catalog :\n\n```bash\ndocker exec -it docker_trino_1 /usr/bin/trino\n```\n\n\n```\ntrino\u003e show tables in insights.default;\n       Table       \n-------------------\n customer_orders   \n customer_payments \n dim_customers     \n fct_orders        \n order_payments    \n stg_customers     \n stg_orders        \n stg_payments      \n(8 rows)\n\nQuery 20210918_214153_00126_zb79t, FINISHED, 1 node\nSplits: 19 total, 19 done (100.00%)\n0.21 [8 rows, 240B] [37 rows/s, 1.09KB/s]\n\n\ntrino\u003e select * from insights.default.dim_customers;\n customer_id | first_order | most_recent_order | number_of_orders | customer_lifetime_value \n-------------+-------------+-------------------+------------------+-------------------------\n           1 | 2018-01-01  | 2018-02-10        |                2 |                      33 \n           2 | 2018-01-11  | 2018-01-11        |                1 |                      23 \n           3 | 2018-01-02  | 2018-03-11        |                3 |                      65 \n           4 | NULL        | NULL              |             NULL |                    NULL \n           5 | NULL        | NULL              |             NULL |                    NULL \n           6 | 2018-02-19  | 2018-02-19        |                1 |                       8 \n....\n\n```\n\n**NOTE** in the snippet above that the tables:\n\n- `stg_customers`\n- `stg_orders`\n- `stg_payments`\n\nare actually *views*, virtual tables, whose contents are defined by queries that are actually performed on\nexternal databases via Trino.\n\n\nBy looking at the `dbt` model [dim_customers.sql](docker/dbt/jaffle_shop/models/marts/core/dim_customers.sql)\ncan be observed that the data corresponding to the domain models:\n\n- customer\n- order\n- payment\n\nis being joined in order to provide insights across all the aforementioned domain models.\n\nEven though this proof of concept project uses for simplicity Postgres databases to store the data for \neach of the domain models, there is a [myriad](https://trino.io/docs/current/connector.html) of other data sources \nthat are supported on Trino and can be joined together.\n\n\n### Clean up\n\nClean up the test environment by running the shutting down the docker containers:\n\n\n```bash\ndocker-compose -f ./docker/docker-compose.yaml up -d\n\nStopping docker_customersdb_1 ... done\nStopping docker_trino_1       ... done\nStopping docker_paymentsdb_1  ... done\nStopping docker_ordersdb_1    ... done\nRemoving docker_customersdb_1 ... done\nRemoving docker_trino_1       ... done\nRemoving docker_paymentsdb_1  ... done\nRemoving docker_ordersdb_1    ... done\nNetwork trino_jaffle_shop_network is external, skipping\n```\n\nRemove the docker network \n\n```bash\ndocker network rm trino_jaffle_shop_network\n```\n\n## Troubleshooting\n\nIn case you may be eventually dealing with problems in running this showcase on your workstation,\ncheck out generic information about `dbt` via the command:\n\n```bash\ndocker run --network=trino_jaffle_shop_network --rm -it --entrypoint sh dbt-trino-jaffle-shop\n\n# dbt debug\nRunning with dbt=0.20.0\ndbt version: 0.20.0\npython version: 3.8.11\npython path: /usr/local/bin/python\nos info: Linux-5.4.0-84-generic-x86_64-with-glibc2.2.5\nUsing profiles.yml file at /root/.dbt/profiles.yml\nUsing dbt_project.yml file at /jaffle_shop/dbt_project.yml\n\nConfiguration:\n  profiles.yml file [OK found and valid]\n  dbt_project.yml file [OK found and valid]\n\nRequired dependencies:\n - git [OK found]\n\nConnection:\n  host: trino\n  port: 8080\n  user: admin\n  database: insights\n  schema: default\n  Connection test: OK connection ok\n```\n\nIn case that a `dbt` transformation doesn't successfully run, use the following commands to debug the cause of the issue:\n\n```bash\ndocker run --network=trino_jaffle_shop_network --rm -it --entrypoint sh dbt-trino-jaffle-shop\n\n# dbt run\n...\n# cat logs/dbt.log\n```\n\n\n\n## Conclusion\n\nThis proof of concept project gives a quick demo on how to perform `dbt` transformations in the context of a\ndistributed _Data mesh_.  \n\nFeel free to head over to [Trino](https://trino.io/) website and give it a try in case you may be dealing\nwith any of the scenarios:\n\n- joining data from different data sources\n- joining relational data with data found on AWS S3 (btw. AWS Athena is built on the same technology)\n\nNote that the support for `MERGE` statements in still [in progress](https://github.com/trinodb/trino/issues/7708) in Trino.\n\nIn case that this showcase project got you interested, head over to [dbt-trino](https://github.com/findinpath/dbt-trino) project\npage and give it a try. Feedback is very much welcome.\n\n## Documentation\n\n- [Data Mesh introductory article](https://martinfowler.com/articles/data-monolith-to-mesh.html)\n- [Trino distributed query engine](https://trino.io/)\n- [dbt data transformation tool](https://www.getdbt.com/)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Fdbt_jaffle_shop_data_mesh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffindinpath%2Fdbt_jaffle_shop_data_mesh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Fdbt_jaffle_shop_data_mesh/lists"}