{"id":22152123,"url":"https://github.com/findinpath/trino_data_mesh","last_synced_at":"2025-09-20T07:19:31.745Z","repository":{"id":112791177,"uuid":"382719204","full_name":"findinpath/trino_data_mesh","owner":"findinpath","description":"Proof of concept on how to gain insights with Trino across different databases from a distributed data mesh","archived":false,"fork":false,"pushed_at":"2021-07-05T18:18:42.000Z","size":40,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T13:27:18.128Z","etag":null,"topics":["data-mesh","dbt","trinodb"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/findinpath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-03T22:24:25.000Z","updated_at":"2022-07-24T06:01:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"dd7d5826-88f5-4d43-b499-513c04da5422","html_url":"https://github.com/findinpath/trino_data_mesh","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Ftrino_data_mesh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Ftrino_data_mesh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Ftrino_data_mesh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Ftrino_data_mesh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/findinpath","download_url":"https://codeload.github.com/findinpath/trino_data_mesh/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248243497,"owners_count":21071054,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-mesh","dbt","trinodb"],"created_at":"2024-12-02T00:47:13.488Z","updated_at":"2025-09-20T07:19:26.720Z","avatar_url":"https://github.com/findinpath.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"Trino in the data mesh\n==============================\n\nThis project is inspired by the following two projects:\n\n- [jaffle_shop](https://github.com/dbt-labs/jaffle_shop/) which is showcasing the functionality \n  of [dbt](https://www.getdbt.com/) data warehouse transformation  tool.\n- [trino-dbt-demo](https://github.com/victorcouste/trino-dbt-demo) which showcases how to use `dbt` in a scenario where the \n  data to be transformed is found in several databases. See also the associated \n  [blog](https://medium.com/geekculture/trino-dbt-a-match-in-sql-heaven-1df2a3d12b5e) for this project. \n\n\n## Introduction\n\nOne frequently asked question in the context of using `dbt` tool is: \n\n\u003e Can I connect my dbt project to two databases?\n\n(see the answered [question](https://docs.getdbt.com/faqs/connecting-to-two-dbs-not-allowed) on the dbt website).\n\n**tldr;** `dbt` stands for transformation as in `T` within `ELT`  pipelines, it doesn't move data from source to a warehouse.\n\nThe creators of the `dbt` tool however have added support for handling such scenarios via \n[dbt-presto](https://github.com/dbt-labs/dbt-presto) plugin.\n\nThe project [trino-dbt-demo](https://github.com/victorcouste/trino-dbt-demo) can be used as a primer in the context\nof stitching together content from multiple databases over `dbt`.\n\n[Trino](https://trino.io/) is a [fork](https://trino.io/blog/2020/12/27/announcing-trino.html) of the popular \npresto high performance, distributed SQL query engine for big data.\nThis SQL query engine offers a helping hand in performing SQL queries on top of a myriad of data sources.\nTrino supports talking to the common relational databases (Postgres, MySQL) and also to data sources\nthat don't support SQL (AWS S3, Apache Kafka, Apache Cassandra, etc.). Feel free to check\nthe list of supported Trino [connectors](https://trino.io/docs/current/connector.html) for more details.\n\nBy using Trino/Presto, there can be queried data from fully separated databases. This makes Trino \n_the analytics engine for data mesh_ (quote from [Starburst](https://www.starburst.io/) website).\n\n\n## jaffle shop data mesh\n\n[Data Mesh](https://www.thoughtworks.com/radar/techniques/data-mesh) is a paradigm to the data engineering domain\nwhich provides an alternative to the common recipe of using a centralized, monolithic data warehouse.\n\nThe principles on which this paradigm is being founded are quoted below:\n\n- domain-oriented decentralization of data ownership and architecture \n- domain-oriented data served as a product \n- self-serve data infrastructure as a platform to enable autonomous, domain-oriented data teams \n- federated governance to enable ecosystems and interoperability. \n\nIn the context of the project [jaffle_shop](https://github.com/dbt-labs/jaffle_shop/) there are being used\nthe domains:\n\n- customers\n- orders (customers make orders)\n- payments (each completed order has a corresponding payment)\n\nThis project provides the answer to the questions answered by the project [jaffle_shop](https://github.com/dbt-labs/jaffle_shop/) \n(which was operating in the centralized data warehouse context) in a _data mesh_ decentralized data warehouse context \nwhere each domain (customer, order, payment) is being stored in a separate database.\n\n![jaffle shop data mesh](img/data_mesh.png)\n\nBy spinning up the [docker](https://www.docker.com/) environment:\n\n```bash\ndocker-compose up\n```\n\nthere will be started the following containers:\n\n- `trino_jaffle_shop_paymentsdb_1`: contains the payments data in a Postgres database\n- `trino_jaffle_shop_ordersdb_1`: contains the orders data in a Postgres database\n- `trino_jaffle_shop_customersdb_1`: contains the customers data in a Postgres database\n- `trino_jaffle_shop_trino_1`: contains corresponding catalogs for the tree databases mentioned previously\n\nOnce the containers are spinned up, there can be made interactions with Trino via Trino CLI:\n\n```bash\ndocker exec -it trino_jaffle_shop_trino_1 /usr/bin/trino\n```\n\nThe catalogs shown should correspond to the databases belonging to the jaffle shop domains:\n\n```sql\ntrino\u003e show catalogs;\n   Catalog   \n-------------\n customersdb \n ordersdb    \n paymentsdb  \n system      \n(4 rows)\n```\n\nHere are some basic queries performed on each of the database to make sure that\neverything is running as expected:\n\n```sql\ntrino\u003e select count(*) from customersdb.public.customers;\n _col0 \n-------\n   100 \n(1 row)\n\nQuery 20210703_204341_00001_pdadk, FINISHED, 1 node\nSplits: 17 total, 17 done (100.00%)\n0.32 [1 rows, 0B] [3 rows/s, 0B/s]\n\ntrino\u003e select count(*) from ordersdb.public.orders;\n _col0 \n-------\n    99 \n(1 row)\n\nQuery 20210703_204405_00002_pdadk, FINISHED, 1 node\nSplits: 17 total, 17 done (100.00%)\n0.22 [1 rows, 0B] [4 rows/s, 0B/s]\n\ntrino\u003e select count(*) from paymentsdb.public.payments;\n _col0 \n-------\n   113 \n(1 row)\n\nQuery 20210703_204420_00003_pdadk, FINISHED, 1 node\nSplits: 17 total, 17 done (100.00%)\n0.22 [1 rows, 0B] [4 rows/s, 0B/s]\n```\n\nNow that the access to each of the databases from Trino has been verified, \nthe query corresponding to the [customers](https://github.com/dbt-labs/jaffle_shop/blob/main/models/customers.sql)\n`dbt` model from the [jaffle_shop](https://github.com/dbt-labs/jaffle_shop/) project can be executed:\n\n\n```sql\nwith customers as (\n\n  select *\n  from customersdb.public.customers\n),\n orders as (\n   select *\n   from ordersdb.public.orders\n ),\n payments as (\n   select *\n   from paymentsdb.public.payments\n ),\n customer_orders as (\n   select\n     user_id,\n     min(order_date) as first_order,\n     max(order_date) as most_recent_order,\n     count(id) as number_of_orders\n   from orders\n   group by 1\n ),\n customer_payments as (\n\n   select\n     orders.user_id,\n     sum(amount) as total_amount\n\n   from payments\n          left join orders on payments.order_id = orders.id\n   group by 1\n ),\n\n final as (\n\n   select\n     customers.id as customer_id,\n     customers.first_name,\n     customers.last_name,\n     customer_orders.first_order,\n     customer_orders.most_recent_order,\n     customer_orders.number_of_orders,\n     customer_payments.total_amount as customer_lifetime_value\n   from customers\n          left join customer_orders on customers.id = customer_orders.user_id\n          left join customer_payments on customers.id = customer_payments.user_id\n )\n\nselect * from final\norder by customer_id;\n```\n\nHere's a sneak peak of the results:\n\n```sql\n customer_id | first_name | last_name | first_order | most_recent_order | number_of_orders | customer_lifetime_value \n-------------+------------+-----------+-------------+-------------------+------------------+-------------------------\n           1 | Michael    | P.        | 2018-01-01  | 2018-02-10        |                2 |                    3300 \n           2 | Shawn      | M.        | 2018-01-11  | 2018-01-11        |                1 |                    2300 \n           3 | Kathleen   | P.        | 2018-01-02  | 2018-03-11        |                3 |                    6500 \n           4 | Jimmy      | C.        | NULL        | NULL              |             NULL |                    NULL \n           5 | Katherine  | R.        | NULL        | NULL              |             NULL |                    NULL\n```\n\nNote in the previous query that the data corresponding to the domain models:\n\n- customer\n- order\n- payment\n\nis being joined transparently in order to provide insights across all the aforementioned domain models.\n\nEven though this proof of concept project uses for simplicity a Postgres database to store the data for \neach of the domain models, there is a [myriad](https://trino.io/docs/current/connector.html) of other data sources \nthat are supported on Trino and can be joined together.\n\nClean up the test environment by running the command:\n\n```bash\ndocker-compose down\n```\n\n\n## Conclusion\n\nThis proof of concept project gives a quick demo on how to join and process data in the context of a\ndistributed _Data mesh_.  \n\nFeel free to head over to [Trino](https://trino.io/) website and give it a try in case you may be dealing\nwith any of the scenarios:\n\n- joining data from different data sources\n- joining relational data with data found on AWS S3 (btw. AWS Athena is built on the same technology)\n\nIn case that the insights gained from analyzing the data mesh, need to be stored on a persistent storage\nTrino does offer the ability to create tables. \nNote that the support for `MERGE` statements in still [in progress](https://github.com/trinodb/trino/issues/7708) in Trino.\n\nFor transformations requiring multiple steps it is worth taking into account the [dbt data transformation tool](https://www.getdbt.com/).\n\n\n## Documentation\n\n- [Data Mesh introductory article](https://martinfowler.com/articles/data-monolith-to-mesh.html)\n- [Trino distributed query engine](https://trino.io/)\n- [dbt tool](https://www.getdbt.com/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Ftrino_data_mesh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffindinpath%2Ftrino_data_mesh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Ftrino_data_mesh/lists"}