{"id":22152146,"url":"https://github.com/findinpath/dbt-trino-incremental-hive","last_synced_at":"2025-07-23T12:34:50.875Z","repository":{"id":112790773,"uuid":"409878728","full_name":"findinpath/dbt-trino-incremental-hive","owner":"findinpath","description":"Test dbt project used to test `incremental` materialization with dbt-trino adapter.","archived":false,"fork":false,"pushed_at":"2021-09-24T07:49:00.000Z","size":5,"stargazers_count":1,"open_issues_count":0,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-11T07:51:16.114Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/findinpath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-24T07:48:33.000Z","updated_at":"2023-05-14T20:20:08.000Z","dependencies_parsed_at":"2023-09-15T01:38:46.914Z","dependency_job_id":null,"html_url":"https://github.com/findinpath/dbt-trino-incremental-hive","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/findinpath/dbt-trino-incremental-hive","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fdbt-trino-incremental-hive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fdbt-trino-incremental-hive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fdbt-trino-incremental-hive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fdbt-trino-incremental-hive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/findinpath","download_url":"https://codeload.github.com/findinpath/dbt-trino-incremental-hive/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fdbt-trino-incremental-hive/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266680298,"owners_count":23967791,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-02T00:47:24.623Z","updated_at":"2025-07-23T12:34:50.845Z","avatar_url":"https://github.com/findinpath.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"dbt-trino-incremental-hive\n=====================\n\nThis is a test [dbt](https://www.getdbt.com/) project used to test `incremental` materialization with dbt-trino adapter.\n\n[dbt-trino](https://github.com/findinpath/dbt-trino) is a `dbt` adapter used to interact with [Trino](https://trino.io/)\n\nThe dbt [incremental](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/materializations#incremental) materialization provides two strategies:\n\n- `append` : appends new entries to the model\n- `insert_overwrite` : deletes eventually from the model the entries which exist in the staging table and then appends the staging entries to the model\n\n\nThis project is an attempt to test the `incremental` materializations on Trino by using the [hive](https://trino.io/docs/current/connector/hive.html) connector.\n\n\n\n\n## Local setup\n\nUse [virtualenv](https://pypi.org/project/virtualenv/) for creating a _virtual_ python environment:\n\n```\npip3 install virtualenv\nvirtualenv venv\nsource venv/bin/activate\n\n```\n\n\nInstall the feature branch of dbt-trino which contains the `incremental` materialization implemenation:\n\n\nhttps://github.com/findinpath/dbt-trino/tree/feature/incremental\n\nRun the command\n\n```bash\npip3 install the_location_of_the_dbt_trino_directory\n```\n\nSpin up Docker environment for working with Trino \u0026 [minio](https://min.io/) hive compatible\nobject storage.\n\n```\ndocker-compose up -d\n```\n\n\nIn `~/.dbt/profiles.yml` add the following configuration:\n\n\n```\ntrino-incremental-hive:\n  target: dev\n  outputs:\n    dev:\n      type: trino\n      method: none\n      user: admin\n      password:\n      catalog: minio\n      schema: tiny\n      host: localhost\n      port: 8080\n      http_scheme: http\n      threads: 1  \n```\n\n\nCheck now that everything is ok before performing the dbt transformations:\n\n```\n(venv)#  dbt debug\n```\n\n### Create a bucket in MinIO\n\n\nOpen [MinIO UI](http://localhost:9000/) by using the following credentials:\n\n- access key: minio\n- secret key: minio123\n\nCreate the bucket `tiny`\n\n\n\n### Trino CLI\n\n```\ndocker container exec -it trino-dbt-trino-incremental-hive_trino-coordinator_1 trino\n```\n\n```\ntrino\u003e show catalogs;\n```\n\n\n### Create Trino `minio.tiny` schema\n\n```\nCREATE SCHEMA minio.tiny\nWITH (location = 's3a://tiny/');\n```\n\n### Create `minio.tiny.raw_customer` hive table\n\n\n```sql\nCREATE TABLE minio.tiny.raw_customer (            \n    id bigint,                                     \n    first_name varchar(32),                        \n    last_name varchar(32),                         \n    email varchar(256)                             \n )                                                 \n WITH (                                            \n    external_location = 's3a://tiny/raw_customer', \n    format = 'ORC'                                 \n );\n```\n\n### Populate `minio.tiny.raw_customer` hive table\n\n\n```\nINSERT INTO minio.tiny.raw_customer (id, first_name, last_name, email)\n    VALUES \n        (1,'Michael','Perez','mperez0@chronoengine.com'),\n        (2,'Shawn','Mccoy','smccoy1@reddit.com'),\n        (3,'Kathleen','Payne','kpayne2@cargocollective.com');\n\n```\n\n\n### Run dbt for creating the models\n\n```bash\n(venv)# dbt run\n```\n\n\n### Troubleshooting\n\nThe problems start occuring when running `dbt` more than once.\n\nHere is an example\n\n```\nRuntime Error in model customer_insert_overwrite (models/marts/core/customer_insert_overwrite.sql)\n  TrinoUserError(type=USER_ERROR, name=NOT_SUPPORTED, message=\"Deletes must match whole partitions for non-transactional tables\", query_id=20210924_074331_00048_njuyb)\n```\n\n\n\n### Stop the Trino environment\n\n```\ndocker-compose down\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Fdbt-trino-incremental-hive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffindinpath%2Fdbt-trino-incremental-hive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Fdbt-trino-incremental-hive/lists"}