https://github.com/astrojuanlu/orchest-elt-meltano-dbt

Example of an ELT pipeline in Orchest using meltano and dbt
https://github.com/astrojuanlu/orchest-elt-meltano-dbt

Last synced: about 1 month ago
JSON representation

Example of an ELT pipeline in Orchest using meltano and dbt

• Host: GitHub
• URL: https://github.com/astrojuanlu/orchest-elt-meltano-dbt
• Owner: astrojuanlu
• Created: 2022-07-22T14:41:27.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2022-07-27T12:00:55.000Z (over 2 years ago)
• Last Synced: 2024-10-14T19:59:32.136Z (2 months ago)
• Language: Jupyter Notebook
• Size: 18.6 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# ELT pipeline in Orchest with meltano and dbt

[![Open in Orchest](https://github.com/orchest/orchest-examples/raw/main/imgs/open_in_orchest.svg)](https://cloud.orchest.io/?import_url=https://github.com/astrojuanlu/orchest-elt-meltano-dbt/)

Creating an ELT pipeline in Orchest that extracts data from PostgreSQL and loads it to BigQuery

using meltano and dbt.

![Pipeline visualization](https://pviz.orchest.io/?pipeline=https://github.com/astrojuanlu/orchest-elt-meltano-dbt/blob/main/main.orchest)

## Credentials

To make the main pipeline work, you need a `GOOGLE_APPLICATION_CREDENTIALS` environment variable pointing to a path containing a JSON key with BigQuery credentials (see [instructions](https://github.com/adswerve/target-bigquery#step-2-authenticate-with-a-service-account)).

## Development

### Init commands

These commands were used to create the configuration file,

using the Jupyter Console. There is no need to run them again.

```

! meltano init meltano

% cd meltano

! meltano add extractor tap-postgres

! meltano add loader target-bigquery

! meltano config tap-postgres set user postgres

! meltano config tap-postgres set dbname postgres

! meltano config tap-postgres set password ''

! meltano select tap-postgres '*' '*'

! meltano config tap-postgres set _metadata '*' replication-method FULL_TABLE

! meltano add transformer dbt-bigquery

```