https://github.com/taeefnajib/target-iceberg

Meltano target for Apache Iceberg
https://github.com/taeefnajib/target-iceberg

Last synced: about 15 hours ago
JSON representation

Meltano target for Apache Iceberg

• Host: GitHub
• URL: https://github.com/taeefnajib/target-iceberg
• Owner: taeefnajib
• License: apache-2.0
• Created: 2023-12-04T17:36:44.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2023-12-28T03:42:26.000Z (about 1 year ago)
• Last Synced: 2023-12-28T04:29:38.954Z (about 1 year ago)
• Language: Python
• Homepage:
• Size: 356 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
## How to Add `target-iceberg` to Meltano

Run `meltano add --custom loader target-iceberg` 

Take the default value by pressing enter for name, namespace, settings and executable. However when it asks for pip_url:

Add: `git+https://github.com/taeefnajib/target-iceberg.git` 

In your `meltano.yml` file you'll see something like this:

```yml

...

loaders:

  - name: target-iceberg

    namespace: target_iceberg

    pip_url: git+https://github.com/taeefnajib/target-iceberg.git

    executable: target-iceberg

...

```

Add the config properties manually:

```yml

config:

      table_name: <>

      aws_access_key: $AWS_ACCESS_KEY

      aws_secret_key: $AWS_SECRET_KEY

      aws_region: <>

      hive_thrift_uri: <>

      warehouse_uri: <>

      partition_by:

      - <>

```

Example:

```yml

config:

      table_name: testtable

      aws_access_key: $AWS_ACCESS_KEY

      aws_secret_key: $AWS_SECRET_KEY

      aws_region: us-east-1

      hive_thrift_uri: thrift://0.0.0.0:9083

      warehouse_uri: s3a://iceberg-data-lake/warehouse

      partition_by:

      - Month

```

Also, create a `.env` file and include the environment variables for AWS authentication:

```

export AWS_ACCESS_KEY="XXXXXXXXXXXXXXXXX"

export AWS_SECRET_KEY="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

```

Assuming you already have an extractor set up, you can start syncing. Example:

```bash

meltano el tap-csv target-iceberg

```