https://github.com/z3z1ma/files-datahub

A meltano file bundle for datahub utility
https://github.com/z3z1ma/files-datahub

Last synced: about 1 month ago
JSON representation

A meltano file bundle for datahub utility

• Host: GitHub
• URL: https://github.com/z3z1ma/files-datahub
• Owner: z3z1ma
• License: mit
• Created: 2022-06-14T21:21:43.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2022-06-14T21:24:56.000Z (over 2 years ago)
• Last Synced: 2024-10-13T07:28:22.353Z (2 months ago)
• Language: Python
• Size: 2.93 KB
• Stars: 2
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# files-datahub

Meltano project [file bundle](https://meltano.com/docs/command-line-interface.html#file-bundle) for [Datahub](https://datahubproject.io/docs/).

Files:

- [`utilities`](./bundle/utilities/) (directory)

- [`utilities/datahub/`](./bundle/utilities/datahub/) (directory)

- [`utilities/datahub/README.md`](./bundle/utilities/datahub/README.md)

- [`utilities/datahub/dbt.dhub.yml`](./bundle/utilities/datahub/dbt.dhub.yml)

```py

# add datahub to your Meltano project (including this file bundle)

meltano add utility datahub

# add only this file bundle to your Meltano project

meltano add files datahub

```

We ship a predefined ingestion recipe for dbt artifacts. You must change the target_platform to your database [platform name](https://github.com/datahub-project/datahub/blob/master/metadata-service/war/src/main/resources/boot/data_platforms.json)

```yaml

source:

  type: "dbt"

  config:

    # Coordinates

    manifest_path: ${MELTANO_PROJECT_ROOT}/.meltano/transformers/dbt/target/manifest.json

    catalog_path: ${MELTANO_PROJECT_ROOT}/.meltano/transformers/dbt/target/catalog.json

    sources_path: ${MELTANO_PROJECT_ROOT}/.meltano/transformers/dbt/target/sources.json

    # TODO: Change me to the appropriate platform, ie. bigquery, postgres, etc.

    # https://github.com/datahub-project/datahub/blob/master/metadata-service/war/src/main/resources/boot/data_platforms.json

    target_platform: "CHANGE ME"

sink:

  type: datahub-rest

  config:

    server: ${DATAHUB_GMS_HOST}

    token: ${DATAHUB_GMS_TOKEN}

```