https://github.com/os-climate/pcaf_data_mesh_local_infra

https://github.com/os-climate/pcaf_data_mesh_local_infra

Last synced: 7 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/os-climate/pcaf_data_mesh_local_infra
• Owner: os-climate
• License: apache-2.0
• Created: 2023-10-10T16:03:21.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-07-03T11:17:24.000Z (4 months ago)
• Last Synced: 2024-08-13T19:46:10.009Z (3 months ago)
• Language: Jupyter Notebook
• Homepage:
• Size: 522 KB
• Stars: 1
• Watchers: 3
• Forks: 4
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        


> [!IMPORTANT]

> On June 26 2024, Linux Foundation announced the merger of its financial services umbrella, the Fintech Open Source Foundation ([FINOS](https://finos.org)), with OS-Climate, an open source community dedicated to building data technologies, modeling, and analytic tools that will drive global capital flows into climate change mitigation and resilience; OS-Climate projects are in the process of transitioning to the [FINOS governance framework](https://community.finos.org/docs/governance); read more on [finos.org/press/finos-join-forces-os-open-source-climate-sustainability-esg](https://finos.org/press/finos-join-forces-os-open-source-climate-sustainability-esg)

# PCAF Data Mesh POC Infrastructure

A repository containing the local infrastructure for running the PCAF data ingestion pipeline. 

## How to build image 

make sure existing images , conatianes and volumes are  removed before builidng new iamge  

    

    docker compose rm -fsv 

## Build new image.  

    docker compose build --no-cache 

## How to run the all service defined in you docker compose 

    docker compose up

## How to run the specific service in you docker compose 

    docker compose up 

## Adding new DAG to your project

all DAG  must be added to the filder airflow/dags/

## Adding dependencies

if any of your DAG need dependency python packaged , the dependencies should be added to " Dockerfile located in root of the project 

Example : RUN pip install apache-airflow-providers-trino

## OpenmetaData integration configuration 

Airflow required to have authendication token from MetaData server to get auhtendicaed to send leanage data back to OpenMetaData. Currently this token by default generated from Open MetaData application. In order to get tocken, after run the "docker compose up", You need to login to OpenMeta applcation with the following credential 

url      -   localhost:8585

user id  -   admin 

pasword  - admin

and then copy the token from openMetaData applcation by navigating to ( settings--> BOT --> ingestion-bot --> ) and update environment variable "AIRFLOW__LINEAGE__JWT_TOKEN" to the docker-compose.xml file located in the root of the project . 

## Restart only Airflow container 

    docker compose up airflow-webserver