https://github.com/rdeak/neo4j-schema-enforcment-in-etl

About Basic example how Graphql API can enforce schema in neo4j in ETL process executed with Airflow
https://github.com/rdeak/neo4j-schema-enforcment-in-etl

airflow etl neo4j

Last synced: 3 days ago
JSON representation

About Basic example how Graphql API can enforce schema in neo4j in ETL process executed with Airflow

• Host: GitHub
• URL: https://github.com/rdeak/neo4j-schema-enforcment-in-etl
• Owner: rdeak
• License: mit
• Created: 2024-04-08T10:57:57.000Z (9 months ago)
• Default Branch: main
• Last Pushed: 2024-04-08T16:19:03.000Z (9 months ago)
• Last Synced: 2024-11-03T08:42:01.066Z (about 2 months ago)
• Topics: airflow, etl, neo4j
• Language: Python
• Homepage:
• Size: 38.1 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# ETL sample

This repo contains example how enforce neo4j database schema with Graphql API in ETL process.

## Installation

1. `echo -e "AIRFLOW_UID=$(id -u)" > .env`

2. `docker-compose up airflow-init`

3. `docker-compose up`

4. Add `pg_conn` connection for pg service into Connections

5. Create neo4j database (can be done with neo4j browser)

   ```cypher

   :use system;

   create database data

   ``

7. In `api` directory run

   ```bash 

   nvm use

   pnpm install

   pnpm dev

   ```

To remove all just run

`docker compose down --volumes --rmi all`

## Usage

| Service                           | Host                           | User     | Pass         | Database |

|-----------------------------------|--------------------------------|----------|--------------|----------|

| Airflow                           | http://localhost:8080          | airflow  | airflow      |          |

| Graphql API (interface for Neo4J) | http://localhost:4000          |          |              |          |

| Postgres (data source)            | localhost                      | etl_user | etl_password | data     |

| Neo4j                             | http://localhost:7474/browser/ | neo4j    | password     | data     |

## Useful links

* [Neo4j Graphql Toolbox](https://graphql-toolbox.neo4j.io/)

* [Custom Scalars](https://neo4j.com/docs/graphql/current/type-definitions/types/)

## Licence

MIT