https://github.com/manesioz/airflow-without-code

Dynamically generate DAGs to ingest SQL files into BigQuery with one line of "code"
https://github.com/manesioz/airflow-without-code

airflow airflow-plugin bigquery python sql

Last synced: 2 months ago
JSON representation

Dynamically generate DAGs to ingest SQL files into BigQuery with one line of "code"

• Host: GitHub
• URL: https://github.com/manesioz/airflow-without-code
• Owner: manesioz
• License: mit
• Created: 2019-12-20T14:36:08.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2019-12-26T04:42:31.000Z (over 6 years ago)
• Last Synced: 2025-01-05T16:12:15.635Z (over 1 year ago)
• Topics: airflow, airflow-plugin, bigquery, python, sql
• Language: Python
• Homepage:
• Size: 14.6 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# airflow-without-code

Dynamically generate DAGs to run SQL files and ingest into BigQuery with one line of "code" at the header. 

### Example usage 

Given the directory: 

    .

    ├── ...

    ├── etl                          # ETL directory 

    │   ├── sql_script_1.sql         # first sql file to ingest into BigQuery

    │   ├── sql_script_2.sql         # second sql file to ingest into BigQuery

    │   └── airflow_without_code.py  # airflow-without-code main python file

    └── ...

Now you can easily manage simple `DAGs` for both `sql_script_1.sql` and `sql_script_2.sql` by including the configurable header: 

```sql

--{"schedule_interval": "@weekly", "author": "Zachary Manesiotis", "catchup": true, "destination_table": "project.dataset.table"} 

with fake_sql as (

    select col 

      from table

)

...

```

in each of the `SQL` files.