Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/chezou/cookiecutter-digdag
A template generates digdag workflows for SQL and Python
https://github.com/chezou/cookiecutter-digdag
Last synced: about 2 months ago
JSON representation
A template generates digdag workflows for SQL and Python
- Host: GitHub
- URL: https://github.com/chezou/cookiecutter-digdag
- Owner: chezou
- License: apache-2.0
- Created: 2019-11-14T02:33:06.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2021-07-06T03:25:22.000Z (over 3 years ago)
- Last Synced: 2024-10-16T10:33:21.015Z (3 months ago)
- Language: Python
- Size: 8.79 KB
- Stars: 2
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Cookiecutter Digdag
A cookiecutter for digdag project, including py> operator, and td> operator.
## Requirements to use the cookiecutter template:
- Python 3.5 or later
- [Cookiecutter Python package](http://cookiecutter.readthedocs.org/en/latest/installation.html)
```sh
$ pip install --user cookiecutter
```
## To start a new project, run:
```sh
cookiecutter https://github.com/chezou/cookiecutter-digdag
```
## The output directory structure
```text
my_project
├── README.md
├── config
│ ├── params.test.yml <- Configuration file for run through test. Mirror params.yml except for `td.database`
│ └── params.yml <- Configuration file for production
├── awesome_workflow.dig <- Main workflow to be executed
├── ingest.dig <- Data ingestion workflow
├── py_scripts <- Python scripts directory
│ ├── __init__.py
│ ├── data.py <- Script to upload data to Arm Treasure Data
│ └── my_script.py <- Main script to execute e.g. Data enrichment, ML training
├── queries <- SQL directory
├── run_test.sh <- Test shell script for local run through test
└── test.dig <- Test workflow for local run through test
```