https://github.com/mcejp/goeiedag
It's like a Makefile, but you define it in Python
https://github.com/mcejp/goeiedag
build-system dag make makefile ninja ninja-build
Last synced: 5 months ago
JSON representation
It's like a Makefile, but you define it in Python
- Host: GitHub
- URL: https://github.com/mcejp/goeiedag
- Owner: mcejp
- License: mit
- Created: 2022-04-05T19:39:18.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2024-07-07T12:57:30.000Z (about 2 years ago)
- Last Synced: 2025-02-28T17:15:26.576Z (over 1 year ago)
- Topics: build-system, dag, make, makefile, ninja, ninja-build
- Language: Python
- Homepage: https://goeiedag.readthedocs.io/
- Size: 2.83 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# goeieDAG
_/ɣu.jə.ˈdɑx/: hello, good day (Dutch greeting used during daytime)_
goeieDAG provides a unified Python API to Ninja and Make (_TODO_) build systems, aiming to
make it extremely easy to benefit from parallel processing in any graph-like workflow.
## Installation
pip install goeieDAG==0.0.2
## Usage
```python
from pathlib import Path
import goeiedag
from goeiedag import ALL_INPUTS, INPUT, OUTPUT
workdir = Path("output")
graph = goeiedag.Graph()
# Extract OS name from /etc/os-release
graph.add(["grep", "^NAME=", INPUT, ">", OUTPUT],
inputs=["/etc/os-release"],
outputs=["os-name.txt"])
# Get username
graph.add(["whoami", ">", OUTPUT],
inputs=[],
outputs=["username.txt"])
# Glue together to produce output
graph.add(["cat", ALL_INPUTS, ">", OUTPUT.result],
inputs=["os-name.txt", "username.txt"],
outputs=dict(result="result.txt")) # can also use a dictionary and refer to inputs/outputs by name
goeiedag.build_all(graph, workdir)
# Print output
print((workdir / "result.txt").read_text())
```
## Q&A
### Why use the _files and commands_ model rather than _Python objects and functions_?
- It is a tested and proven paradigm (`make` traces back to _1976_!)
- It provides an obvious way of evaluating which products need rebuilding (subject to an
accurate dependency graph)
- It naturally isolates and parallelizes individual build tasks
- It is agnostic as to how data objects are serialized (convenient for the library author...)
- Graph edges are implicitly defined by input/output file names
- A high-quality executor (Ninja) is available and installable via a Python package
### How is this different from using the Ninja package directly?
- Simpler mental model & usage: no need to separately define build rules or think about implicit/explicit inputs and
outputs
- API accepts Paths; no need to cast everything to `str`!
- Higher-level API in general (for example, the output directory is created automatically)
### Why is Python 3.10+ required?
The code makes use of [PEP-604](https://peps.python.org/pep-0604/).
## Similar projects
- [Ninja](https://pypi.org/project/ninja/) (Python package) -- provides a lower-level API,
used by goeieDAG as back-end
- [TaskGraph](https://github.com/natcap/taskgraph/) -- similar project, but centered around
Python functions and in-process parallelism
- [Snakemake](https://snakemake.github.io/) -- similar goals, but a stand-alone tool rather
than a library
- [Dask](https://dask.org/) -- different execution model; caching of intermediate results
is left up to the user
- [doit](https://pydoit.org/)