https://github.com/lu0/git-worktree-airflow
Solves Airflow DAG management in bare Git repos, loading correct DAGs upon branch switch.
https://github.com/lu0/git-worktree-airflow
airflow bare-repo bare-repository git git-hooks git-workflow git-worktree hacktoberfest pre-commit pre-commit-hook
Last synced: 2 months ago
JSON representation
Solves Airflow DAG management in bare Git repos, loading correct DAGs upon branch switch.
- Host: GitHub
- URL: https://github.com/lu0/git-worktree-airflow
- Owner: lu0
- License: gpl-3.0
- Created: 2022-03-27T05:31:14.000Z (about 3 years ago)
- Default Branch: master
- Last Pushed: 2024-02-18T18:40:08.000Z (over 1 year ago)
- Last Synced: 2025-01-24T11:13:27.266Z (4 months ago)
- Topics: airflow, bare-repo, bare-repository, git, git-hooks, git-workflow, git-worktree, hacktoberfest, pre-commit, pre-commit-hook
- Language: Shell
- Homepage:
- Size: 67.4 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
`git-worktree-airflow`
---
This repository contains a post-checkout hook script useful to manage Airflow
`dags_folder`s pointing to bare repositories.
Called after a successful `git checkout`, this hook script
creates a file `.airflowignore` in the root directory of a bare repository,
listing all files and directories except for the worktree directory of the last
*checked out* tree-ish (branch, tag or commit); then the **Airflow UI will show
only the DAGs contained in this directory**.
Table of Contents
---
- [Installation](#installation)
- [Using `pre-commit`](#using-pre-commit)
- [Manual](#manual)
- [Usage](#usage)
- [Recommended option: Using `git-worktree-wrapper`](#recommended-option-using-git-worktree-wrapper)
- [Alternative option: Using vanilla `git`](#alternative-option-using-vanilla-git)
- [Examples](#examples)
# Installation
## Using [`pre-commit`](https://pre-commit.com)
Activate this post-checkout hook by adding it to your `.pre-commit-config.yaml`:
```sh
repos:
- repo: https://github.com/lu0/git-worktree-airflow
rev: v1.1.3
hooks:
- id: airflow-worktree
name: Update .airflowignore to load DAGs from worktree
stages: [post-checkout]
always_run: true
verbose: true
```
And installing it:
```sh
pre-commit install --hook-type post-checkout
```
## Manual
- Copy or link the script `select-airflow-worktree.sh` into the `hooks`
directory of your `dags_folder` (which should be a bare repository), and rename
it to `post-checkout`.
- Example:
ln -srf select-airflow-worktree.sh /path/to/your/dags_folder/hooks/post-checkout
*Note*: Make the script executable with `chmod +x post-checkout` if you
***copied** the script instead of linking it.
# Usage
## Recommended option: Using [`git-worktree-wrapper`](https://github.com/lu0/git-worktree-wrapper)
- Install [`git-worktree-wrapper`](https://github.com/lu0/git-worktree-wrapper).
- Checkout into a tree-ish object. The hook will be triggered automatically.
```sh
$ git checkout
.airflowignore updated to load DAGs from
```
## Alternative option: Using vanilla `git`
1. First ***cd*** into the worktree directory of a tree-ish
```language
$ cd /path/to/the/root/directory/of/the/bare/repo
$ cd tree-ish
```
1. Then trigger the hook
```sh
$ git checkout
.airflowignore updated to load DAGs from
```
## Examples
Let's say we have a `dags` folder pointing to a bare repository in `~/dags` with the following structure:
*Note: Directories and files common to bare repositories are hidden.*
```sh
. dags
├── development
│ ├── dag_1.py
│ └── dag_2.py
├── feature
│ └── dag_4
│ ├── dag_1.py
│ ├── dag_2.py
│ ├── dag_3.py
│ ├── dag_4.py
│ └── examples
│ ├── example_dag_1.py
│ └── example_dag_2.py
└── master
├── dag_1.py
├── dag_2.py
├── dag_3.py
└── examples
├── example_dag_1.py
└── example_dag_2.py
```
And you want the Airflow UI to show the DAGs contained in the `feature/dag_4` worktree.
- If you are using
[`git-worktree-wrapper`](https://github.com/lu0/git-worktree-wrapper) and
[`pre-commit`](#using-pre-commit), just checkout into the tree-ish. You can
checkout from any nested worktree.
```sh
$ git checkout feature/dag_4
```
- If you are using vanilla `git`:
```language
$ cd ~/dags
$ cd feature/dag_4
$ git checkout feature/dag_4
.airflowignore updated to load DAGs from feature/dag_4
```
Either way, the post-checkout hook will create a file named `.airflowignore` with the following contents:
*Note: Directories and files common to bare repositories are hidden.*
```sh
development
master
```
If you trigger the hook in tree-ish `development`, `.airflowignore` will look like this:
*Note: Directories and files common to bare repositories are hidden.*
```sh
feature
master
```