https://github.com/shuyib/just-getting-started-airflow
Examples from Data pipelines with Apache Airflow book. https://www.manning.com/books/data-pipelines-with-apache-airflow
https://github.com/shuyib/just-getting-started-airflow
data-pipeline dev-ops
Last synced: 4 months ago
JSON representation
Examples from Data pipelines with Apache Airflow book. https://www.manning.com/books/data-pipelines-with-apache-airflow
- Host: GitHub
- URL: https://github.com/shuyib/just-getting-started-airflow
- Owner: Shuyib
- License: cc0-1.0
- Created: 2022-06-12T16:09:58.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-05-08T20:32:43.000Z (almost 3 years ago)
- Last Synced: 2025-03-22T09:43:39.027Z (11 months ago)
- Topics: data-pipeline, dev-ops
- Language: Python
- Homepage:
- Size: 7.81 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
This is a practice repo where I run production ready airflow projects at least according to me.
1. Create a virtual environment
```bash
python -m venv .venv
```
2. Activate virtual environment
```bash
source .venv/bin/activate
```
3. Update and install dependecies
```bash
pip install --upgrade pip && pip install -r requirements.txt
```
4. Running locally. It has loads of requirements.
`airflow db init` - long output with Initialization done at the end.
`airflow users create --username admin --password admin firstname Anonymous --lastname Admin --role Admin --email admin@example.org`
`airflow users create \
--username admin \
--firstname FIRST_NAME \
--lastname LAST_NAME \
--role Admin \
--email admin@example.org
`
Prompt for the password. You can add the one above. The last two are continuous processes therefore you need the terminal to be open at runtime.
`airflow webserver`
`airflow scheduler`
- Copy the `*.py` files that you want scheduled to the ~/airflow/dags/
- Start up the webserver
`airflow webserver`
- Start up the scheduler & trigger the dag to run using the UI
`airflow scheduler`
5. Save time and run in a Makefile but remember to add the .venv to your bash profile.
```bash
make build run
```