{"id":24782256,"url":"https://github.com/dev-vivekkumarverma/airflow","last_synced_at":"2025-03-24T05:28:27.928Z","repository":{"id":274459727,"uuid":"922974029","full_name":"dev-vivekkumarverma/airflow","owner":"dev-vivekkumarverma","description":"hey, I am learning Apache Airflow","archived":false,"fork":false,"pushed_at":"2025-01-27T12:38:29.000Z","size":1,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-27T13:46:18.852Z","etag":null,"topics":["apache-airflow","python3"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dev-vivekkumarverma.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-27T12:31:21.000Z","updated_at":"2025-01-27T12:38:33.000Z","dependencies_parsed_at":"2025-01-27T13:57:58.751Z","dependency_job_id":null,"html_url":"https://github.com/dev-vivekkumarverma/airflow","commit_stats":null,"previous_names":["dev-vivekkumarverma/airflow"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-vivekkumarverma%2Fairflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-vivekkumarverma%2Fairflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-vivekkumarverma%2Fairflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-vivekkumarverma%2Fairflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dev-vivekkumarverma","download_url":"https://codeload.github.com/dev-vivekkumarverma/airflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245214807,"owners_count":20578859,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-airflow","python3"],"created_at":"2025-01-29T11:16:38.663Z","updated_at":"2025-03-24T05:28:27.909Z","avatar_url":"https://github.com/dev-vivekkumarverma.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003c!-- ![alt airflow image](./img_src/airflowlogo.png) --\u003e\n\u003cimg src=\"./img_src/airflowlogo.png\" height='250 rem' width='300 rem'\u003e\u003c/img\u003e \n# airflow\n\u003cbr\u003e\nFor tutorial: \n\nhttps://www.youtube.com/watch?v=K9AnJ9_ZAXE\u0026list=PLwFJcsJ61oujAqYpMp1kdUBcPG0sE0QMT\n\n\n1.  What is `airflow` and why do we `need` it ?\n    -   Airflow is a `workflow orchestration platform` that allows users to programmatically create, schedule, and monitor workflows.\n            It's often used to automate machine learning tasks and create complex `data pipelines`. \n\n\u003c!-- \n## Installation on local environment\n1.  Specify the AIRFLOW_HOME dir \n\u003cbr\u003e\nFor default behaviour  use \n\n```sh\nexport AIRFLOW_HOME=~/airflow\n```\nfor using airflow in current directory use\n\n```sh\nexport AIRFLOW_HOME=$(pwd)\n```\n\n\n2.  try method A or B\n\u003cbr\u003e\n`method A`\n\n    ```sh\n    AIRFLOW_VERSION=2.10.4\n\n    # Extract the version of Python you have installed. If you're currently using a Python version that is not supported by Airflow, you may want to set this manually.\n    # See above for supported versions.\n    PYTHON_VERSION=\"$(python -c 'import sys; print(f\"{sys.version_info.major}.{sys.version_info.minor}\")')\"\n\n    CONSTRAINT_URL=\"https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt\"\n    # For example this would install 2.10.4 with python 3.8: https://raw.githubusercontent.com/apache/airflow/constraints-2.10.4/constraints-3.8.txt\n\n    pip install \"apache-airflow==${AIRFLOW_VERSION}\" --constraint \"${CONSTRAINT_URL}\"\n\n    ```\nOr, use \n\u003cbr\u003e\n`method B`\n\n```sh\n# this installs the requirement for airflow specified in requirement.txt\npip install -r requirements.txt\n```\n\n\n3. create `airflow.cfg` file and add the file\n```sh\n# add absolute Path to sqlite after //// slashes\n[core]\nsql_alchemy_conn = sqlite:////abs/path/to/project/dir/airflow.db\n# replace abs/path/to/project/dir with absolute project path dir or dir where you want to create an airflow sqlite database\n```\n\n4.  Run Airflow Standalone:\n\n    The airflow standalone command initializes the database, creates a user, and starts all component\n    \n```sh\nairflow standalone\n```\n\n5. use this as cred (as it is shown in terminal after running step 4)\n```sh\nLogin with username: admin  password: 7a4HAYQ2cmwDAHDy\n```\nfor the UI interface goto (default)\n\nhttp://localhost:8080 \n\n\n6. for running the airflow scheduler, run the command in another terminal \n```sh\nairflow sheduler\n``` --\u003e\n\nHere's a well-structured guide for setting up Apache Airflow in a **virtual environment** and using the **current working directory (`.`) as `AIRFLOW_HOME`**.  \n\nI'll also highlight the **commands that should go in `README.md`** for easy reference.  \n\n---\n\n# **🚀 Apache Airflow Local Setup Guide (Using Virtual Environment \u0026 Local Directory)**  \n\n## **📌 Overview**  \nThis guide covers:  \n✅ Installing Airflow inside a **Python virtual environment**  \n✅ Using the **current directory (`.`) as `AIRFLOW_HOME`**  \n✅ Running **Airflow webserver and scheduler**  \n✅ Managing **DAGs and users**  \n\n---\n\n## **🛠️ Prerequisites**  \nEnsure you have:  \n- **Python 3.11** installed  \n- **pip, venv, and other required system packages**  \n- **Enough disk space and proper permissions**  \n\n---\n\n## **1️⃣ Install Dependencies (Before Setup)**\n### ✅ **For Ubuntu/Debian**:\n```bash\nsudo apt update\nsudo apt install python3.11 python3.11-venv python3.11-distutils python3-pip -y\n```\n\n### ✅ **For macOS (Using Homebrew)**:\n```bash\nbrew install python@3.11\n```\n\n### ✅ **For Windows**:\n1. Download **Python 3.11** from [python.org](https://www.python.org/downloads/).\n2. During installation, **check the box**: **\"Add Python to PATH\"**.\n3. Open **PowerShell** and run:\n   ```powershell\n   python -m ensurepip\n   ```\n\n---\n\n## **2️⃣ Set Up Virtual Environment**  \n🚀 **(Add These Commands to README.md)**\n```bash\n# Navigate to your project directory\ncd ~/your-project-folder  # Change this to your actual folder\n\n# Create and activate a virtual environment\npython3.11 -m venv airflow-env\nsource airflow-env/bin/activate  # Linux/macOS\nairflow-env\\Scripts\\activate     # Windows\n\n# Verify Python version inside the virtual environment\npython --version\n```\n\n---\n\n## **3️⃣ Set `AIRFLOW_HOME` to Current Directory (`.`)**\n🚀 **(Add These Commands to README.md)**\n```bash\n# Set Airflow to use the current directory\nexport AIRFLOW_HOME=$(pwd)  # Linux/macOS\nset AIRFLOW_HOME=%cd%       # Windows\n\n# Add this to your ~/.bashrc or ~/.zshrc to make it persistent\necho 'export AIRFLOW_HOME=$(pwd)' \u003e\u003e ~/.bashrc\nsource ~/.bashrc\n```\n\n---\n\n## **4️⃣ Install Apache Airflow**\n🚀 **(Add These Commands to README.md)**\n```bash\npip install --upgrade pip\npip install apache-airflow==2.7.1\n```\n\n✅ **Verify Installation**  \n```bash\nairflow version\n```\n\n---\n\n## **5️⃣ Initialize Airflow Database**\n🚀 **(Add These Commands to README.md)**\n```bash\nairflow db init\n```\nThis will create:  \n- `airflow.cfg` → Airflow configuration file  \n- `airflow.db` → SQLite database (for local use)  \n\n✅ **Check if the files are created in the current directory**:  \n```bash\nls -l | grep airflow\n```\n\n---\n\n## **6️⃣ Start Airflow Webserver and Scheduler**\n🚀 **(Add These Commands to README.md)**\n```bash\n# Start the Airflow web server (Runs on port 8080 by default)\nairflow webserver --port 8080\n\n# Open in browser: http://localhost:8080\n\n# In a separate terminal, start the scheduler\nairflow scheduler\n```\n\n---\n\n## **7️⃣ Create an Admin User**\n🚀 **(Add These Commands to README.md)**\n```bash\nairflow users create \\\n    --username admin \\\n    --firstname Admin \\\n    --lastname User \\\n    --role Admin \\\n    --email admin@example.com\n```\n🛠 **Now, log in to the Airflow UI at** `http://localhost:8080` **with the admin credentials.**\n\n---\n\n## **8️⃣ Add DAGs to `dags/` Directory**\n🚀 **(Add These Commands to README.md)**\n```bash\nmkdir -p dags\n```\n- Place your DAG Python files inside the `dags/` directory.  \n- Example DAG (`dags/example_dag.py`):\n  ```python\n  from airflow import DAG\n  from airflow.operators.dummy import DummyOperator\n  from datetime import datetime\n\n  with DAG('example_dag', start_date=datetime(2024, 1, 1), schedule_interval=\"@daily\") as dag:\n      start = DummyOperator(task_id=\"start\")\n\n  ```\n\n✅ **Activate DAGs in UI**:  \n1. Start the scheduler:  \n   ```bash\n   airflow scheduler\n   ```\n2. Enable the DAG in **Airflow UI (`http://localhost:8080`)**.\n\n---\n\n## **9️⃣ Stop Airflow and Deactivate Virtual Environment**\n🚀 **(Add These Commands to README.md)**\n```bash\n# Stop Airflow (Find and kill processes)\npkill -f \"airflow webserver\"\npkill -f \"airflow scheduler\"\n\n# Deactivate virtual environment\ndeactivate\n```\n\n---\n\n# **🚀 Summary of Commands for README.md**\n```bash\n# Install dependencies\nsudo apt update\nsudo apt install python3.11 python3.11-venv python3.11-distutils python3-pip -y\n\n# Create and activate virtual environment\npython3.11 -m venv airflow-env\nsource airflow-env/bin/activate  # (Linux/macOS)\nairflow-env\\Scripts\\activate     # (Windows)\n\n# Set AIRFLOW_HOME to current directory\nexport AIRFLOW_HOME=$(pwd)\necho 'export AIRFLOW_HOME=$(pwd)' \u003e\u003e ~/.bashrc\nsource ~/.bashrc\n\n# Install Apache Airflow\npip install --upgrade pip\npip install apache-airflow==2.7.1\n\n# Initialize Airflow database\nairflow db init\n\n# Start Airflow webserver and scheduler (run in separate terminals)\nairflow webserver --port 8080\nairflow scheduler\n\n# Create an admin user\nairflow users create \\\n    --username admin \\\n    --firstname Admin \\\n    --lastname User \\\n    --role Admin \\\n    --email admin@example.com\n\n# Create DAGs directory\nmkdir -p dags\n\n# Stop Airflow and deactivate environment\npkill -f \"airflow webserver\"\npkill -f \"airflow scheduler\"\ndeactivate\n```\n---\n\n# WORKFLOW IS SOMETHING LIKE\n\n`WORKFLOW -\u003e DAG(when to do what) -\u003e TASK(what to do) -\u003e OPERATOR (how to do)`\n\n![alt workflow](./img_src/workflow.png)\n\n`DAG, tasks and operators`\n\n![alt task and operators](./img_src/task%20and%20operators.png)\n\n\n`DAG, TASK AND OPERATORs internals`\n![alt datg, tasks, operators internals](./img_src/DagtaskoperatorInternal.png)\n\n\n# `TAST LIFECYCLE`\n![alt TAST LIFECYCLE](./img_src/task%20life%20Cycle.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdev-vivekkumarverma%2Fairflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdev-vivekkumarverma%2Fairflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdev-vivekkumarverma%2Fairflow/lists"}