{"id":15346506,"url":"https://github.com/kaxil/airflowctl","last_synced_at":"2025-04-08T10:15:53.001Z","repository":{"id":188248395,"uuid":"677924462","full_name":"kaxil/airflowctl","owner":"kaxil","description":"A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects","archived":false,"fork":false,"pushed_at":"2025-03-24T19:20:45.000Z","size":325,"stargazers_count":216,"open_issues_count":6,"forks_count":17,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-01T08:42:49.931Z","etag":null,"topics":["airflow","apache-airflow","cli","dags","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kaxil.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-13T05:11:02.000Z","updated_at":"2025-03-26T14:51:22.000Z","dependencies_parsed_at":"2024-01-08T20:46:35.674Z","dependency_job_id":"8660bfcb-2a39-4ccf-954c-e395d14882ed","html_url":"https://github.com/kaxil/airflowctl","commit_stats":{"total_commits":99,"total_committers":4,"mean_commits":24.75,"dds":"0.19191919191919193","last_synced_commit":"df819f4ed359ea71fa7fc7568aa29603fa5b4080"},"previous_names":["kaxil/airflowctl"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaxil%2Fairflowctl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaxil%2Fairflowctl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaxil%2Fairflowctl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaxil%2Fairflowctl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kaxil","download_url":"https://codeload.github.com/kaxil/airflowctl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247819934,"owners_count":21001394,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","apache-airflow","cli","dags","python"],"created_at":"2024-10-01T11:23:50.693Z","updated_at":"2025-04-08T10:15:52.964Z","avatar_url":"https://github.com/kaxil.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# airflowctl\n\n[![PyPI](https://img.shields.io/pypi/v/airflowctl)](https://pypi.org/project/airflowctl/)\n[![License](https://img.shields.io/:license-Apache%202-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0.txt)\n[![Python](https://img.shields.io/pypi/pyversions/airflowctl.svg)](https://pypi.python.org/pypi/airflowctl)\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/airflowctl)](https://pypi.org/project/airflowctl/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/kaxil/airflowctl/main.svg)](https://results.pre-commit.ci/latest/github/kaxil/airflowctl/main)\n\n`airflowctl` is a command-line tool for managing Apache Airflow™ projects.\nIt provides a set of commands to initialize, build, start, stop, and manage Airflow projects.\nWith `airflowctl`, you can easily set up and manage your Airflow projects, install\nspecific versions of Apache Airflow, and manage virtual environments.\n\nThe main goal of `airflowctl` is for first-time Airflow users to install and setup Airflow using a single command and\nfor existing Airflow users to manage multiple Airflow projects with different Airflow versions on the same machine.\n\n## Features\n\n- **Project Initialization with Connections \u0026 Variables:** Initialize a new Airflow project with customizable project name, Apache Airflow\nversion, and Python version. It also allows you to manage Airflow connections and variables.\n- **Automatic Virtual Environment Management:** Automatically create and manage virtual environments for\nyour Airflow projects, even for Python versions that are not installed on your system.\n- **Airflow Version Management:** Install and manage specific versions of Apache Airflow.\n- **Background Process Management:** Start and stop Airflow in the background with process management capabilities.\n- **Live Logs Display:** Continuously display live logs of background Airflow processes with optional log filtering.\n\n## Table of Contents\n\n- [Installation](#installation)\n- [Quickstart](#quickstart)\n- [Usage](#usage)\n  - [Step 1: Initialize a New Project](#step-1-initialize-a-new-project)\n  - [Step 2: Build the Project](#step-2-build-the-project)\n  - [Step 3: Start Airflow](#step-3-start-airflow)\n  - [Step 4: Monitor Logs](#step-4-monitor-logs)\n  - [Step 5: Stop Airflow](#step-5-stop-airflow)\n  - [Step 6: List Airflow Projects](#step-6-list-airflow-projects)\n  - [Step 7: Show Project Info](#step-7-show-project-info)\n  - [Step 8: Running Airflow Commands](#step-8-running-airflow-commands)\n  - [Step 9: Changing Airflow configuration](#step-9-changing-airflow-configurations)\n- [Using with other Airflow tools](#using-with-other-airflow-tools)\n  - [Astro CLI](#astro-cli)\n\n## Installation\n\n```bash\npip install airflowctl\n```\n\n## Quickstart\n\nTo initialize a new Airflow project with the latest airflow version, build a Virtual environment\nand run the project, run the following command:\n\n```shell\nairflowctl init my_airflow_project --build-start\n```\n\nThis will start Airflow and display the logs in the terminal. You can\naccess the Airflow UI at http://localhost:8080. To stop Airflow, press `Ctrl+C`.\n\n## Usage\n\n### Step 1: Initialize a New Project\n\nTo create a new Apache Airflow project, use the init command.\nThis command sets up the basic project structure, including configuration files,\ndirectories, and sample DAGs.\n\n\n```shell\nairflowctl init \u003cproject_name\u003e --airflow-version \u003cversion\u003e --python-version \u003cversion\u003e\n```\n\nExample:\n\n```shell\nairflowctl init my_airflow_project --airflow-version 2.6.3 --python-version 3.8\n```\n\nThis creates a new project directory with the following structure:\n\n```shell\nmy_airflow_project\n├── .env\n├── .gitignore\n├── dags\n│   └── example_dag_basic.py\n├── plugins\n├── requirements.txt\n└── settings.yaml\n```\n\nDescription of the files and directories:\n- `.env` file contains the environment variables for the project.\n- `.gitignore` file contains the default gitignore settings.\n- `dags` directory contains the sample DAGs.\n- `plugins` directory contains the sample plugins.\n- `requirements.txt` file contains the project dependencies.\n- `settings.yaml` file contains the project settings, including the project name,\nAirflow version, Python version, and virtual environment path.\n\nIn our example `settings.yaml` file would look like this:\n\n```yaml\n# Airflow version to be installed\nairflow_version: \"2.6.3\"\n\n# Python version for the project\npython_version: \"3.8\"\n\n\n# Path to a virtual environment to be used for the project\nmode:\n    name: \"uv\"\n    config:\n      venv_path: \"PROJECT_DIR/.venv\"\n\n# Airflow connections\nconnections:\n    # Example connection\n    # - conn_id: example\n    #   conn_type: http\n    #   host: http://example.com\n    #   port: 80\n    #   login: user\n    #   password: pass\n    #   schema: http\n    #   extra:\n    #      example_extra_field: example-value\n\n# Airflow variables\nvariables:\n    # Example variable\n    # - key: example\n    #   value: example-value\n    #   description: example-description\n```\n\nEdit the `settings.yaml` file to customize the project settings.\n\n### Step 2: Build the Project\n\nThe build command creates the virtual environment, installs the specified Apache Airflow\nversion, and sets up the project dependencies.\n\nRun the build command from the project directory:\n\n```shell\ncd my_airflow_project\nairflowctl build\n```\n\nThe CLI relies on one of [`uv`](https://docs.astral.sh/uv/) or [`pyenv`](https://github.com/pyenv/pyenv) to download and install a Python version if the version is not already installed.\n\nExample, if you have Python 3.8 installed but you specify Python 3.7 in the `settings.yaml` file,\nthe CLI will install Python 3.7 using `uv` or `pyenv` and create a virtual environment with Python 3.7 first.\n\nOptionally, you can choose custom virtual environment path in case you have already installed apache-airflow package\nand other dependencies.\nPass the existing virtualenv path using `--venv_path` option to the `init` command or in `settings.yaml` file.\nMake sure the existing virtualenv has same airflow and python version as your `settings.yaml` file states.\n\n### Step 3: Start Airflow\n\nTo start Airflow services, use the start command.\nThis command activates the virtual environment and launches the Airflow web server and scheduler.\n\nExample:\n\n```shell\nairflowctl start my_airflow_project\n```\n\nYou can also start Airflow in the background with the `--background` flag:\n\n```shell\nairflowctl start my_airflow_project --background\n```\n\n### Step 4: Monitor Logs\n\nTo monitor logs from the background Airflow processes, use the logs command.\nThis command displays live logs and provides options to filter logs for specific components.\n\nExample\n```shell\nairflowctl logs my_airflow_project\n```\n\nTo filter logs for specific components:\n\n```shell\n# Filter logs for scheduler\nairflowctl logs my_airflow_project -s\n\n# Filter logs for webserver\nairflowctl logs my_airflow_project -w\n\n# Filter logs for triggerer\nairflowctl logs my_airflow_project -t\n\n# Filter logs for scheduler and webserver\nairflowctl logs my_airflow_project -s -w\n```\n\n### Step 5: Stop Airflow\n\nTo stop Airflow services if they are still running, use the stop command.\n\nExample:\n\n```shell\nairflowctl stop my_airflow_project\n```\n\n### Step 6: List Airflow Projects\n\nTo list all Airflow projects, use the list command.\n\nExample:\n\n```shell\nairflowctl list\n```\n\n### Step 7: Show Project Info\n\nTo show project info, use the info command.\n\nExample:\n\n```shell\n# From the project directory\nairflowctl info\n\n# From outside the project directory\nairflowctl info my_airflow_project\n```\n\n### Step 8: Running Airflow commands\n\nTo run Airflow commands, use the `airflowctl airflow` command. All the commands after\n`airflowctl airflow` are passed to the Airflow CLI.:\n\n```shell\n# From the project directory\nairflowctl airflow \u003cairflow_command\u003e\n```\n\nExample:\n\n```shell\n$ airflowctl airflow version\n2.6.3\n```\n\nYou can also run `airflowctl airflow --help` to see the list of available commands.\n\n```shell\n$ airflowctl airflow --help\nUsage: airflowctl airflow [OPTIONS] COMMAND [ARGS]...\n\n  Run Airflow commands.\n\nPositional Arguments:\n  GROUP_OR_COMMAND\n\n    Groups:\n      celery         Celery components\n      config         View configuration\n      connections    Manage connections\n      dags           Manage DAGs\n      db             Database operations\n      jobs           Manage jobs\n      kubernetes     Tools to help run the KubernetesExecutor\n      pools          Manage pools\n      providers      Display providers\n      roles          Manage roles\n      tasks          Manage tasks\n      users          Manage users\n      variables      Manage variables\n\n    Commands:\n      cheat-sheet    Display cheat sheet\n      dag-processor  Start a standalone Dag Processor instance\n      info           Show information about current Airflow and environment\n      kerberos       Start a kerberos ticket renewer\n      plugins        Dump information about loaded plugins\n      rotate-fernet-key\n                     Rotate encrypted connection credentials and variables\n      scheduler      Start a scheduler instance\n      standalone     Run an all-in-one copy of Airflow\n      sync-perm      Update permissions for existing roles and optionally DAGs\n      triggerer      Start a triggerer instance\n      version        Show the version\n      webserver      Start a Airflow webserver instance\n\nOptions:\n  -h, --help         show this help message and exit\n```\n\nExample:\n\n```shell\n# Listing dags\n$ airflowctl airflow dags list\ndag_id            | filepath             | owner   | paused\n==================+======================+=========+=======\nexample_dag_basic | example_dag_basic.py | airflow | True\n\n\n# Running standalone\n$ airflowctl airflow standalone\n```\n\nOr you can activate the virtual environment first and then run the commands as shown below.\n\nExample:\n\n```shell\n# From the project directory\nsource .venv/bin/activate\n\n# Source all the environment variables\nsource .env\nairflow version\n```\n\nTo add a new DAG, add the DAG file to the `dags` directory.\n\nTo edit an existing DAG, edit the DAG file in the `dags` directory.\nThe changes will be reflected in the Airflow web server.\n\n### Step 9: Changing Airflow Configurations\n\n`airflowctl` by default uses SQLite as the backend database and `SequentialExecutor` as the executor.\nHowever, if you want to use other databases or executors, you can stop the project and\neither a) edit the `airflow.cfg` file or b) add environment variables to the `.env` file.\n\nExample:\n\n```shell\n# Stop the project\nairflowctl stop my_airflow_project\n\n# Changing the executor to LocalExecutor\n# Change the database to PostgreSQL if you already have it installed\necho \"AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@localhost:5432/airflow\" \u003e\u003e .env\necho \"AIRFLOW__CORE__EXECUTOR=LocalExecutor\" \u003e\u003e .env\n\n# Start the project\n\nairflowctl start my_airflow_project\n```\n\nCheck the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html)\nfor all the available Airflow configurations.\n\n#### Using Local Executor with SQLite\nFor Airflow \u003e= 2.6, you can run `LocalExecutor` with `sqlite` as the backend database by\nadding the following environment variable to the `.env` file:\n\n```shell\n_AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK=1\nAIRFLOW__CORE__EXECUTOR=LocalExecutor\n```\n\n\u003e [!WARNING]\n\u003e Sqlite is not recommended for production use. Use it only for development and testing only.\n\n### Other Commands\n\nFor more information and options, you can use the `--help` flag with each command.\n\n## Using with other Airflow tools\n\n`airflowctl` can be used with other Airflow projects as long as the project structure is the same.\n\n### Astro CLI\n\n`airflowctl` can be used with [Astro CLI](https://github.com/astronomer/astro-cli) projects too.\n\nWhile `airflowctl` is a tool that allows you to run Airflow locally using virtual environments, Astro CLI\nallows you to run Airflow locally using docker.\n\n`airflowctl` can read the `airflow_settings.yaml` file generated by Astro CLI for reading connections \u0026 variables. It\nwill then reuse it as `settings` file for `airflowctl`.\n\nFor example, if you have an Astro CLI project:\n- Run the `airflowctl init . --build-start` command to initialize `airflowctl` from the project directory.\nPress `y` to continue when prompted.\n- It will then ask you for the Airflow version, enter the version you are using, by\ndefault uses the latest Airflow version, press enter to continue\n- It will use the installed Python version as the project's python version. If you\nwant to use a different Python version, you can specify it in the `airflow_settings.yaml` file in the\n`python_version` field.\n\n```shell\n# From the project directory\n$ cd astro_project\n$ airflowctl init . --build-start\nDirectory /Users/xyz/astro_project is not empty. Continue? [y/N]: y\nProject /Users/xyz/astro_project added to tracking.\nAirflow project initialized in /Users/xyz/astro_project\nDetected Astro project. Using Astro settings file (/Users/kaxilnaik/Desktop/proj1/astro_project/airflow_settings.yaml).\n'airflow_version' not found in airflow_settings.yaml file. What is the Airflow version? [2.6.3]:\nVirtual environment created at /Users/xyz/astro_project/.venv\n...\n...\n```\n\nIf you see an error like the following, remove `airflow.cfg` file from the project directory and remove\n`AIRFLOW_HOME` from `.env` file if it exists and try again.\n\n```shell\nError: there might be a problem with your project starting up.\nThe webserver health check timed out after 1m0s but your project will continue trying to start.\nRun 'astro dev logs --webserver | --scheduler' for details.\n```\n\n## License\n\nThis project is licensed under the terms of the [Apache 2.0 License](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaxil%2Fairflowctl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkaxil%2Fairflowctl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaxil%2Fairflowctl/lists"}