{"id":13652007,"url":"https://github.com/qubole/afctl","last_synced_at":"2025-05-07T13:05:45.689Z","repository":{"id":49937755,"uuid":"229906165","full_name":"qubole/afctl","owner":"qubole","description":"afctl helps to manage and deploy Apache Airflow projects faster and smoother.","archived":false,"fork":false,"pushed_at":"2022-09-16T18:15:44.000Z","size":78,"stargazers_count":130,"open_issues_count":10,"forks_count":7,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-05-07T13:05:37.832Z","etag":null,"topics":["airflow","cli","deployment","docker","management"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qubole.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-24T08:47:50.000Z","updated_at":"2024-01-31T20:50:45.000Z","dependencies_parsed_at":"2022-09-26T17:10:56.783Z","dependency_job_id":null,"html_url":"https://github.com/qubole/afctl","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubole%2Fafctl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubole%2Fafctl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubole%2Fafctl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qubole%2Fafctl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qubole","download_url":"https://codeload.github.com/qubole/afctl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252883207,"owners_count":21819160,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","cli","deployment","docker","management"],"created_at":"2024-08-02T02:00:54.410Z","updated_at":"2025-05-07T13:05:45.670Z","avatar_url":"https://github.com/qubole.png","language":"Python","readme":"# afctl\n\nThe proposed CLI tool is authored to make creating and deployment of Apache Airflow (https://airflow.apache.org/) projects faster and smoother. \nAs of now, there is no tool out there that can empower the user to create a boilerplate code structure for airflow \nprojects and make development + deployment of projects seamless.\n\n## Requirements\n* Python 3.5+\n* Docker\n\n## Getting Started\n\n### 1. Installation\n\nCreate a new python virtualenv. You can use the following command. \u003cbr /\u003e\n```bash\npython3 -m venv \u003cname\u003e\n```\nActivate your virtualenv\u003cbr/\u003e\n```bash\nsource /path_to_venv/bin/activate\n```\n\n```bash\npip3 install afctl\n```\n\n\n### 2. Initialize a new afctl project. \nThe project is created in your present working directory. Along with this a configuration file with the same name is \ngenerated in **/home/.afctl_configs** directory.\n\n\n```bash\nafctl init \u003cname of the project\u003e\n```\nEg.\n```bash\nafctl init project_demo\n```\n* The following directory structure will be generated\n```bash\n.\n├── deployments\n│   └── project_demo-docker-compose.yml\n├── migrations\n├── plugins\n├── project_demo\n│   ├── commons\n│   └── dags\n├── requirements.txt\n└── tests\n```\n\nIf you already have a git repository and want to turn it into an afctl project.\nRun the following command :-\n```bash\nafctl init .\n```\n\u003cbr\u003e\n\n### 3. Add a new module in the project.\n\n```bash\nafctl generate module -n \u003cname of the module\u003e\n```\n\nThe following directory structure will be generated :\n\n```bash\nafctl generate module -n first_module\nafctl generate module -n second_module\n\n.\n├── deployments\n│   └── project_demo-docker-compose.yml\n├── migrations\n├── plugins\n├── project_demo\n│   ├── commons\n│   └── dags\n│       ├── first_module\n│       └── second_module\n├── requirements.txt\n└── tests\n    ├── first_module\n    └── second_module\n\n```\n\n### 4. Generate dag\n```bash\nafctl generate dag -n \u003cname of dag\u003e -m \u003cname of module\u003e\n```\n\nThe following directory structure will be generate :\n\n```bash\nafctl generate dag -n new -m first_module\n\n.\n├── deployments\n│   └── project_demo-docker-compose.yml\n├── migrations\n├── plugins\n├── project_demo\n│   ├── commons\n│   └── dags\n│       ├── first_module\n│       │   └── new_dag.py\n│       └── second_module\n├── requirements.txt\n└── tests\n    ├── first_module\n    └── second_module\n```\n\nThe dag file will look like this :\n\n```python\nfrom airflow import DAG\nfrom datetime import datetime, timedelta\n\ndefault_args = {\n'owner': 'project_demo',\n# 'depends_on_past': ,\n# 'start_date': ,\n# 'email': ,\n# 'email_on_failure': ,\n# 'email_on_retry': ,\n# 'retries': 0\n\n}\n\ndag = DAG(dag_id='new', default_args=default_args, schedule_interval='@once')\n```\n\n### 5. Deploy project locally\n\nYou can add python packages that will be required by your dags in `requirements.txt`. They will automatically get\ninstalled.\n\n* To deploy your project, run the following command (make sure docker is running) :\n\n```bash\nafctl deploy local\n```\n\nIf you do not want to see the logs, you can run \n```bash\nafctl deploy local -d\n```\nThis will run it in detached mode and won't print the logs on the console.\n\n* You can access your airflow webserver on browser at `localhost:8080`\n\n### 6. Deploy project on production\n\n* Here we will be deploying our project to **Qubole**. Sign up at us.qubole.com.\n* add git-origin and access-token (if want to keep the project as private repo\non Github) to the configs. [See how](#manage-configurations)\n* Push the project once completed to Github.\n* Deploying to Qubole will require adding deployment configurations.\n\n```bash\nafctl config add -d qubole -n \u003cname of deployment\u003e -e \u003cenv\u003e -c \u003ccluster-label\u003e -t \u003cauth-token\u003e\n```\nThis command will modify your config file. You can see your config file with the following command :\n```bash\nafctl config show\n```\n\nFor example - \n```bash\nafctl config add -d qubole -n demo -e https://api.qubole.com -c airflow_1102 -t khd34djs3\n```\n\n* To deploy run the following command\n```bash\nafctl deploy qubole -n \u003cname\u003e\n```\n\n### The following video also contains all the steps of deploying project using afctl - \u003c/br\u003e\nhttps://www.youtube.com/watch?v=A4rcZDGtJME\u0026feature=youtu.be\n\n## Manage configurations\n\nThe configuration file is used for deployment contains the following information.\n```yaml\nglobal:\n-airflow_version:\n-git:\n--origin:\n--access-token:\ndeployment:\n-qubole:\n--local:\n---compose:\n```\n\u003cbr\u003e\n\n* `airflow_version` can be added to the project when you initialize the project.\n```bash\nafctl init \u003cname\u003e -v \u003cversion\u003e\n```\n\n* global configs (airflow_version, origin, access-token) can all be added/ updated with the following command :\n\n```bash\nafctl config global -o \u003cgit-origin\u003e -t \u003caccess-token\u003e -v \u003cairflow_version\u003e\n``` \n\n## Usage\n\nCommands right now supported are\n* init\n* config\n* deploy\n* list\n* generate\n\nTo learn more, run \n```bash\nafctl \u003ccommand\u003e -h\n```\n\u003cbr\u003e\n\n### Caution\nNot yet ported for Windows.\n\n#### Credits\nDocker-compose file : https://github.com/puckel/docker-airflow \n\n","funding_links":[],"categories":["Python","Libraries, Hooks, Utilities"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqubole%2Fafctl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqubole%2Fafctl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqubole%2Fafctl/lists"}