{"id":17047836,"url":"https://github.com/residentmario/airflow-tests","last_synced_at":"2025-03-23T03:44:57.373Z","repository":{"id":66134234,"uuid":"95931982","full_name":"ResidentMario/airflow-tests","owner":"ResidentMario","description":null,"archived":false,"fork":false,"pushed_at":"2017-07-02T00:45:17.000Z","size":3,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-28T10:26:38.871Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ResidentMario.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-01T00:07:57.000Z","updated_at":"2017-07-01T00:08:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"42bdc7f9-5be7-4e30-8016-8df7f8a2d144","html_url":"https://github.com/ResidentMario/airflow-tests","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fairflow-tests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fairflow-tests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fairflow-tests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fairflow-tests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ResidentMario","download_url":"https://codeload.github.com/ResidentMario/airflow-tests/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245052636,"owners_count":20553163,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T09:50:26.083Z","updated_at":"2025-03-23T03:44:57.354Z","avatar_url":"https://github.com/ResidentMario.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"* `airflow` DAGs are stored in a single centrally located folder, defined by the `airflow.cfg` file in the \n`~/airflow/` HOME folder and the `AIRFLOW_HOME` Linux environment variable (in some combination?).\n* When `airflow` is initially installed, this is configured to be `~/airflow/dags/`.\n* (In `datablocks`, I will want to write a run hook in the package `__init__.py` that moves the folder reference to \nthe currently installed folder.)\n* `airflow list_dags` is a command that lists known DAGs. It initially contains many tutorial DAGs that are included \nwith the package on install.\n* `airflow list_tasks {{ filename }}` lists tasks in the chosen DAG.\n* Adding the `--tree` operator lists tasks in the shape of a tree. In this output, the indentation reflects the \ndependence of endpoint tasks on tasks that come before them.\n* By default, `airflow` uses a sequential execution engine. It can be configured to use...well, there are a lot of \noptions. To get parallel execution, you will need to configure for extras.\n* To test-run a DAG, `airflow test {{ dag_id }} {{ task_id}} date`, where date is a datetime that you want to run \nthe task on. e.g. `airflow test tutorial print_date 2015-06-01`. Can also be e.g. `2017-1-23T10:34`. \n* FYI, there is no way to delete a DAG directly. Instead, you have to use a custom script. See [here](https://stackoverflow.com/questions/40651783/airflow-how-to-delete-a-dag).\n* Due to [this bug](https://issues.apache.org/jira/browse/AIRFLOW-1165), the management server is broken right now \nwhen `airflow` is deployed in localhost (default) mode.\n  * Deploy it with `airflow webserver --debug`. \n  *`\u0026`  (backgrounding) is helpful.\n  * Note that when you `KeyboardInterrupt` this, the socket connection is *not* closed in the OS (explanation [here](https://stackoverflow.com/questions/4465959/python-errno-98-address-already-in-use)).\n  To free the port, you need to kill the leftover process. Easiest way is by doing `sudo fuser -k 8080/tcp`.\n  * This bug is fixed in master, RC `1.9.0`. But I am using the latest release, `1.8.0`...\n  * The workaround is to generate and self-sign an SSL certificate private/public key pair, and then associate that \n  with `airflow` webserver via [these instructions](http://airflow.readthedocs.io/en/latest/security.html#ssl). cf. [here](https://issues.apache.org/jira/browse/AIRFLOW-1165). \n  * However, this results in a different error: `SSL Wrong Version Number`. The site ([link](http://localhost:8080/)) \n  still doesn't work.\n* `airflow backfill {{ dag_id}} -s {{ start date }} [ -e {{end date}} ]` runs the entire DAG on a range of simulated \ndates. The end date is optional.\n* `airflow` uses `sqlite` as its backend by default, with a `SequentialExecutor` running scheduled jobs. With more \nconfiguration, you can get something that runs in parallel.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresidentmario%2Fairflow-tests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fresidentmario%2Fairflow-tests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresidentmario%2Fairflow-tests/lists"}