{"id":18655862,"url":"https://github.com/michaelosthege/apache-airflow-flowitems","last_synced_at":"2025-04-11T17:32:04.210Z","repository":{"id":46887081,"uuid":"504556812","full_name":"michaelosthege/apache-airflow-flowitems","owner":"michaelosthege","description":"This package helps to reduce the amount of boilerplate code when creating Airflow DAGs from Python callables.","archived":false,"fork":false,"pushed_at":"2023-03-05T11:57:18.000Z","size":34,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-09T03:36:57.525Z","etag":null,"topics":["airflow-dags","apache-airflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michaelosthege.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-17T14:03:47.000Z","updated_at":"2023-10-30T23:54:39.000Z","dependencies_parsed_at":"2023-01-25T22:01:59.582Z","dependency_job_id":null,"html_url":"https://github.com/michaelosthege/apache-airflow-flowitems","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelosthege%2Fapache-airflow-flowitems","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelosthege%2Fapache-airflow-flowitems/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelosthege%2Fapache-airflow-flowitems/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelosthege%2Fapache-airflow-flowitems/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michaelosthege","download_url":"https://codeload.github.com/michaelosthege/apache-airflow-flowitems/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248449873,"owners_count":21105581,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow-dags","apache-airflow"],"created_at":"2024-11-07T07:20:37.402Z","updated_at":"2025-04-11T17:32:03.793Z","avatar_url":"https://github.com/michaelosthege.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![PyPI version](https://img.shields.io/pypi/v/apache-airflow-flowitems)](https://pypi.org/project/apache-airflow-flowitems)\n[![pipeline](https://github.com/michaelosthege/apache-airflow-flowitems/workflows/tests/badge.svg)](https://github.com/michaelosthege/apache-airflow-flowitems/actions)\n[![coverage](https://codecov.io/gh/michaelosthege/apache-airflow-flowitems/branch/main/graph/badge.svg)](https://codecov.io/gh/michaelosthege/apache-airflow-flowitems)\n\n# `apache-airflow-flowitems`\nThis project helps to reduce the amount of boilerplate code when writing Airflow DAGs with lots of Python callables.\n\nFor example, consider the following DAG:\n\n```mermaid\ngraph LR;\n    all_ready--\u003eget_city;\n    get_city--\u003eget_temperature;\n    get_temperature--\u003esummarize;\n    get_runid--\u003esummarize;\n```\n\nFrom the following Python callables, there are multiple ways to create the above DAG.\nBut what if `weather_reports.py` is owned by another project, so we can't `@task` decorate it?\n\n```python\n# file: weather_reports.py\ndef get_city():\n    return \"London\"\n\ndef get_temperature(city):\n    return {\n        \"London\": 10,\n        \"Paris\": 15,\n    }\n\n\n# file: airflow_weather.py\ndef get_runid(context, **kwargs):\n    kwargs.setdefault(\"city\", \"London\")\n    dagrun = context[\"run_id\"]\n    return f\"{city}_{dagrun}\"\n\ndef summarize(runid, temperature):\n    return {\n        \"report_id\": runid,\n        \"text\": f\"The temperature in {city} is {temperature} °C.\",\n    }\n```\n\nWith `apache-airflow-flowitems`, the DAG definition can be very concise:\n\n```python\nfrom apache_airflow_flowitems import PythonItem\nimport weather_airflow as gluecode\nimport weather_reports as external\n\nwith airflow.DAG(...):\n    t_ready = BashOperator(bash_command=\"...\", task_id=\"all_ready\")\n    # Upstream dependencies can be passed as 👇 args\n    t_city = PythonItem(external.get_city, t_ready)()\n    # Function parameters must be passed as kwargs 👇\n    t_temp = PythonItem(external.get_temperature, city=t_city)(task_id=\"temp_London\")\n\n    t_id = PythonItem(gluecode.get_runid)()\n    t_summary = PythonItem(gluecode.summarize, runid=t_id, temperature=t_temp)\n```\n\nFor comparison, the 2nd task manually wrapped in a `@task` decorator would look like this:\n\n```python\n    t_temp = task(task_id=\"temp_London\")(external.get_temperature)(city=t_city.output)\n    t_ready \u003e\u003e t_temp\n```\n\nOkay, it's not _that_ much shorter, but `FlowItem`s can do more cool stuff:\n* Inputs can be `XComArg` or `Operator`, whereas the `@task` decorator only supports `Operator`\n* If the callable passed to a `PythonItem` has a `context` argument, the Airflow `context` dict will be passed.\n  In contrast to `@task`, this means that the function may ALSO take `**kwargs`.\n\nBut in the end this is, of course, just a slightly different syntax for writing DAGs.\n\nI'm open-sourcing this package mostly to make it easier for myself to use it in different projects.\n\n## Installation\n```bash\npip install apache_airflow_flowitems\n```\n\n## Contributing\nFirst make sure that you can run the tests:\n\n```bash\npip install -r requirements-dev.txt\npytest --cov=./apache_airflow_flowitems --cov-report xml --cov-report term-missing .\n```\n\nAlso set up `pre-commit` for automated code style enforcement:\n\n```bash\npip install pre-commit\npre-commit install\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelosthege%2Fapache-airflow-flowitems","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichaelosthege%2Fapache-airflow-flowitems","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelosthege%2Fapache-airflow-flowitems/lists"}