{"id":15950030,"url":"https://github.com/casassg/corrent","last_synced_at":"2025-04-04T04:43:51.402Z","repository":{"id":91091561,"uuid":"230493585","full_name":"casassg/corrent","owner":"casassg","description":"Corrent: Experimental Airflow functional DAG API ","archived":false,"fork":false,"pushed_at":"2019-12-29T01:38:01.000Z","size":191,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-09T16:14:06.606Z","etag":null,"topics":["airflow","airflow-dags","apache-airflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/casassg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-27T18:15:49.000Z","updated_at":"2022-02-22T21:19:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"82afd840-d1dd-4076-bbaf-2abce8510e12","html_url":"https://github.com/casassg/corrent","commit_stats":{"total_commits":14,"total_committers":2,"mean_commits":7.0,"dds":0.2142857142857143,"last_synced_commit":"526a3c9b3eff2bdaca617aa54f2774bbb6178670"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casassg%2Fcorrent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casassg%2Fcorrent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casassg%2Fcorrent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casassg%2Fcorrent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/casassg","download_url":"https://codeload.github.com/casassg/corrent/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247123097,"owners_count":20887259,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-dags","apache-airflow"],"created_at":"2024-10-07T12:57:45.727Z","updated_at":"2025-04-04T04:43:46.392Z","avatar_url":"https://github.com/casassg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Corrent: Functional Airflow\n\nCorrent is an Airflow functional DAG API. \n\n## Purpose\n\nThis is an experiment of mine to come up with an alternative API that make Data pipelines on Airflow easier by abstracting away some XCom operations and PythonOperator operations. \n\n**Fun fact**: Less than 200 lines of code under _[corrent](corrent)_ folder, take a peak 👀\n\n## Next steps\n\n- Add unit tests\n- Create DAG using a more complex operator together with the `operation` operators and several xcom arguments.\n- Add template compatibility on `XComArg` (understand better how to do it)\n- Explore making an abstraction on top of PySpark that caches, and passes references to the stored datasets in the same session (needs to dig deeper)\n\n## Features\n- Functional extension for Airflow DAGs.\n- Compatible with existing Airflow DAGs.\n- Adds implicit XCom data dependencies.\n- Simplify XCom pulls needed for `pre_execute` or `execute`.\n- Simple `PythonOperator` creation using `operation` decorators on functions.\n- Copy operators generated using `operation` decorators with `copy_operation` helper function.\n\n## Example\n\nHere's an example DAG with Corrent Functional API. This will compile on Airflow and will discover implicit dependencies. Make sure to have `corrent` installed or on your `PYTHONPATH` when starting Airflow's webserver, scheduler and workers. \n\n```python\nfrom airflow import DAG\n\nfrom corrent.decorators import operation, copy_operation\nfrom corrent.core import inject\n\ninject()\n\n\n\n\nwith DAG(\"corrent_test\", schedule_interval=None) as dag:\n  @operation\n  def generate_list(length: int = 5) -\u003e List[int]:\n    return list(range(length))\n\n\n  @operation\n  def add_one_list(int_list: List[int]) -\u003e List[int]:\n    return [i + 1 for i in int_list]\n\n\n  @operation\n  def print_result(result: List[int]) -\u003e None:\n    print(result)\n\n  l = generate_list(10)\n\n  print_generated = print_result.copy('print_generated')\n  print_generated(l)\n\n  l1 = add_one_list(l)\n\n  print_result(l1)\n\n  print_result \u003c\u003c print_generated\n```\n\n[View example DAGs here](dags/)\n\n### DAG on Airflow UI\n\n![Maintainability](docs/images/corrent_test_dag.png)\n\n## Development environment\nRequirements: `python3.7`, `pipenv`\n\n1. `pipenv install`\n2. `pipenv shell`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcasassg%2Fcorrent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcasassg%2Fcorrent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcasassg%2Fcorrent/lists"}