{"id":37069259,"url":"https://github.com/factorpricingmodel/prefect-yaml","last_synced_at":"2026-01-14T08:00:51.690Z","repository":{"id":64685837,"uuid":"576347749","full_name":"factorpricingmodel/prefect-yaml","owner":"factorpricingmodel","description":"Schedule your tasks with YAML configuration files in Prefect perfectly","archived":false,"fork":false,"pushed_at":"2023-02-24T12:47:30.000Z","size":214,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-23T16:27:14.354Z","etag":null,"topics":["orchestration","prefect","scheduler","yaml-configuration"],"latest_commit_sha":null,"homepage":"https://prefect-yaml.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/factorpricingmodel.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":["factorpricingmodel"]}},"created_at":"2022-12-09T16:08:14.000Z","updated_at":"2024-08-19T11:21:01.000Z","dependencies_parsed_at":"2023-02-10T12:16:28.920Z","dependency_job_id":null,"html_url":"https://github.com/factorpricingmodel/prefect-yaml","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/factorpricingmodel/prefect-yaml","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorpricingmodel%2Fprefect-yaml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorpricingmodel%2Fprefect-yaml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorpricingmodel%2Fprefect-yaml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorpricingmodel%2Fprefect-yaml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/factorpricingmodel","download_url":"https://codeload.github.com/factorpricingmodel/prefect-yaml/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorpricingmodel%2Fprefect-yaml/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28413527,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T05:26:33.345Z","status":"ssl_error","status_checked_at":"2026-01-14T05:21:57.251Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["orchestration","prefect","scheduler","yaml-configuration"],"created_at":"2026-01-14T08:00:35.627Z","updated_at":"2026-01-14T08:00:51.619Z","avatar_url":"https://github.com/factorpricingmodel.png","language":"Python","funding_links":["https://github.com/sponsors/factorpricingmodel"],"categories":[],"sub_categories":[],"readme":"# Prefect YAML\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/factorpricingmodel/prefect-yaml/actions?query=workflow%3ACI\"\u003e\n    \u003cimg src=\"https://github.com/factorpricingmodel/prefect-yaml/actions/workflows/ci.yml/badge.svg\" alt=\"CI Status\" \u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://prefect-yaml.readthedocs.io\"\u003e\n    \u003cimg src=\"https://img.shields.io/readthedocs/prefect-yaml.svg?logo=read-the-docs\u0026logoColor=fff\u0026style=flat-square\" alt=\"Documentation Status\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://codecov.io/gh/factorpricingmodel/prefect-yaml\"\u003e\n    \u003cimg src=\"https://img.shields.io/codecov/c/github/factorpricingmodel/prefect-yaml.svg?logo=codecov\u0026logoColor=fff\u0026style=flat-square\" alt=\"Test coverage percentage\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://python-poetry.org/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/packaging-poetry-299bd7?style=flat-square\u0026logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAA4AAAASCAYAAABrXO8xAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAJJSURBVHgBfZLPa1NBEMe/s7tNXoxW1KJQKaUHkXhQvHgW6UHQQ09CBS/6V3hKc/AP8CqCrUcpmop3Cx48eDB4yEECjVQrlZb80CRN8t6OM/teagVxYZi38+Yz853dJbzoMV3MM8cJUcLMSUKIE8AzQ2PieZzFxEJOHMOgMQQ+dUgSAckNXhapU/NMhDSWLs1B24A8sO1xrN4NECkcAC9ASkiIJc6k5TRiUDPhnyMMdhKc+Zx19l6SgyeW76BEONY9exVQMzKExGKwwPsCzza7KGSSWRWEQhyEaDXp6ZHEr416ygbiKYOd7TEWvvcQIeusHYMJGhTwF9y7sGnSwaWyFAiyoxzqW0PM/RjghPxF2pWReAowTEXnDh0xgcLs8l2YQmOrj3N7ByiqEoH0cARs4u78WgAVkoEDIDoOi3AkcLOHU60RIg5wC4ZuTC7FaHKQm8Hq1fQuSOBvX/sodmNJSB5geaF5CPIkUeecdMxieoRO5jz9bheL6/tXjrwCyX/UYBUcjCaWHljx1xiX6z9xEjkYAzbGVnB8pvLmyXm9ep+W8CmsSHQQY77Zx1zboxAV0w7ybMhQmfqdmmw3nEp1I0Z+FGO6M8LZdoyZnuzzBdjISicKRnpxzI9fPb+0oYXsNdyi+d3h9bm9MWYHFtPeIZfLwzmFDKy1ai3p+PDls1Llz4yyFpferxjnyjJDSEy9CaCx5m2cJPerq6Xm34eTrZt3PqxYO1XOwDYZrFlH1fWnpU38Y9HRze3lj0vOujZcXKuuXm3jP+s3KbZVra7y2EAAAAAASUVORK5CYII=\" alt=\"Poetry\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/ambv/black\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/code%20style-black-000000.svg?style=flat-square\" alt=\"black\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/pre-commit/pre-commit\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white\u0026style=flat-square\" alt=\"pre-commit\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/prefect-yaml/\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/v/prefect-yaml.svg?logo=python\u0026logoColor=fff\u0026style=flat-square\" alt=\"PyPI Version\"\u003e\n  \u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/pypi/pyversions/prefect-yaml.svg?style=flat-square\u0026logo=python\u0026amp;logoColor=fff\" alt=\"Supported Python versions\"\u003e\n  \u003cimg src=\"https://img.shields.io/pypi/l/prefect-yaml.svg?style=flat-square\" alt=\"License\"\u003e\n\u003c/p\u003e\n\nPackage to run prefect with YAML configuration. For further details, please refer\nto the [documentation](https://prefect-yaml.readthedocs.io/en/latest/)\n\n## Installation\n\nInstall this via pip (or your favourite package manager):\n\n`pip install prefect-yaml`\n\n## Usage\n\nRun the command line `prefect-yaml` with the specified configuration\nfile.\n\nFor example, the following YAML configuration is located in [examples/simple_config.yaml](examples/simple_config.yaml).\n\n```\nmetadata:\n  output:\n    directory: .output\n\ntask:\n  task_a:\n    caller: math:fabs\n    parameters:\n      - -9.0\n    output:\n      format: json\n  task_b:\n    caller: math:sqrt\n    parameters:\n      - !data task_a\n    output:\n      directory: null\n  task_c:\n    caller: math:fsum\n    parameters:\n      - [!data task_b, 1]\n```\n\nRun the following command to generate all the task outputs to the\ndirectory `.output` in the running directory.\n\n```shell\nprefect-yaml -c examples/simple_config.yaml\n```\n\nThe output directory contains all the task outputs in the specified\nformat.\n\n```shell\n% tree .output\n.output\n├── task_a.json\n└── task_c.pickle\n\n0 directories, 2 files\n```\n\nThe expected behavior is to\n\n1. run `task_a` to dump the value `fabs(-9.0)` to the output directory in JSON format,\n2. run `task_b` to get the value `sqrt(9.0)` (from the output of `task_a`)\n3. run `task_c` to dump the value `fsum([3.0, 1.0])` to the output directory in pickle format.\n\nAs the output directory in `task_b` is overridden as `null`, the output of `task_b` is passed to `task_c` in memory. Also, the output format in `task_c`\nis not specified so it is dumped in default format (pickle).\n\nFor further details, please see the section [configuration](https://prefect-yaml.readthedocs.io/en/latest/configuration.html) in the documentation.\n\n## Configuration\n\nThe output section defines how the task writes and loads the task return. The section in `metadata` applies for all tasks globally while that in each `task`\noverrides the global parameters.\n\nFor further details, please see the [documentation](https://prefect-yaml.readthedocs.io/en/latest/configuration.html#output) for parameter definitions\nin each section.\n\n## Output\n\nThe default output format is either pickle (default) or JSON, while users\ncan define their own output format.\n\nFor example, if you would like to use `pandas` to load and dump the parquet file\nin pyarrow engine by default, you can define the configuration like below.\n\n```\nmetadata:\n  format: parquet\n  dump-caller: object.to_parquet\n  dump-parameters:\n    engine: pyarrow\n  load-caller: pandas:read_parquet\n  load-parameters:\n    engine: pyarrow\n```\n\nAll the output parameters, like directory, dumper and loaders, can be overridden\nin the task level. You can also specify which tasks to export to the output\ndirectory, while the others to only be passed down to downstream in memory.\n\nFor further details, please see the [output](https://prefect-yaml.readthedocs.io/en/latest/output.html) section in documentation.\n\n## Roadmap\n\nCurrently the project is still under development. The basic features are\nmostly available while the following features are coming soon\n\n- Multi cloud storage support\n- Subtasks supported in each task\n-\n\n## Contributing\n\nAll levels of contributions are welcomed. Please refer to the [contributing](https://prefect-yaml.readthedocs.io/en/latest/contributing.html)\nsection for development and release guidelines.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffactorpricingmodel%2Fprefect-yaml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffactorpricingmodel%2Fprefect-yaml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffactorpricingmodel%2Fprefect-yaml/lists"}