{"id":22274006,"url":"https://github.com/larribas/dagger","last_synced_at":"2025-07-28T15:30:47.175Z","repository":{"id":40276095,"uuid":"349670138","full_name":"larribas/dagger","owner":"larribas","description":"Define sophisticated data pipelines with Python and run them on different distributed systems (such as Argo Workflows).","archived":false,"fork":false,"pushed_at":"2024-05-30T01:32:55.000Z","size":10451,"stargazers_count":15,"open_issues_count":0,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-29T11:55:52.045Z","etag":null,"topics":["argo-workflows","data-engineering","data-pipelines","data-science","distributed-systems","pipelines-as-code","workflows"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/larribas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-20T08:35:47.000Z","updated_at":"2024-10-11T07:31:41.000Z","dependencies_parsed_at":"2022-08-23T16:40:45.341Z","dependency_job_id":null,"html_url":"https://github.com/larribas/dagger","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larribas%2Fdagger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larribas%2Fdagger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larribas%2Fdagger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larribas%2Fdagger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/larribas","download_url":"https://codeload.github.com/larribas/dagger/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227922929,"owners_count":17840940,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["argo-workflows","data-engineering","data-pipelines","data-science","distributed-systems","pipelines-as-code","workflows"],"created_at":"2024-12-03T13:17:30.893Z","updated_at":"2024-12-03T13:17:31.828Z","avatar_url":"https://github.com/larribas.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dagger\n\nDefine sophisticated data pipelines and run them on different distributed systems (such as Argo Workflows).\n\n![Python Versions Supported](https://img.shields.io/badge/python-3.8+-blue.svg)\n[![Latest PyPI version](https://badge.fury.io/py/py-dagger.svg)](https://badge.fury.io/py/py-dagger)\n[![Test Coverage (Codecov)](https://codecov.io/gh/larribas/dagger/branch/main/graph/badge.svg?token=fKU68xYUm8)](https://codecov.io/gh/larribas/dagger)\n![Continuous Integration](https://github.com/larribas/dagger/actions/workflows/continuous-integration.yaml/badge.svg)\n\n---\n\n## Features\n\n- Define tasks and DAGs, and compose them together seamlessly.\n- Create dynamic for loops and map-reduce operations.\n- Run your DAGs locally or using a distributed workflow orchestrator (such as Argo Workflows).\n- Take advantage of advanced runtime features (e.g. Retry strategies, Kubernetes scheduling directives, etc.)\n- ... All with a simple _Pythonic_ DSL that feels just like coding regular Python functions.\n\n\nOther nice features of _Dagger_ are: Zero dependencies, 100% test coverage, great documentation and plenty of examples to get you started.\n\n\n## Installation\n\n_Dagger_ is published to the Python Package Index (PyPI) under the name `py-dagger`. To install it, you can simply run:\n\n```\npip install py-dagger\n```\n\n## Looking for Tutorials and Examples?\n\nCheck our [Documentation Portal](https://larribas.me/dagger)!\n\n\n\n\n## Architecture Overview\n\n_Dagger_ is built around 3 components:\n\n- A set of __core data structures__ that represent the intended behavior of a DAG.\n- A __domain-specific language (DSL)__ that uses metaprogramming to capture how a DAG should behave, and represents it using the core data structures.\n- Multiple __runtimes__ that inspect the core data structures to run the corresponding DAG, or prepare the DAG to run in a specific pipeline executor.\n\n\n[![components](docs/assets/images/diagrams/components.png)](docs/assets/images/diagrams/components.png)\n\n\n## How to contribute\n\nDo you have some feedback about the library? Have you implemented a Serializer or a Runtime that may be useful for the community? Do you think a tutorial or example could be improved?\n\nEvery contribution to _Dagger_ is greatly appreciated.\n\nPlease read our [Contribution Guidelines](CONTRIBUTING.md) for more details.\n\n\n\n### Local development\n\nWe use Poetry to manage the dependencies of this library. In the codebase, you will find a `Makefile` with some useful commands to run and test your contributions. Namely:\n\n- `make install` - Install the project's dependencies\n- `make test` - Run tests and report test coverage. It will fail if coverage is too low.\n- `make ci` - Run all the quality checks we run for each commit/PR. This includes type hint checking, linting, formatting and documentation.\n- `make build` - Build the project.\n- `make docker-build` - Package the project in a Docker image\n- `make docs-build` - Build the documentation portal.\n- `make docs-serve` - Serve the documentation portal.\n- `make k3d-set-up` - Create a k3d cluster and image registry for the project.\n- `make k3d-docker-push` - Build and push the project's Docker image to the local k3d registry.\n- `make k3d-install-argo` - Install Argo on k3d, for local testing of Argo Workflows.\n- `make k3d-tear-down` - Destroy the k3d cluster and registry.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flarribas%2Fdagger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flarribas%2Fdagger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flarribas%2Fdagger/lists"}