{"id":37066226,"url":"https://github.com/joschnitzbauer/dalymi","last_synced_at":"2026-01-14T07:46:19.215Z","repository":{"id":62566346,"uuid":"112862496","full_name":"joschnitzbauer/dalymi","owner":"joschnitzbauer","description":"A lightweight, data-focused and non-opinionated pipeline manager written in and for Python.","archived":false,"fork":false,"pushed_at":"2019-11-24T22:45:47.000Z","size":162,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-01-08T00:28:19.554Z","etag":null,"topics":["dag","data","data-science","pipeline","python","workflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joschnitzbauer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-02T17:32:46.000Z","updated_at":"2023-06-08T20:50:02.000Z","dependencies_parsed_at":"2022-11-03T16:16:04.236Z","dependency_job_id":null,"html_url":"https://github.com/joschnitzbauer/dalymi","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/joschnitzbauer/dalymi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joschnitzbauer%2Fdalymi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joschnitzbauer%2Fdalymi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joschnitzbauer%2Fdalymi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joschnitzbauer%2Fdalymi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joschnitzbauer","download_url":"https://codeload.github.com/joschnitzbauer/dalymi/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joschnitzbauer%2Fdalymi/sbom","scorecard":{"id":533131,"data":{"date":"2025-08-11","repo":{"name":"github.com/joschnitzbauer/dalymi","commit":"06da859bda3135d53ddf095e998e51c43be2194d"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 0/21 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/publish.yml:11: update your workflow using https://app.stepsecurity.io/secureworkflow/joschnitzbauer/dalymi/publish.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/publish.yml:13: update your workflow using https://app.stepsecurity.io/secureworkflow/joschnitzbauer/dalymi/publish.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:11: update your workflow using https://app.stepsecurity.io/secureworkflow/joschnitzbauer/dalymi/tests.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:13: update your workflow using https://app.stepsecurity.io/secureworkflow/joschnitzbauer/dalymi/tests.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/tests.yml:60: update your workflow using https://app.stepsecurity.io/secureworkflow/joschnitzbauer/dalymi/tests.yml/master?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/publish.yml:18","Warn: pipCommand not pinned by hash: .github/workflows/publish.yml:19","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:18","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:19","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:22","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:29","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:30","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:34","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:57","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned","Info:   0 out of   9 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/publish.yml:1","Warn: no topLevel permission defined: .github/workflows/tests.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.txt:0","Info: FSF or OSI recognized license: MIT License: LICENSE.txt:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 10 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-20T06:13:46.415Z","repository_id":62566346,"created_at":"2025-08-20T06:13:46.415Z","updated_at":"2025-08-20T06:13:46.415Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28413490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T05:26:33.345Z","status":"ssl_error","status_checked_at":"2026-01-14T05:21:57.251Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dag","data","data-science","pipeline","python","workflow"],"created_at":"2026-01-14T07:46:18.556Z","updated_at":"2026-01-14T07:46:19.191Z","avatar_url":"https://github.com/joschnitzbauer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dalymi\n\n*[data like you mean it]*\n\n[![Documentation Status](https://readthedocs.org/projects/dalymi/badge/?version=latest)](http://dalymi.readthedocs.io/en/latest/?badge=latest) ![](https://github.com/joschnitzbauer/dalymi/workflows/tests/badge.svg?branch=master)\n\nA lightweight, data-focused and non-opinionated pipeline manager written in and for Python.\n\n--------------------------------------------------------------------------------\n\n_dalymi_ allows to build data processing pipelines as [directed acyclic graphs]([https://en.wikipedia.org/wiki/Directed_acyclic_graph]) (DAGs) and facilitates rapid, but controlled, model development. The goal is to prototype quickly, but scale to production with ease.\nTo achieve this, _dalymi_ uses \"make\"-style workflows, _i.e._ tasks with missing input trigger the execution of input-producing tasks before being executed themselves. At the same time, _dalymi_ provides fine control to run and undo specific pipeline parts for quick test iterations. This ensures output reproducability and minimizes manual errors.\n\nSeveral features facilitate _dalymi_'s goal:\n\n- simple, non-opinionated API (most choices left to user)\n- no external dependencies for pipeline execution\n- one-line installation (ready for use)\n- no configuration\n- auto-generated command line interface for pipeline execution\n- quick start, but high flexibility to customize and extend:\n    - task output can be stored in any format Python can touch (local files being the default)\n    - customizable command line arguments\n    - templated output location (e.g. timestamped files)\n    - support for automated checks on data integrity during runtime\n- DAG visualization using [graphviz](https://www.graphviz.org/)\n- API design encourages good development practices (modular code, defined data schemas, self-documenting code, easy workflow viz, etc.)\n\n## Installation\n_dalymi_ requires Python \u003e= 3.5.\n\n``` bash\npip install dalymi\n```\n\nFor the latest development:\n``` bash\npip install git+https://github.com/joschnitzbauer/dalymi.git\n```\n\n## Documentation\nhttp://dalymi.readthedocs.io/\n\n## Simple example\nsimple.py:\n``` python\nfrom dalymi import Pipeline\nfrom dalymi.resources import PandasCSV\nimport pandas as pd\n\n\n# Define resources:\nnumbers_resource = PandasCSV(name='numbers', loc='numbers.csv', columns=['number'])\nsquares_resource = PandasCSV(name='squares', loc='squares.csv', columns=['number', 'square'])\n\n\n# Define the pipeline\npl = Pipeline()\n\n\n@pl.output(numbers_resource)\ndef create_numbers(**context):\n    return pd.DataFrame({'number': range(11)})\n\n\n@pl.output(squares_resource)\n@pl.input(numbers_resource)\ndef square_numbers(numbers, **context):\n    numbers['square'] = numbers['number']**2\n    return numbers\n\n\nif __name__ == '__main__':\n    # Run the default command line interface\n    pl.cli()\n```\n\nCommand line:\n```bash\npython simple.py run     # executes the pipeline. skips tasks for which output already exists.\n```\n\nMore examples can be found [here](https://github.com/joschnitzbauer/dalymi/tree/master/examples).\n\n## Roadmap\n- More docstrings\n- Unit tests\n- Parallel task processing\n- REST API during pipeline run\n- Web interface for pipeline run\n\n## Warranty\nAlthough _dalymi_ is successfully used in smaller applications, it is not battle-tested yet and lacks unit tests. If you decide to use it, be prepared to communicate issues or fix bugs (it's not a lot of code... :)).\n\n## Contributions\n... are welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoschnitzbauer%2Fdalymi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoschnitzbauer%2Fdalymi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoschnitzbauer%2Fdalymi/lists"}