{"id":21605627,"url":"https://github.com/aspuru-guzik-group/funsies","last_synced_at":"2026-03-05T09:03:19.927Z","repository":{"id":46633336,"uuid":"321148599","full_name":"aspuru-guzik-group/funsies","owner":"aspuru-guzik-group","description":"funsies is a lightweight workflow engine 🔧","archived":false,"fork":false,"pushed_at":"2021-10-02T18:17:47.000Z","size":1469,"stargazers_count":41,"open_issues_count":1,"forks_count":4,"subscribers_count":6,"default_branch":"master","last_synced_at":"2026-02-07T13:14:17.989Z","etag":null,"topics":["automation","data-engineering","data-ops","hashtree","infrastructure","python","redis","workflow-engine"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aspuru-guzik-group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-12-13T19:58:37.000Z","updated_at":"2024-10-02T13:40:41.000Z","dependencies_parsed_at":"2022-08-29T15:01:54.613Z","dependency_job_id":null,"html_url":"https://github.com/aspuru-guzik-group/funsies","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/aspuru-guzik-group/funsies","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aspuru-guzik-group%2Ffunsies","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aspuru-guzik-group%2Ffunsies/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aspuru-guzik-group%2Ffunsies/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aspuru-guzik-group%2Ffunsies/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aspuru-guzik-group","download_url":"https://codeload.github.com/aspuru-guzik-group/funsies/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aspuru-guzik-group%2Ffunsies/sbom","scorecard":{"id":212922,"data":{"date":"2025-08-11","repo":{"name":"github.com/aspuru-guzik-group/funsies","commit":"12656f49420e9062edb9dd4c34aa18bcc94880f1"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.2,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 2/27 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/ci.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:11: update your workflow using https://app.stepsecurity.io/secureworkflow/aspuru-guzik-group/funsies/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:13: update your workflow using https://app.stepsecurity.io/secureworkflow/aspuru-guzik-group/funsies/ci.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/aspuru-guzik-group/funsies/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:29: update your workflow using https://app.stepsecurity.io/secureworkflow/aspuru-guzik-group/funsies/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:31: update your workflow using https://app.stepsecurity.io/secureworkflow/aspuru-guzik-group/funsies/ci.yml/master?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:18","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:36","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned","Info:   0 out of   2 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":8,"reason":"2 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2024-48 / GHSA-fj7x-q9j7-g6q6","Warn: Project is vulnerable to: PYSEC-2022-14 / GHSA-39ph-wr67-j4xq"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 5 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-17T01:05:19.882Z","repository_id":46633336,"created_at":"2025-08-17T01:05:19.882Z","updated_at":"2025-08-17T01:05:19.882Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29398155,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T06:24:03.484Z","status":"ssl_error","status_checked_at":"2026-02-13T06:23:12.830Z","response_time":78,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","data-engineering","data-ops","hashtree","infrastructure","python","redis","workflow-engine"],"created_at":"2024-11-24T20:16:18.344Z","updated_at":"2026-03-05T09:03:18.912Z","avatar_url":"https://github.com/aspuru-guzik-group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# funsies\nis a python library and execution engine to build reproducible,\nfault-tolerant, distributed and composable computational workflows.\n\n- 🐍 Workflows are specified in pure python.\n- 🐦 Lightweight with few dependencies.\n- 🚀 Easy to deploy to compute clusters and distributed systems.\n- 🔧 Can be embedded in your own apps.\n- 📏 First-class support for static analysis. Use\n  [mypy](http://mypy-lang.org/) to check your workflows!\n\nWorkflows are encoded in a [redis server](https://redis.io/) and executed\nusing the distributed job queue library [RQ](https://python-rq.org/). A hash\ntree data structure enables automatic and transparent caching and incremental\ncomputing.\n\n[Source docs can be found\nhere.](https://aspuru-guzik-group.github.io/funsies/) Some example funsies\nscripts can be found in the [recipes folder.](./recipes)\n\n## Installation\nUsing `pip`, \n\n```bash\npip install funsies\n```\n\nThis will enable the `funsies` CLI tool as well as the `funsies` python\nmodule. Python 3.7, 3.8 and 3.9 are supported. To run workflows, you'll need a\nRedis server, version 4.x or higher. On Linux Redis can be installed using conda,\n\n```bash\nconda install redis\n```\n\npip,\n\n```bash\npip install redis-server\n```\n\nor your system package manager. On Mac OSX, Redis can be downloaded using\nHomebrew,\n\n```bash\nbrew install redis\n```\n\n(Windows is not supported by Redis, but a third-party package can be obtained\nfrom [this repository](https://github.com/tporadowski/redis). This has **not**\nbeen tested, however.)\n\n## Hello, funsies!\nTo run workflows, three components need to be connected:\n\n- 📜 a python script describing the workflow\n- 💻 a redis server that holds workflows and data\n- 👷 worker processes that execute the workflow\n\nfunsies is distributed: all three components can be on different computers or\neven be connected at different time. Redis is started using `redis-server`,\nworkers are started using `funsies worker` and the workflow is run using\npython.\n\nFor running on a single machine, the `start-funsies` script takes care of starting the database and workers,\n\n```bash\nstart-funsies \\\n    --no-pw \\\n    --workers 2\n```\n\nHere is an example workflow script,\n\n```python\nfrom funsies import Fun, reduce, shell\nwith Fun():\n    # you can run shell commands\n    cmd = shell('sleep 2; echo 👋 🪐')\n    # and python ones\n    python = reduce(sum, [3, 2])\n    # outputs are saved at hash addresses\n    print(f\"my outputs are saved to {cmd.stdout.hash[:5]} and {python.hash[:5]}\")\n```\n\nThe workflow is just python, and is run using the python interpreter,\n\n```bash\n$ python hello-world.py\nmy outputs are saved to 4138b and 80aa3\n```\n\nThe `Fun()` context manager takes care of connecting to the database. The\nscript should execute immediately; no work is done just yet because workflows\nare lazily executed.\n\nTo execute the workflow, we trigger using the hashes above using the CLI,\n\n```bash\n$ funsies execute 4138b 80aa3\n```\n\nOnce the workers are finished, results can be printed directly to stdout using\ntheir hashes,\n\n```bash\n$ funsies cat 4138b\n👋 🪐\n$ funsies cat 80aa3\n5\n```\n\nThey can also be accessed from within python, from other steps in the\nworkflows etc. Shutting down the database and workers can also be performed\nusing the CLI,\n\n```bash\n$ funsies shutdown --all\n```\n\n## How does it work?\n\nThe design of **funsies** is inspired by\n[git](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects) and\n[ccache](https://ccache.dev/). All files and variable values are abstracted\ninto a provenance-tracking DAG structure. Basically, \"files\" are identified\nentirely based on what operations lead to their creation. This (somewhat\nopinionated) design produces interesting properties that are not common in\nworkflow engines:\n\n#### Incremental computation\n\nfunsies automatically and transparently saves all input and output \"files\".\nThis produces automatic and transparent checkpointing and [incremental\ncomputing](https://en.wikipedia.org/wiki/Incremental_computing). Re-running\nthe same funsies script, **even on a different machine**, will not perform any\ncomputations (beyond database lookups). Modifying the script and re-running it\nwill only recompute changed results. \n\nIn contrast with e.g. Make, this is not based on modification date but\ndirectly on the data history, which is more robust to changes in the workflow.\n\n#### Decentralized workflows\n\nWorkflows and their elements are not identified based on any global indexing\nscheme. This makes it possible to generate workflows fully dynamically from\nany connected computer node, to merge or compose DAGs from different databases\nand to dynamically re-parametrize them, etc.\n\n#### No local file operations\n\nAll \"files\" are encoded in a redis instance or to a data directory, with no\nlocal filesystem management required. funsies workers can even operate without\nany permanent data storage, as is often the case in file-driven workflows\nusing only a container's [tmpfs](https://docs.docker.com/storage/tmpfs/).\n\n## Recovering from failures\n\nRaised exceptions in python codes, worker failures, missing output files and\nother error conditions are automatically caught by funsies workers, providing\nfault tolerance to workflows. Errors are logged on `stderr` with full\ntraceback and can be recovered from the database.\n\nSteps that depend on failed ones propagate those\nerrors and their provenance. Errors can then be dealt with wherever it is most\nappropriate to do so [using techniques from functional\nprogramming.](https://fsharpforfunandprofit.com/rop/) \n\nAs an example, consider a workflow that first runs a CLI program `simulate`\nthat ought to produce a `results.csv` file, which is subsequently analyzed\nusing a python function `analyze_data()`,\n\n```python\nimport funsies as f\n\nsim = f.shell(\"simulate data.inp\", inp={\"data.inp\":\"some input\"}, out=[\"results.csv\"])\nfinal = f.reduce(analyze_data, sim.out[\"results.csv\"])\n```\n\nIn a normal python program, `analyze_data()` would need to guard against the\npossibility that `results.csv` is absent, or risk a fatal exception. In the\nabove funsies script, if `results.csv` is not produced, then it is replaced by\nan instance of `Error` which tracks the failing step. The workflow engine\nautomatically shortcircuit the execution of `analyze_data` and insteads\nforward the `Error` to `final`. In this way, the value of `final` provides\ndirect error tracing to the failed step. Furthermore, it means that\n`analyze_data` does not need it's own error handling code if its output is\noptional or if the error is better dealt with in a later step.\n\nThis error-handling approach is heavily influenced by the `Result\u003cT,E\u003e` type\nfrom [the Rust programming language.](https://doc.rust-lang.org/std/result/)\n\n\n## Is it production-ready?\n\n🧪 warning: funsies is research-grade code ! 🧪\n\nAt this time, the funsies API is fairly stable. However, users should know\nthat database dumps are not yet fully forward- or backward-compatible, and\nbreaking changes are likely to be introduced on new releases.\n\n## Related projects\nfunsies is intended as a lightweight alternative to industrial workflow\nengines, such as [Apache Airflow](https://airflow.apache.org/) or\n[Luigi](https://github.com/spotify/luigi). We rely heavily on awesome python\nlibraries: [RQ library](https://github.com/rq/rq),\n[loguru](https://github.com/Delgan/loguru),\n[Click](https://click.palletsprojects.com/) and\n[chevron](https://github.com/noahmorrison/chevron). We are inspired by\n[git](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects),\n[ccache](https://ccache.dev/),\n[snakemake](https://snakemake.readthedocs.io/en/stable/)\n[targets](https://github.com/ropensci/targets),\n[rain](https://github.com/substantic/rain) and others. A comprehensive list of\nother worfklow engine can be found\n[here.](https://github.com/pditommaso/awesome-pipeline)\n\n\n## License\n\nfunsies is provided under the MIT license.\n\n## Contributing\n\nAll contributions are welcome! Consult [the CONTRIBUTING](./CONTRIBUTING.md)\nfile for help. Please file issues for any bugs and documentation problems.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faspuru-guzik-group%2Ffunsies","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faspuru-guzik-group%2Ffunsies","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faspuru-guzik-group%2Ffunsies/lists"}