{"id":15070046,"url":"https://github.com/martindbp/merkl","last_synced_at":"2026-02-07T18:35:14.798Z","repository":{"id":64308478,"uuid":"320224302","full_name":"martindbp/merkl","owner":"martindbp","description":"ML pipelines in pure Python with great caching","archived":false,"fork":false,"pushed_at":"2022-12-07T19:23:15.000Z","size":557,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-28T17:12:03.241Z","etag":null,"topics":["data-science","mlops","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/martindbp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-10T09:36:41.000Z","updated_at":"2022-02-11T14:03:08.000Z","dependencies_parsed_at":"2023-01-15T11:15:24.645Z","dependency_job_id":null,"html_url":"https://github.com/martindbp/merkl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/martindbp/merkl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martindbp%2Fmerkl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martindbp%2Fmerkl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martindbp%2Fmerkl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martindbp%2Fmerkl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/martindbp","download_url":"https://codeload.github.com/martindbp/merkl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martindbp%2Fmerkl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29203785,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-07T17:44:10.191Z","status":"ssl_error","status_checked_at":"2026-02-07T17:44:07.936Z","response_time":63,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","mlops","python3"],"created_at":"2024-09-25T01:46:44.037Z","updated_at":"2026-02-07T18:35:14.782Z","avatar_url":"https://github.com/martindbp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MerkL - ML pipelines in pure Python with great caching\n\nMerkL is a tool for creating ML pipelines in pure Python that are useful for\ndevelopment and experimentation, but also easy to deploy to production without\nmodifications. Results are cached using Merkle hashes of the code and inputs as\nkeys, greatly simplifying caching and reducing the need for a feature store.\n\nNOTE: this project is not production ready and maybe never will be. At this\npoint I'm the only user. If you're interested in talking about it I can be\nreached at me@martindbp.com.\n\n## Installation\n\n`pip install merkl`\n\n## Tour and examples\n\nIn MerkL, pipelines are built using functions decorated with the `task` decorator. When a task is called, the function\nbody is not exectuted immediately, but instead `Future` objects are returned in place of real outputs. These can then be\npassed on to other tasks:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003epipeline1.py\u003c/th\u003e\n\u003cth\u003emerkl dot pipeline1.my_pipeline 42 | dot -Tpng | display\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd valign=\"top\"\u003e\n\n```python\nfrom merkl import task\n\n@task\ndef task1(input_value):\n    return 2 * input_value\n\n@task\ndef task2(input_value):\n    return input_value ** 2\n\ndef my_pipeline(input_value: int):\n    val = task1(input_value)\n    final_val = task2(val)\n    return final_val\n```\n\n\u003c/td\u003e\n\u003ctd align=\"center\" valign=\"top\"\u003e\n\n![](docs/pipeline1.png)\n\n\u003c/td\u003e\n\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003cth colspan=\"2\"\u003emerkl run pipeline1.my_pipeline 42\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"2\" valign=\"top\"\u003e\n\n```\n7056\n```\n\n\u003c/td\u003e\n\u003ctr\u003e\n\u003c/table\u003e\n\nNo function body is executed before `.eval()` is called on a Future, or it is returned by a pipeline executed by the `run` command.\nInstead a graph is built and each Future is assigned a hash that uniquely identifies its future value. If the code or input values change, then the\noutput Future hashes also change. These hashes are then used to find cached results.\n\nAs seen in the table above, one can use the `merkl run` command to run a pipline, and the `merkl dot` command to create\na dot file for visualizing the computational graph. The graph can be rendered and displayed by piping the dot file\nthrough the `dot` and `visualize` programs (note that these require that graphviz and imagemagick are installed).\n\nArguments can be passed to the pipeline through the `run` command. See [clize](https://clize.readthedocs.io/en/stable/)\nfor more information on how to pass parameters from the command line.\n\nBy default, all computed outputs are cached, unless the `--no-cache` option is supplied. If we run the pipeline a second time, we can see that the hashes turn green, which indicates that the values are already in the cache. Changing the code in `task2`, for example by changing the `2` to a `3`, we can see that the output value is\nnot cached anymore:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eFirst run\u003c/th\u003e\n\u003cth\u003eSecond run\u003c/th\u003e\n\u003cth\u003eModify task2\u003c/th\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd\u003e\n\n![](docs/pipeline1.png)\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n![](docs/pipeline1_2.png)\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n![](docs/pipeline1_3.png)\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\nHere's a slightly more realistic pipeline consisting of a model training and evaluation task:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003epipeline2.py\u003c/th\u003e\n\u003cth\u003emerkl dot pipeline2.train_eval  | dot -Tpng | display\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd valign=\"top\"\u003e\n\n```python\nfrom merkl import task, read_future\n\n@task\ndef train(data, iterations):\n    return 'trained model'\n\n@task\ndef evaluate(model, data):\n    return 99.3\n\ndef train_eval():\n    train_data = read_future('train.csv')\n    test_data = read_future('test.csv')\n    model = train(train_data, iterations=100)\n    model \u003e 'model.bin'\n    score = evaluate(model, test_data)\n    return score, model\n```\n\n\u003c/td\u003e\n\u003ctd align=\"center\" valign=\"top\"\u003e\n\n![](docs/pipeline2.png)\n\n\u003c/td\u003e\n\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003cth colspan=\"2\"\u003emerkl run pipeline2.train_eval \u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"2\" valign=\"top\"\u003e\n\n```\n(99.3, 'trained model')\n```\n\n\u003c/td\u003e\n\u003ctr\u003e\n\u003c/table\u003e\n\n### Hashing\n\nIn order to compute the recursive Merkle hashes, the content (md5) hash of the\ninput files are needed. MerkL hashes these on demand as they are needed, but\nstores these hashes in the `.merkl/cache.sqlite3` database. If the timestamp of\nthe file ever changes, the file is hashed again.\n\nFiles created by MerkL tasks or pipelines are also tracked in the database so that MerkL knows if a file needs to be updated or not.\n\n### Multiple outs\n\nThe number of outputs that you want to track separately has to be determinable at DAG \"build\" time,\ntherefore it cannot be dependent on some value computed inside a task.\n\nBy default, MerkL tries to guess the number of outs by looking at the return statements in the AST (Abstract Syntax\nTree) using Python's inspection capabilities. If the return statement is a Tuple or Dict literal, the values are split\ninto separate Futures:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eTuple multiple outs\u003c/th\u003e\n\u003cth\u003eTuple single out\u003c/th\u003e\n\u003cth\u003eDict multiple outs\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd valign=\"top\"\u003e\n\n```python\n@task\ndef my_task():\n    return 1, 2, 3\n\nprint(my_task())\n```\n\n\u003c/td\u003e\n\u003ctd valign=\"top\"\u003e\n\n```python\n@task\ndef my_task():\n    values = 1, 2, 3\n    return values\n\nprint(my_task())\n```\n\n\u003c/td\u003e\n\u003ctd valign=\"top\"\u003e\n\n```python\n@task\ndef my_task():\n    return {\n        'key1': 1,\n        'key2': 2,\n        'key3': 3,\n    }\n\nprint(my_task())\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003cth colspan=\"3\"\u003eSTDOUT\u003c/th\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd valign=\"top\"\u003e\n\n```\n(\n    \u003cFuture: b7a152bd\u003e,\n    \u003cFuture: 3612309d\u003e,\n    \u003cFuture: 541053ee\u003e,\n)\n```\n\n\u003c/td\u003e\n\u003ctd valign=\"top\"\u003e\n\n```\n\u003cFuture: bc930b90\u003e\n```\n\n\u003c/td\u003e\n\u003ctd valign=\"top\"\u003e\n\n```\n{\n    'key1': \u003cFuture: 7d7f02e7\u003e,\n    'key2': \u003cFuture: 09f5f892\u003e,\n    'key3': \u003cFuture: aaba8931\u003e,\n}\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nIn some cases we want the number of outs to be dynamic and depend on the input to the function. In this case you can\nsupply a callable to the `outs` task parameter with the same signature as the task function, and return the number of\nouts:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eouts_lambda.py\u003c/th\u003e\n\u003cth\u003emerkl dot outs_lambda.pipeline  | dot -Tpng | display\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd valign=\"top\"\u003e\n\n```python\nfrom merkl import task\n\n@task(outs=lambda data, k: k)\ndef split(data, k):\n    ksize = len(data) // k\n    return [data[i*ksize:(i+1)*ksize]\n            for i in range(k)]\n\ndef pipeline():\n    return split([1, 2, 3, 4, 5, 6], k=2)\n```\n\n\u003c/td\u003e\n\u003ctd align=\"center\" valign=\"top\"\u003e\n\n![](docs/pipeline3.png)\n\n\u003c/td\u003e\n\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003cth colspan=\"2\"\u003emerkl run outs_lambda.pipeline \u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"2\" valign=\"top\"\u003e\n\n```\n([1, 2, 3], [4, 5, 6])\n```\n\n\u003c/td\u003e\n\u003ctr\u003e\n\u003c/table\u003e\n\nIf all else fails, you can set `outs` to any positive integer or an iterable of output dictionary keys.\n\n### Pipe syntax\n\nFor convenience, you can optionally chain tasks using the `|` operator, if the tasks have a single input and output. A\nfuture can also be \"piped\" directly to a file path using the `\u003e` operator:\n\n```python\n# Original\nwrite_future(train(preprocess(read_future('train.csv'))), 'model.bin')\n\n# With pipe syntax\nread_future('train.csv') | preprocess | train \u003e 'model.bin'\n```\n\n### Pipelines\n\nPipeline functions can optionally be decorated with the `pipeline` decorator. The decorator provides caching of the\ngraph construction (as opposed to only the evaluation). This is useful when for example there is an inference stage\nwhich depends on a training stage which preprocesses thousands of images. Even though the _result_ of the training is\ncached, the _construction_ of the graph may become slow.\n\n```python\n\n@task\ndef preprocess(image):\n    ...\n\n@task(outs=lambda image, k: k)\ndef augment(image, k):\n    ...\n\n@task\ndef train(\n\n```\n\n### Batch tasks\n\nSome functions have a batch version that has a more efficient implementation than the single item version. In this case, we\nmight want to track the outputs from the batch task _as if_ they were produced by the single item version, such that\noutputs between them will be cached. In this case you can use the `batch` decorator:\n\n```python\n@batch(embed_word)\ndef embed_words(words):\n    ...\n    return embedded_words\n```\n\nIn some cases, you might only have a batch implementation, but you want each output to be treated individually. In that\ncase you can also use the `batch` decorator but without an argument:\n\n```python\n\n@batch\ndef embed_words(words):\n    ...\n    return embedded_words\n\n```\nThe difference here is that identical inputs will have the same output Future hash, which is not true for a regular\ntask. You can tell the difference in these two graphs:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e1\u003c/th\u003e\n\u003cth\u003e2\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd valign=\"top\"\u003e\n\n![](docs/pipeline4_1.png)\n\n\u003c/td\u003e\n\u003ctd valign=\"top\"\u003e\n\n![](docs/pipeline4_2.png)\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\n\n```python\nouts = embed_sentences_batch(['my sentence', 'my sentence'])\nassert outs[0].hash == outs[1].hash\n\nouts = embed_sentences_task(['my sentence', 'my sentence'])\nassert outs[0].hash != outs[1].hash\n```\n\n### HashMode and dependencies\n\nWhen the hash of a task function is determined, there are three different `HashMode`s: `FUNCTION`, `MODULE` and\n`FIND_DEPS`. The default `HashMode` for both `task`, `batch` and `pipeline` is `FIND_DEPS`.\n\n* `FUNCTION`: Uses the code of the task function for hashing\n* `MODULE`: Uses the whole module file\n* `FIND_DEPS`: Finds dependencies such as other functions or nonlocal variables used within the function that are\n  defined in the same module. If a dep is a `task` then its dependencies are added in turn.\n\nDeps can be added manually and could be a module, function, task, bytes, string or any JSON-serializable object:\n\n```python\n@task(deps=['my string'])\ndef my_task():\n    pass\n```\n\nAs a convenience, you can use the `pip_version()` function to easily add a library's version string as a hash key:\n\n```python\n@task(deps=[pip_version('numpy')])\ndef my_task():\n    pass\n```\n\nFiles and directories can be added as dependencies using the `FileRef` and `DirRef` classes:\n\n```python\nfrom merkl import task, FileRef\n@task(deps=[FileRef('my_file.txt')])\ndef my_task():\n    pass\n```\n\nRefs added as dependencies will contribute the file content hash to the task.\n\n### Filesystem IO\n\nTBD\n\n### Task versions\n\nTBD\n\n### The Cache\n\nTBD\n\n### Wrapping shell commands\n\nTBD\n\n## Dev Notes\n\nReleasing a new version:\n\n1. Bump version in setup.py\n2. Commit\n3. `git tag $VERSION`\n4. `make release`. This bundles the package and uploads it using twine\n\nNote that twine is required: `pip install twine`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmartindbp%2Fmerkl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmartindbp%2Fmerkl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmartindbp%2Fmerkl/lists"}