{"id":20837748,"url":"https://github.com/astrazeneca/runnable","last_synced_at":"2025-05-08T20:30:39.418Z","repository":{"id":39856254,"uuid":"449367132","full_name":"AstraZeneca/runnable","owner":"AstraZeneca","description":"Runnable","archived":false,"fork":false,"pushed_at":"2025-04-06T20:28:42.000Z","size":10582,"stargazers_count":39,"open_issues_count":16,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-06T21:28:43.736Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AstraZeneca.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-18T16:43:54.000Z","updated_at":"2025-04-06T20:26:28.000Z","dependencies_parsed_at":"2024-04-06T08:22:27.802Z","dependency_job_id":"be325ff0-ae10-4d7f-b8e2-ad4c5044ce4e","html_url":"https://github.com/AstraZeneca/runnable","commit_stats":{"total_commits":139,"total_committers":5,"mean_commits":27.8,"dds":0.06474820143884896,"last_synced_commit":"5e4215f9a3ce452598f1d60088638a4aa080f7b4"},"previous_names":["astrazeneca/runnable","astrazeneca/magnus-core"],"tags_count":95,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AstraZeneca%2Frunnable","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AstraZeneca%2Frunnable/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AstraZeneca%2Frunnable/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AstraZeneca%2Frunnable/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AstraZeneca","download_url":"https://codeload.github.com/AstraZeneca/runnable/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253144527,"owners_count":21861073,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T01:08:28.604Z","updated_at":"2025-05-08T20:30:39.366Z","avatar_url":"https://github.com/AstraZeneca.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n\n\n\n\u003c/p\u003e\n\u003chr style=\"border:2px dotted orange\"\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://pypi.org/project/runnable/\"\u003e\u003cimg alt=\"python:\" src=\"https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10-blue.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/runnable/\"\u003e\u003cimg alt=\"Pypi\" src=\"https://badge.fury.io/py/runnable.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/vijayvammi/runnable/blob/main/LICENSE\"\u003e\u003cimg alt\"License\" src=\"https://img.shields.io/badge/license-Apache%202.0-blue.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/psf/black\"\u003e\u003cimg alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/python/mypy\"\u003e\u003cimg alt=\"MyPy Checked\" src=\"https://www.mypy-lang.org/static/mypy_badge.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/vijayvammi/runnable/actions/workflows/release.yaml\"\u003e\u003cimg alt=\"Tests:\" src=\"https://github.com/vijayvammi/runnable/actions/workflows/release.yaml/badge.svg\"\u003e\n\u003c/p\u003e\n\u003chr style=\"border:2px dotted orange\"\u003e\n\n\n[Please check here for complete documentation](https://astrazeneca.github.io/runnable/)\n\n## Example\n\nThe below data science flavored code is a well-known\n[iris example from scikit-learn](https://scikit-learn.org/stable/auto_examples/linear_model/plot_iris_logistic.html).\n\n\n```python\n\"\"\"\nExample of Logistic regression using scikit-learn\nhttps://scikit-learn.org/stable/auto_examples/linear_model/plot_iris_logistic.html\n\"\"\"\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom sklearn import datasets\nfrom sklearn.inspection import DecisionBoundaryDisplay\nfrom sklearn.linear_model import LogisticRegression\n\n\ndef load_data():\n    # import some data to play with\n    iris = datasets.load_iris()\n    X = iris.data[:, :2]  # we only take the first two features.\n    Y = iris.target\n\n    return X, Y\n\n\ndef model_fit(X: np.ndarray, Y: np.ndarray, C: float = 1e5):\n    logreg = LogisticRegression(C=C)\n    logreg.fit(X, Y)\n\n    return logreg\n\n\ndef generate_plots(X: np.ndarray, Y: np.ndarray, logreg: LogisticRegression):\n    _, ax = plt.subplots(figsize=(4, 3))\n    DecisionBoundaryDisplay.from_estimator(\n        logreg,\n        X,\n        cmap=plt.cm.Paired,\n        ax=ax,\n        response_method=\"predict\",\n        plot_method=\"pcolormesh\",\n        shading=\"auto\",\n        xlabel=\"Sepal length\",\n        ylabel=\"Sepal width\",\n        eps=0.5,\n    )\n\n    # Plot also the training points\n    plt.scatter(X[:, 0], X[:, 1], c=Y, edgecolors=\"k\", cmap=plt.cm.Paired)\n\n    plt.xticks(())\n    plt.yticks(())\n\n    plt.savefig(\"iris_logistic.png\")\n\n    # TODO: What is the right value?\n    return 0.6\n\n\n## Without any orchestration\ndef main():\n    X, Y = load_data()\n    logreg = model_fit(X, Y, C=1.0)\n    generate_plots(X, Y, logreg)\n\n\n## With runnable orchestration\ndef runnable_pipeline():\n    # The below code can be anywhere\n    from runnable import Catalog, Pipeline, PythonTask, metric, pickled\n\n    # X, Y = load_data()\n    load_data_task = PythonTask(\n        function=load_data,\n        name=\"load_data\",\n        returns=[pickled(\"X\"), pickled(\"Y\")],  # (1)\n    )\n\n    # logreg = model_fit(X, Y, C=1.0)\n    model_fit_task = PythonTask(\n        function=model_fit,\n        name=\"model_fit\",\n        returns=[pickled(\"logreg\")],\n    )\n\n    # generate_plots(X, Y, logreg)\n    generate_plots_task = PythonTask(\n        function=generate_plots,\n        name=\"generate_plots\",\n        terminate_with_success=True,\n        catalog=Catalog(put=[\"iris_logistic.png\"]),  # (2)\n        returns=[metric(\"score\")],\n    )\n\n    pipeline = Pipeline(\n        steps=[load_data_task, model_fit_task, generate_plots_task],\n    )  # (4)\n\n    pipeline.execute()\n\n    return pipeline\n\n\nif __name__ == \"__main__\":\n    # main()\n    runnable_pipeline()\n\n```\n\n\n1. Return two serialized objects X and Y.\n2. Store the file `iris_logistic.png` for future reference.\n3. Define the sequence of tasks.\n4. Define a pipeline with the tasks\n\nThe difference between native driver and runnable orchestration:\n\n!!! tip inline end \"Notebooks and Shell scripts\"\n\n    You can execute notebooks and shell scripts too!!\n\n    They can be written just as you would want them, *plain old notebooks and scripts*.\n\n\n\n\n\u003cdiv class=\"annotate\" markdown\u003e\n\n```diff\n\n- X, Y = load_data()\n+load_data_task = PythonTask(\n+    function=load_data,\n+     name=\"load_data\",\n+     returns=[pickled(\"X\"), pickled(\"Y\")], (1)\n+    )\n\n-logreg = model_fit(X, Y, C=1.0)\n+model_fit_task = PythonTask(\n+   function=model_fit,\n+   name=\"model_fit\",\n+   returns=[pickled(\"logreg\")],\n+   )\n\n-generate_plots(X, Y, logreg)\n+generate_plots_task = PythonTask(\n+   function=generate_plots,\n+   name=\"generate_plots\",\n+   terminate_with_success=True,\n+   catalog=Catalog(put=[\"iris_logistic.png\"]), (2)\n+   )\n\n\n+pipeline = Pipeline(\n+   steps=[load_data_task, model_fit_task, generate_plots_task], (3)\n\n```\n\u003c/div\u003e\n\n\n---\n\n- [x] ```Domain``` code remains completely independent of ```driver``` code.\n- [x] The ```driver``` function has an equivalent and intuitive runnable expression\n- [x] Reproducible by default, runnable stores metadata about code/data/config for every execution.\n- [x] The pipeline is `runnable` in any environment.\n\n\n## Documentation\n\n[More details about the project and how to use it available here](https://astrazeneca.github.io/runnable/).\n\n\u003chr style=\"border:2px dotted orange\"\u003e\n\n## Installation\n\nThe minimum python version that runnable supports is 3.8\n\n```shell\npip install runnable\n```\n\nPlease look at the [installation guide](https://astrazeneca.github.io/runnable-core/usage)\nfor more information.\n\n\n## Pipelines can be:\n\n### Linear\n\nA simple linear pipeline with tasks either\n[python functions](https://astrazeneca.github.io/runnable-core/concepts/task/#python_functions),\n[notebooks](https://astrazeneca.github.io/runnable-core/concepts/task/#notebooks), or [shell scripts](https://astrazeneca.github.io/runnable-core/concepts/task/#shell)\n\n[![](https://mermaid.ink/img/pako:eNpl0bFuwyAQBuBXQVdZTqTESpxMDJ0ytkszhgwnOCcoNo4OaFVZfvcSx20tGSQ4fn0wHB3o1hBIyLJOWGeDFJ3Iq7r90lfkkA9HHfmTUpnX1hFyLvrHzDLl_qB4-1BOOZGGD3TfSikvTDSNFqdj2sT2vBTr9euQlXNWjqycsN2c7UZWFMUE7udwP0L3y6JenNKiyfvz8t8_b-gavT9QJYY0PcDtjeTLptrAChriBq1JzeoeWkG4UkMKZCoN8k2Bcn1yGEN7_HYaZOBIK4h3g4EOFi-MDcgKa59SMja0_P7s_vAJ_Q_YOH6o?type=png)](https://mermaid.live/edit#pako:eNpl0bFuwyAQBuBXQVdZTqTESpxMDJ0ytkszhgwnOCcoNo4OaFVZfvcSx20tGSQ4fn0wHB3o1hBIyLJOWGeDFJ3Iq7r90lfkkA9HHfmTUpnX1hFyLvrHzDLl_qB4-1BOOZGGD3TfSikvTDSNFqdj2sT2vBTr9euQlXNWjqycsN2c7UZWFMUE7udwP0L3y6JenNKiyfvz8t8_b-gavT9QJYY0PcDtjeTLptrAChriBq1JzeoeWkG4UkMKZCoN8k2Bcn1yGEN7_HYaZOBIK4h3g4EOFi-MDcgKa59SMja0_P7s_vAJ_Q_YOH6o)\n\n### [Parallel branches](https://astrazeneca.github.io/runnable-core/concepts/parallel)\n\nExecute branches in parallel\n\n[![](https://mermaid.ink/img/pako:eNp9k01rwzAMhv-K8S4ZtJCzDzuMLmWwwkh2KMQ7eImShiZ2sB1KKf3vs52PpsWNT7LySHqlyBeciRwwwUUtTtmBSY2-YsopR8MpQUfAdCdBBekWNBpvv6-EkFICzGAtWcUTDW3wYy20M7lr5QGBK2j-anBAkH4M1z6grnjpy17xAiTwDII07jj6HK8-VnVZBspITnpjztyoVkLLJOy3Qfrdm6gQEu2370Io7WLORo84PbRoA_oOl9BBg4UHbHR58UkMWq_fxjrOnhLRx1nH0SgkjlBjh7ekxNKGc0NelDLknhePI8qf7MVNr_31nm1wwNTeM2Ao6pmf-3y3Mp7WlqA7twOnXfKs17zt-6azmim1gQL1A0NKS3EE8hKZE4Yezm3chIVFiFe4AdmwKjdv7mIjKNYHaIBiYsycySPFlF8NxzotkjPPMNGygxXu2pxp2FSslKzBpGC1Ml7IKy3krn_E7i1f_wEayTcn?type=png)](https://mermaid.live/edit#pako:eNp9k01rwzAMhv-K8S4ZtJCzDzuMLmWwwkh2KMQ7eImShiZ2sB1KKf3vs52PpsWNT7LySHqlyBeciRwwwUUtTtmBSY2-YsopR8MpQUfAdCdBBekWNBpvv6-EkFICzGAtWcUTDW3wYy20M7lr5QGBK2j-anBAkH4M1z6grnjpy17xAiTwDII07jj6HK8-VnVZBspITnpjztyoVkLLJOy3Qfrdm6gQEu2370Io7WLORo84PbRoA_oOl9BBg4UHbHR58UkMWq_fxjrOnhLRx1nH0SgkjlBjh7ekxNKGc0NelDLknhePI8qf7MVNr_31nm1wwNTeM2Ao6pmf-3y3Mp7WlqA7twOnXfKs17zt-6azmim1gQL1A0NKS3EE8hKZE4Yezm3chIVFiFe4AdmwKjdv7mIjKNYHaIBiYsycySPFlF8NxzotkjPPMNGygxXu2pxp2FSslKzBpGC1Ml7IKy3krn_E7i1f_wEayTcn)\n\n### [loops or map](https://astrazeneca.github.io/runnable-core/concepts/map)\n\nExecute a pipeline over an iterable parameter.\n\n[![](https://mermaid.ink/img/pako:eNqVlF1rwjAUhv9KyG4qKNR-3AS2m8nuBgN3Z0Sy5tQG20SSdE7E_76kVVEr2CY3Ied9Tx6Sk3PAmeKACc5LtcsKpi36nlGZFbXciHwfLN79CuWiBLMcEULWGkBSaeosA2OCxbxdXMd89Get2bZASsLiSyuvQE2mJZXIjW27t2rOmQZ3Gp9rD6UjatWnwy7q6zPPukd50WTydmemEiS_QbQ79RwxGoQY9UaMuojRA8TCXexzyHgQZNwbMu5Cxl3IXNX6OWMyiDHpzZh0GZMHjOK3xz2mgxjT3oxplzG9MPp5_nVOhwJjteDwOg3HyFj3L1dCcvh7DUc-iftX18n6Waet1xX8cG908vpKHO6OW7cvkeHm5GR2b3drdvaSGTODHLW37mxabYC8fLgRhlfxpjNdwmEets-Dx7gCXTHBXQc8-D2KbQEVUEzckjO9oZjKo9Ox2qr5XmaYWF3DGNdbzizMBHOVVWGSs9K4XeDCKv3ZttSmsx7_AYa341E?type=png)](https://mermaid.live/edit#pako:eNqVlF1rwjAUhv9KyG4qKNR-3AS2m8nuBgN3Z0Sy5tQG20SSdE7E_76kVVEr2CY3Ied9Tx6Sk3PAmeKACc5LtcsKpi36nlGZFbXciHwfLN79CuWiBLMcEULWGkBSaeosA2OCxbxdXMd89Get2bZASsLiSyuvQE2mJZXIjW27t2rOmQZ3Gp9rD6UjatWnwy7q6zPPukd50WTydmemEiS_QbQ79RwxGoQY9UaMuojRA8TCXexzyHgQZNwbMu5Cxl3IXNX6OWMyiDHpzZh0GZMHjOK3xz2mgxjT3oxplzG9MPp5_nVOhwJjteDwOg3HyFj3L1dCcvh7DUc-iftX18n6Waet1xX8cG908vpKHO6OW7cvkeHm5GR2b3drdvaSGTODHLW37mxabYC8fLgRhlfxpjNdwmEets-Dx7gCXTHBXQc8-D2KbQEVUEzckjO9oZjKo9Ox2qr5XmaYWF3DGNdbzizMBHOVVWGSs9K4XeDCKv3ZttSmsx7_AYa341E)\n\n### [Arbitrary nesting](https://astrazeneca.github.io/runnable-core/concepts/nesting/)\nAny nesting of parallel within map and so on.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastrazeneca%2Frunnable","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fastrazeneca%2Frunnable","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastrazeneca%2Frunnable/lists"}