{"id":19430733,"url":"https://github.com/alpacatechjp/alexflow","last_synced_at":"2025-02-25T05:43:58.973Z","repository":{"id":56606511,"uuid":"286792981","full_name":"AlpacaTechJP/alexflow","owner":"AlpacaTechJP","description":"ALEXFlow is a python workflow library built for reproducible complex workflow","archived":false,"fork":false,"pushed_at":"2020-10-29T06:18:51.000Z","size":72,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-01-07T20:14:41.435Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AlpacaTechJP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-11T16:23:07.000Z","updated_at":"2024-01-27T01:45:50.000Z","dependencies_parsed_at":"2022-08-15T21:50:23.611Z","dependency_job_id":null,"html_url":"https://github.com/AlpacaTechJP/alexflow","commit_stats":null,"previous_names":["alpacadb/alexflow"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlpacaTechJP%2Falexflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlpacaTechJP%2Falexflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlpacaTechJP%2Falexflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlpacaTechJP%2Falexflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AlpacaTechJP","download_url":"https://codeload.github.com/AlpacaTechJP/alexflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240612535,"owners_count":19829027,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T14:26:20.439Z","updated_at":"2025-02-25T05:43:58.944Z","avatar_url":"https://github.com/AlpacaTechJP.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# alexflow\nALEXFlow is a python workflow library built for reproducible complex workflow, mainly for machine learning training.\n\n\n## Get Started\n\nFor the installation from pypi, simply install via pip.\n\n`pip install alexflow` \n\n\n## Remarks\n\n##### Support of type hints with `dataclasses`\n\nluigi does not work well with type hints, which makes it difficult to build workflow when it is complex. With use of dataclasses, we'd like to gain benefit of type hints.\n\n##### Build workflow by composition, rather than parameter bucket relies.\n\nParameter bucket rely finally build a huge global state at the entrypoint of workflow, which is pretty difficult to maintain in general as it is works similarly with global variables... Instead, we've decided to compose workflow with compositions. With this architecture we can gain the benefit of divide and conquer strategy.\n\n##### Focus of reproducibility with immutability tasks\n\nTask class is designed to be a immutable dataclass object, for distributed execution, strong consistency, and reproducibility. And also those `Task` objects can be serialized as json object, and you can easily trace the exact parameters used to generate the `Output`. \n\n##### Dependency via Outputs, rather than Tasks\n\nDescription of workflow dependency by `Output` makes it easy to run partially graph.\n\n## A exmaple of Task construction\n\nAlso you can see the example workflow at `examples/workflow.py`.\n\n```python\nfrom typing import Tuple\nfrom sklearn import linear_model\nfrom dataclasses import dataclass, field\nfrom alexflow import Task, no_default, NoDefaultVar, Output, BinaryOutput\n\n\n@dataclass(frozen=True)\nclass Train(Task):\n    # Here you can write parameter of task as dataclass fields. Task's unique id will be \n    # generated from given parameters' and each task is executed at once while the entire\n    # graph computation.\n    X: NoDefaultVar[Output] = no_default\n    y: NoDefaultVar[Output] = no_default\n    model_type: NoDefaultVar[str] = no_default\n    # Here you can describe in-significant parameter with compare=False, with following\n    # dataclass' object equality. Even you changed those variables, Task's unique id is\n    # consistent.\n    verbose: bool = field(default=True, compare=False)\n\n    def input(self):\n        \"\"\"Here describes the dependent output of your task\"\"\"\n        return self.X, self.y\n\n    def output(self):\n        \"\"\"Here describes the dependent output of your task\"\"\"\n        return self.build_output(BinaryOutput, key=\"model.pkl\")\n\n    def run(self, input: Tuple[BinaryOutput, BinaryOutput], output: BinaryOutput):\n        # Dependent output you defined in `input()` method is available as input variable.\n        X = input[0].load()\n        y = input[1].load()\n\n        model_class = getattr(linear_model, self.model_type)\n\n        cls = model_class().fit(X, y)\n        \n        # And you can store what you want to output in following manner.\n        output.store(cls)\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpacatechjp%2Falexflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falpacatechjp%2Falexflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpacatechjp%2Falexflow/lists"}