{"id":19591915,"url":"https://github.com/catalyst/moodle-tool_dataflows","last_synced_at":"2025-07-09T11:07:03.256Z","repository":{"id":36965041,"uuid":"478333436","full_name":"catalyst/moodle-tool_dataflows","owner":"catalyst","description":"A generic workflow and processing engine which can be configured to do a large variety of tasks.","archived":false,"fork":false,"pushed_at":"2025-07-08T05:19:46.000Z","size":4674,"stargazers_count":15,"open_issues_count":227,"forks_count":13,"subscribers_count":24,"default_branch":"MOODLE_405_STABLE","last_synced_at":"2025-07-08T06:52:19.903Z","etag":null,"topics":["etl","moodle","moodle-plugin","workflow"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/catalyst.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-04-05T23:30:19.000Z","updated_at":"2025-07-08T03:18:52.000Z","dependencies_parsed_at":"2023-10-25T02:31:33.314Z","dependency_job_id":"c438769a-56e7-4230-9d01-6b4e74c612ed","html_url":"https://github.com/catalyst/moodle-tool_dataflows","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/catalyst/moodle-tool_dataflows","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catalyst%2Fmoodle-tool_dataflows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catalyst%2Fmoodle-tool_dataflows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catalyst%2Fmoodle-tool_dataflows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catalyst%2Fmoodle-tool_dataflows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/catalyst","download_url":"https://codeload.github.com/catalyst/moodle-tool_dataflows/tar.gz/refs/heads/MOODLE_405_STABLE","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catalyst%2Fmoodle-tool_dataflows/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264446716,"owners_count":23609632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","moodle","moodle-plugin","workflow"],"created_at":"2024-11-11T08:31:44.572Z","updated_at":"2025-07-09T11:07:03.232Z","avatar_url":"https://github.com/catalyst.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# moodle-tool_dataflows\n\n\u003ca href=\"https://github.com/catalyst/moodle-tool_dataflows/actions\"\u003e\n\u003cimg src=\"https://github.com/catalyst/moodle-tool_dataflows/workflows/ci/badge.svg\"\u003e\n\u003c/a\u003e\n\n- [moodle-tool_dataflows](#moodle-tool_dataflows)\n  - [What is this?](#what-is-this)\n  - [Branches](#branches)\n  - [Alternatives](#alternatives)\n    - [tool_etl](#tool_etl)\n    - [tool_trigger](#tool_trigger)\n    - [local_webhooks](#local_webhooks)\n  - [Installation](#installation)\n    - [Dependencies](#dependencies)\n  - [Configuration](#configuration)\n    - [Best practices for workflows](#best-practices-for-workflows)\n  - [Guides](#guides)\n  - [Support](#support)\n  - [Warm thanks](#warm-thanks)\n    - [University College London](#university-college-london)\n    - [NSW Department of Education](#nsw-department-of-education)\n\n## What is this?\n\nDataflows is a generic workflow and processing engine which can be configured to do a large variety of tasks.\n\n\n## Branches\n\n| Moodle version | Branch            | PHP       |\n|----------------|-------------------|-----------|\n| Moodle 4.1-4.2 | MOODLE_401_STABLE | 7.4       |\n| Moodle 4.5     | MOODLE_405_STABLE | 8.1 - 8.3 |\n| Totara 10+     | MOODLE_35_STABLE  | 7.1 - 7.4 |\n\nNote: Moodle 402 is supported with PHP 8.0 maximum at the moment\n\n## Alternatives\n\n### tool_etl\n\nThis was our original plugin which was more tightly focus on just the ETL use case. Long term we expect\ntool_etl to be deprecated in favor of tool_dataflows as it matures.\n\nhttps://github.com/catalyst/moodle-tool_etl\n\n### tool_trigger\n\nTrigger is focused on a very narrow use case of handling a workflow which starts with a Moodle\nevent and has it's own workflow engine conceptually similar to dataflow. Dataflows has an\n'event trigger' step and is super set of the tool_trigger functionality.\n\nhttps://github.com/catalyst/moodle-tool_trigger/\n\n### local_webhooks\n\nThis is very similar to tool_trigger in that can only work with moodle events and the only action\nit can take is a curl call in a fairly specific shape. It does not support retries, timeouts and\ncannot do asynchronous trigger so has a performance impact on pages which trigger events.\n\nhttps://github.com/valentineus/moodle-webhooks\n\n\n## Installation\n\nFrom Moodle siteroot:\n\n```\ngit clone git@github.com:catalyst/moodle-tool_dataflows.git admin/tool/dataflows\n```\n\n### Dependencies\n\nMake sure graphviz is installed\n\n```\napt install graphviz\n```\n\nhttps://graphviz.org/documentation/\n\n## Configuration\n\nThere are a few concepts to understand how the dataflows plugins works:\n\n1) Dataflows, are a collections of Steps which perform a series of actions\n2) There are 3 classes of steps 'Connector steps', 'Flow steps' and 'Trigger steps'\n3) There are many types of steps in each class, eg curl connector, copy connector, directory read connector\n4) A flow can have 0 or 1 Trigger step, and this is what starts the dataflow execution. eg you might have a 'Cron trigger', or an 'Event trigger'. If a dataflow does not have a Trigger step then it can only ever be run manually.\n5) A Connector Step only ever runs once, for example a step which copies a file from A to B\n6) A flow step is a step which runs in a loop over a stream of data. So you could have a flow step which make a curl call for every row in a csv file\n7) Each type of step defines what inputs it accepts and what outputs it creates. It may have a 'connector' input, and a 'flow' output. For instance the various 'reader' steps are connectors that have an output of a 'flow', eg 'CSV reader', 'JSON reader', 'SQL reader'\n8) Some triggers are also a flow step combined, for instance the event trigger can listen for events and buffer them and then trigger the flow to execute a series of events as a batch (it can also operate one at a time as well).\n9) Almost all steps require configuration, such as the name of a file to read, or the url to curl\n10) When authoring a dataflow you assemble all the steps together and link them into a graph of the execution order. Some steps can have multiple outputs like a unix 'tee' and some steps can have conditional multiple outputs like an 'if' or 'case' statement.\n11) Each step can expose different variables when it executes and these are stored in its own step namespace so they don't clash.\n12) When wiring steps together you can use any variable in expression written in the symphony expression language. For instance you could read a csv file which populates a flow record, and then use these values in a curl call to an api. Each step dynamically documents what variables it exposes.\n13) The dataflow engine validates that the steps are all wired together in a way that makes sense, and you cannot run a dataflow if it is in an invalid state. But invalid states are allowed to ease the authoring process.\n14) Dataflows can be enabled and disabled, and can be exported and imported and also locked after authoring so they cannot be tampered with.\n\nThe best way is to see some example flows in action. TBA add some fixture flows to repo\n\n\n### Best practices for workflows\n\n## Guides\n\n* [Creating a new step type](./NEW_STEP.md)\n\n\n## Support\n\nIf you have issues please log them in\n[GitHub](https://github.com/catalyst/moodle-tool_dataflows/issues).\n\nPlease note our time is limited, so if you need urgent support or want to\nsponsor a new feature then please contact\n[Catalyst IT Australia](https://www.catalyst-au.net/contact-us).\n\n\n## Warm thanks\n\nThanks to various organisations for support in developing this plugin:\n\n### University College London\n![image](https://user-images.githubusercontent.com/187449/180128782-474fcdab-62c5-4848-ab6b-92ff4ece5d6f.png)\n\n### NSW Department of Education\nFor sponsoring the development of the event trigger functionality.\n\n\n![NSW DET](https://user-images.githubusercontent.com/17095477/201774199-aa1d2ce9-eccf-4aca-ab69-2fef75971ae1.png)\n\nThis plugin was developed by [Catalyst IT Australia](https://www.catalyst-au.net/).\n\n\u003cimg alt=\"Catalyst IT\" src=\"https://cdn.rawgit.com/CatalystIT-AU/moodle-auth_saml2/MOODLE_39_STABLE/pix/catalyst-logo.svg\" width=\"400\"\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcatalyst%2Fmoodle-tool_dataflows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcatalyst%2Fmoodle-tool_dataflows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcatalyst%2Fmoodle-tool_dataflows/lists"}