{"id":22045101,"url":"https://github.com/michielvdvelde/spider","last_synced_at":"2025-08-23T19:06:42.485Z","repository":{"id":238139062,"uuid":"795959499","full_name":"MichielvdVelde/spider","owner":"MichielvdVelde","description":"Task Workflow Engine for the Web","archived":false,"fork":false,"pushed_at":"2024-05-05T12:02:51.000Z","size":34,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T06:34:11.227Z","etag":null,"topics":["graph","tasks","web"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MichielvdVelde.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-04T14:11:46.000Z","updated_at":"2024-05-05T12:02:54.000Z","dependencies_parsed_at":"2024-11-30T13:11:56.932Z","dependency_job_id":"1eab0e87-d5d7-41bb-8afd-4fbc8e1beab0","html_url":"https://github.com/MichielvdVelde/spider","commit_stats":null,"previous_names":["michielvdvelde/spider"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MichielvdVelde/spider","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichielvdVelde%2Fspider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichielvdVelde%2Fspider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichielvdVelde%2Fspider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichielvdVelde%2Fspider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MichielvdVelde","download_url":"https://codeload.github.com/MichielvdVelde/spider/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichielvdVelde%2Fspider/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271764534,"owners_count":24817171,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["graph","tasks","web"],"created_at":"2024-11-30T13:11:25.123Z","updated_at":"2025-08-23T19:06:42.459Z","avatar_url":"https://github.com/MichielvdVelde.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spider - Task Workflow Engine for the Web\n\n\u003e 🚧 **This project is a work in progress. It is currently unusable.** 🚧\n\nSpider is a TypeScript framework designed to orchestrate complex workflows where\ntasks may depend on each other's outputs. It efficiently handles asynchronous\ntask execution, dependency management, and resource allocation, making it\nsuitable for environments that demand robust management of interdependent tasks.\n\n## Features\n\n- **Asynchronous Task Execution**: Executes tasks asynchronously, supporting\n  I/O-bound or compute-intensive operations efficiently.\n- **Dependency Management**: Define clear dependencies between tasks, either as\n  singular dependencies or groups, to ensure tasks execute in the correct order.\n- **Dynamic Task Resolution**: Tasks execute based on the readiness determined\n  by their dependencies.\n- **Resource Pooling**: Manages task execution concurrency through a runner\n  pool, optimizing system resource usage.\n- **Abort Capability**: Allows workflows to be aborted at any moment, providing\n  control to stop execution gracefully.\n- **Error Handling**: Manages task failures and propagates errors appropriately,\n  ensuring the stability of the entire workflow.\n- **Shared Array Buffers**: Utilizes shared array buffers for efficient data\n  sharing between tasks, reducing memory overhead and improving performance.\n\n## Usage\n\n### Start the Engine\n\nSpider's engine runs within a Shared Worker, ensuring that UI operations are not\nblocked by heavy processing tasks. Being in a Shared Worker also means the\nengine is shared across all tabs from the same origin. To start the engine,\nsimply call the `start` or `waitForReady` function (they are functionally\nequivalent).\n\n```ts\nimport { start, waitForReady } from \"./api\";\n\nawait start();\n\n// or\n\nawait waitForReady();\n```\n\n### Understanding Workflows\n\nA workflow is a structured sequence of tasks that are designed to achieve a\nspecific outcome. Each workflow is a collection of tasks that can have\ndependencies on the outputs of other tasks. This structured approach allows you\nto define complex processes where the order of operations matters, ensuring that\neach task is executed only after its prerequisites are met.\n\n### Defining Tasks\n\nTasks are the fundamental units of work within a Spider workflow. Each task\nrepresents a discrete operation or process that performs a specific function.\nTasks in Spider are defined with unique identifiers, types, dependencies, and\nconfiguration options. The type of a task determines what the task does and how\nit interacts with other tasks or external systems. Dependencies are used to\nspecify which tasks must be completed before the current task can start, thereby\nenforcing the correct order of task execution within the workflow.\n\n### Defining a Workflow\n\nWorkflows are defined using the `WorkflowDescriptor` type, encapsulating the\nstructure and dependencies of your tasks. Each task includes:\n\n- **id**: Unique identifier for the task.\n- **type**: Identifies the type of task to execute. Corresponding task types\n  must be registered with the engine.\n- **dependencies**: Maps dependency keys to task IDs. Dependencies can be\n  individual tasks or groups of tasks.\n- **config**: Task-specific configuration options.\n\nA global configuration can also be set for the entire workflow through a\n`config` object.\n\n### Execution in Runners\n\nTo execute tasks, Spider utilizes runners that operate as Web Workers. Each\nrunner can execute one task at a time, receiving task details from the engine,\nhandling their execution, and managing their lifecycles from initiation to\ncompletion. Spider manages resource use, scales processing capabilities, and\nmaintains responsive user interfaces, even under heavy load conditions.\n\n### Example Usage\n\nSpider aims to provide a simple and intuitive API for defining and executing\nworkflows. Below is an example of how you can define a workflow and execute it\nusing Spider's API.\n\n```typescript\nimport { executeWorkflow, type WorkflowDescriptor } from \"./api\";\n\n// Define a workflow\nconst workflow: WorkflowDescriptor = {\n  tasks: [\n    {\n      id: \"a\",\n      type: \"task1\",\n      dependencies: null, // no dependencies\n      config: null, // no configuration\n    },\n    {\n      id: \"b\",\n      type: \"task2\",\n      dependencies: {\n        a: \"a\", // a single dependency\n        b: [\"a\"], // a dependency group\n      },\n      config: { // configuration\n        key: \"value\",\n      },\n    },\n  ],\n  config: { // workflow configuration\n    key: \"value\",\n  },\n};\n\nconst ac = new AbortController();\nconst signal = ac.signal;\n\n// Execute the workflow\nfor await (const result of executeWorkflow(workflow, { signal })) {\n  console.log(\"Workflow result:\", result);\n  console.log(\"Error:\", result.error ?? \"None\");\n  console.log(\"Finished:\", result.type === \"workflow:result\" \u0026\u0026 result.finish);\n  // Use `ac.abort()` to abort the workflow if needed\n}\n```\n\nSpider’s `executeWorkflow` function returns an async generator that yields both\nintermediate and final results of the workflow execution. This feature allows\nyou to handle results asynchronously as they become available, integrating\nsmoothly with different application architectures.\n\n---\n\nMade with ❤️. [MIT Licensed](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichielvdvelde%2Fspider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichielvdvelde%2Fspider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichielvdvelde%2Fspider/lists"}