{"id":26148878,"url":"https://github.com/skalar/soflow","last_synced_at":"2026-05-13T13:52:02.967Z","repository":{"id":30281404,"uuid":"84038828","full_name":"Skalar/soflow","owner":"Skalar","description":"Easily run distributed workflows with AWS Simple Workflow Service and Lambda","archived":false,"fork":false,"pushed_at":"2024-05-14T08:03:17.000Z","size":852,"stargazers_count":3,"open_issues_count":24,"forks_count":0,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-05-15T13:51:07.763Z","etag":null,"topics":["aws","lambda","swf"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Skalar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-06T06:37:54.000Z","updated_at":"2024-05-14T08:01:53.000Z","dependencies_parsed_at":"2024-05-14T08:48:18.946Z","dependency_job_id":"4847a5e8-67c6-43fd-9eaa-5b5e3e179a1f","html_url":"https://github.com/Skalar/soflow","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Skalar%2Fsoflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Skalar%2Fsoflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Skalar%2Fsoflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Skalar%2Fsoflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Skalar","download_url":"https://codeload.github.com/Skalar/soflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242975435,"owners_count":20215459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","lambda","swf"],"created_at":"2025-03-11T05:22:01.037Z","updated_at":"2026-05-13T13:51:57.920Z","avatar_url":"https://github.com/Skalar.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# soflow\n\nEasily run distributed workflows with AWS [Simple Workflow Service](https://aws.amazon.com/swf/) and [Lambda](https://aws.amazon.com/lambda/)\n\n## Table of contents\n\n* [Installation](#installation)\n* [Basic usage](#basic-usage)\n  * [Defining tasks](#defining-tasks)\n  * [Defining workflows](#defining-workflows)\n  * [Deploying AWS resources](#deploying-aws-resources)\n  * [Running workers](#running-workers)\n  * [Executing workflow](#executing-workflow)\n  * [Terminating workflow executions](#terminating-workflow-executions)\n  * [Tearing down AWS resources](#tearing-down-aws-resources)\n  * [Executing workflow without AWS](#executing-workflow-without-aws)\n* [Configuration](#configuration)\n* [API docs](#api-docs)\n* [Development](#development)\n  * [Starting dev-environment](#starting-dev-environment)\n  * [Running tests](#running-tests)\n\n## Installation\n\n\u003e A minimum of node 10 is required\n\n```bash\nyarn add soflow\n```\n\n## Basic usage\n\n\u003e You can find implementation examples over at [Skalar/soflow-examples](https://github.com/Skalar/soflow-examples)\n\n### Defining tasks\n\n\u003e tasks.js\n\n```javascript\nasync function addOrderToDatabase(data) {\n  // do your stuff\n  return result\n}\n\naddOrderToDatabase.config = {\n  concurrency: 100, // ReservedConcurrentExecutions in lambda context\n  type: 'both', // deploy as lambda function and register activity type\n  memorySize: 128, // only enforced by lambda\n  scheduleToStartTimeout: 10, // only applies when run as activity\n  startToCloseTimeout: 60,\n  scheduleToCloseTimeout: 20, // only applies to activities\n}\n\nexports.addOrderToDatabase = addOrderToDatabase\n```\n\n### Defining workflows\n\n\u003e workflows.js\n\n```javascript\nasync function CreateOrder({\n  input: {confirmationEmailRecipient, products, customerId},\n  tasks: {addOrderToDatabase, sendOrderConfirmation},\n}) {\n  const orderData = await addOrderToDatabase(customerId, products)\n  await sendOrderConfirmation(orderData)\n\n  return orderData\n}\n\nCreateOrder.config = {\n  startToCloseTimeout: 30,\n  tasks: {\n    addOrderToDatabase: {type: 'faas'},\n    sendOrderConfirmation: {type: 'activity'},\n  },\n}\n\nexports.CreateOrder = CreateOrder\n```\n\n### Deploying AWS resources\n\n```javascript\nconst {SWF} = require('soflow')\n\nconst deployPromise = SWF.Orchestration.setup({\n  progressIndicator: true   // default: false\n  deciderEnvironment: {\n    // environment variables available in lambda decider worklow functions\n    MY_CUSTOM_ENVIRONMENT_VARIABLE: 'myvalue', \n  }\n  // File glob patterns to include in the lambda package.\n  // Everything needed by your tasks must be included (including the soflow npm module).\n  // Default: [`${tasksPath}/**`, `${workflowsPath}/**`]\n  files: [\n    [\n      'stripped_node_modules/**',\n      // Provided callback can return a new filename or true/false for whether to include the file\n      path =\u003e path.replace('stripped_node_modules', 'node_modules')\n    ],\n    'workflows/**',\n    'tasks/**',\n    'lib/**',\n  ],\n})\n```\n\n### Running workers\n\n#### Decider worker\n\n\u003e This worker serves as the workflow decider/conductor.\n\n```javascript\n#!/usr/bin/env node\n\nconst {SWF} = require('soflow')\nconst workflows = require('./workflows')\nconst tasks = require('./tasks')\n\nconst deciderWorker = new SWF.DeciderWorker({\n  workflows,\n  tasks,\n  concurrency: 2,\n})\n\ndeciderWorker.on('error', error =\u003e {\n  console.error(error)\n  process.exit(1)\n})\n\ndeciderWorker.start()\n```\n\n#### Activity worker\n\n\u003e This worker executes scheduled activity tasks.\n\n```javascript\n#!/usr/bin/env node\n\nconst {SWF} = require('soflow')\nconst workflows = require('./workflows')\nconst tasks = require('./tasks')\n\nconst activityWorker = new SWF.ActivityWorker({\n  workflows,\n  tasks,\n  concurrency: 2,\n})\n\nactivityWorker.on('error', error =\u003e {\n  console.error(error)\n  process.exit(1)\n})\n\nactivityWorker.start()\n```\n\n#### Lambda decider\n\n\u003e Soflow supports running SWF deciders as Lamda functions.  \n\u003e Due to the nature of Lambda and SWF, the implementation has some important details.\n\n##### Enable lambda decider\n\n\u003e When the lambda decider is enabled, soflow enables a scheduled CloudWatch event rule that triggers the decider lambda function every minute.  \n\u003e The lambda function will run for 65-130 seconds, exiting when there no longer time (60s + 5s slack) to do an empty poll. This is to prevent decision tasks being temporarily \"stuck\".\n\u003e This means between 1 and 2 deciders are running at any given time, each able to handle multiple decision tasks concurrenctly.\n\u003e\n\u003e Note that it can take up to 1 minute for the first invocation to happen.\n\n```javascript\nawait SWF.Orchestration.Lambda.enableDecider()\n```\n\n##### Disable lambda decider\n\n\u003e Disables the CloudWatch event rule. It can take up to 2 minutes for all deciders to be shut down.\n\n```javascript\nawait SWF.Orchestration.Lambda.disableDecider()\n```\n\n##### Manually invoke lambda decider function\n\n\u003e May be used with enableDecider() to ensure a decider is running immediately, or to temporarily scale up the decider capacity.\n\n```javascript\nawait SWF.Orchestration.Lambda.invokeDecider()\n```\n\n##### Manually shut down running lambda deciders\n\n```javascript\nawait SWF.Orchestration.Lambda.shutdownDeciders()\n```\n\n### Executing workflow\n\n```javascript\nconst {SWF} = require('soflow')\n\nasync function startCreateOrderWorkflow() {\n  // Initiate workflow execution\n  const execution = await SWF.executeWorkflow({\n    type: 'CreateOrder',\n    workflowId: 'CreateOrder-12345',\n    input: {productIds: [1, 2, 3]},\n  })\n\n  // Optionally await workflow result\n  const result = await execution.promise\n}\n```\n\n### Terminating workflow executions\n\n```javascript\nconst {SWF} = require('soflow')\n\nasync function terminationExample() {\n  await SWF.terminateAllExecutions() // terminate ALL workflow executions within namespace\n  await SWF.terminateExecution({workflowId: 'myid'})\n}\n```\n\n### Tearing down AWS resources\n\n\u003e Warning: this removes all AWS resources within the given namespace\n\n```javascript\nconst {SWF} = require('soflow')\n\nasync function teardownExample() {\n  await SWF.Orchstration.teardown({\n    removeBucket: true, // default: false\n    progressIndicator: true, // default: false\n  })\n}\n```\n\n### Executing workflow without AWS\n\nSoflow provides a limited LocalWorkflow backend, with the same API as the SWF backend.  \nThis can be useful during development or testing, but be aware that it:\n\n* runs all workflow (decider) functions in the current process\n* does not enforce worklow timeouts\n* only allows workflow signaling within the same process\n* runs tasks in separate child processes, on the local node\n* only enforces task startToCloseTimeout\n* is not able to terminate workflow executions\n\n```javascript\nconst {LocalWorkflow} = require('soflow')\n\nasync function runWorkflowWithoutSWF() {\n  const execution = await LocalWorkflow.executeWorkflow({\n    type: 'CreateOrder'\n    workflowId: 'order-1234'\n    input: {}\n    // ...\n  })\n\n  // Optionally await workflow result\n  const result = await execution.promise\n}\n```\n\n## Configuration\n\n\u003e You can provide configuration as `environment variables`, via `soflow.config` or `passed as an argument` to a soflow function.\n\n```javascript\nconst {SWF, config} = require('soflow')\n\nconfig.update({\n  namespace: 'mynamespace',\n  swfDomain: 'MyDomain'\n})\n\n// above code must be required/invoked before your code that uses soflow.\n\nSWF.executeWorkflow({namespace: 'othernamespace', ...})\n```\n\n| Variable name        | ENV variable                 | Description                                                                                                                                                       |\n| -------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `namespace`          | `SOFLOW_NAMESPACE`           | Prefix for all AWS resources (globally unique) \u003cbr\u003e _default_: undefined                                                                                          |\n| `version`            | `SOFLOW_WORKFLOWS_VERSION`   | Developer-specified workflows version to use \u003cbr\u003e _default_: undefined                                                                                            |\n| `swfDomain`          | `SOFLOW_SWF_DOMAIN`          | Under which AWS SWF Domain to operate \u003cbr\u003e _default_: 'SoFlow'                                                                                                    |\n| `codeRoot`           | `SOFLOW_CODE_ROOT`           | Path to root directory of code to be deployed \u003cbr\u003e _default_: process.cwd()                                                                                       |\n| `tasksPath`          | `SOFLOW_TASKS_PATH`          | Requireable path to tasks, relative to codeRoot \u003cbr\u003e _default_: 'tasks'                                                                                           |\n| `workflowsPath`      | `SOFLOW_WORKFLOWS_PATH`      | Requireable path to workflows, relative to codeRoot \u003cbr\u003e _default_: 'workflows'                                                                                   |\n| `soflowPath`         | `SOFLOW_PATH`                | Requireable path to soflow, relative to codeRoot \u003cbr\u003e _default_: 'node_modules/soflow'                                                                            |\n| `s3Bucket`           | `SOFLOW_S3_BUCKET`           | Name of S3 bucket to for lambda packages \u003cbr\u003e _default_: namespace                                                                                                |\n| `s3Prefix`           | `SOFLOW_S3_PREFIX`           | Key prefix for S3 objects \u003cbr\u003e _default_: 'soflow/'                                                                                                               |\n| `awsRegion`          | `SOFLOW_AWS_REGION`          | Which AWS region to operate in \u003cbr\u003e _default_: 'eu-west-1'                                                                                                        |\n| `executionRetention` | `SOFLOW_EXECUTION_RETENTION` | Number of days to keep workflow executions. \u003cbr\u003e **note**: Can only be set the first time an SWF domain is created, after which it is immutable \u003cbr\u003e _default_: 1 |\n\n## Development\n\n### Starting dev environment\n\n```bash\n# Bring up a local dynamodb and s3 as well as linting every time the code changes.\n\ndocker-compose up --build\n\n# Or you could use the tmux-session script:\n\nln -s $PWD/scripts/tmux-session ~/.bin/soflow\n\nsoflow         # start or resume tmux dev session\n               # brings up linting, unit and integration tests with file watching\n\nsoflow clean   # stops/cleans docker containers, tmux session\n```\n\n### Running tests\n\n\u003e Requires the development environment to be running\n\n```bash\n# Unit and integration tests for all node targets\ndocker-compose exec dev scripts/test\n\n# Unit tests with file watching and verbose output\ndocker-compose exec dev ash -c \\\n  \"scripts/unit-tests --watch --verbose\"\n\n# Integration tests with 'local' profile\ndocker-compose exec dev ash -c \\\n  \"PROFILES=local scripts/integration-tests --verbose\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskalar%2Fsoflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskalar%2Fsoflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskalar%2Fsoflow/lists"}