{"id":39584718,"url":"https://github.com/babbel/floto","last_synced_at":"2026-01-26T15:01:19.431Z","repository":{"id":57431466,"uuid":"51018480","full_name":"babbel/floto","owner":"babbel","description":"Task Orchestration Tool Based on SWF and boto3","archived":true,"fork":false,"pushed_at":"2018-10-06T20:17:30.000Z","size":267,"stargazers_count":38,"open_issues_count":3,"forks_count":9,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-08-30T03:28:53.752Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/babbel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-02-03T17:49:19.000Z","updated_at":"2023-01-28T11:33:52.000Z","dependencies_parsed_at":"2022-09-02T11:51:28.681Z","dependency_job_id":null,"html_url":"https://github.com/babbel/floto","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/babbel/floto","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/babbel%2Ffloto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/babbel%2Ffloto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/babbel%2Ffloto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/babbel%2Ffloto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/babbel","download_url":"https://codeload.github.com/babbel/floto/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/babbel%2Ffloto/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28781308,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T13:55:28.044Z","status":"ssl_error","status_checked_at":"2026-01-26T13:55:26.068Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-18T07:35:27.099Z","updated_at":"2026-01-26T15:01:19.425Z","avatar_url":"https://github.com/babbel.png","language":"Python","funding_links":[],"categories":["\u003ca name=\"Python\"\u003e\u003c/a\u003ePython"],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/babbel/floto.svg?branch=master)](https://travis-ci.com/babbel/floto)\n\n# floto\nfloto is a task orchestration tool based on AWS SWF (Simple Workflow Service) written in Python. It uses Python 3 and boto3, the AWS SDK for Python.\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n**Table of Contents**\n\n- [Introduction](#introduction)\n- [Defining the Workflow's Logic](#defining-the-workflows-logic)\n  - [Tasks](#tasks)\n    - [Activity Task](#activity-task)\n    - [Generator](#generator)\n    - [ChildWorkflow](#childworkflow)\n    - [Timer](#timer)\n    - [Retry Strategy](#retry-strategy)\n- [Decider](#decider)\n  - [Dynamic Decider](#dynamic-decider)\n  - [Decider Daemon](#decider-daemon)\n  - [JSON Representation of Decider Specifications](#json-representation-of-decider-specifications)\n- [Activities](#activities)\n  - [Activity](#activity)\n  - [Generator](#generator-1)\n  - [Activity Context](#activity-context)\n  - [Activity Result](#activity-result)\n- [Activity Worker](#activity-worker)\n  - [Activity Worker Heartbeats](#activity-worker-heartbeats)\n- [floto's simple SWF API](#flotos-simple-swf-api)\n  - [Interface to SWF](#interface-to-swf)\n  - [Start the Workflow](#start-the-workflow)\n  - [Register Domains, Workflow Type and Activity Type](#register-domains-workflow-type-and-activity-type)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n## Introduction\nThe \u003ca href=\"https://aws.amazon.com/swf/\" target=\"_blank\"\u003eAWS Simple Workflow Service\u003c/a\u003e allows to \nmanage distributed applications in a scalable, resilient and fault tolerant way.\nIt minimizes the complexity by decoupling the execution logic from the application worker. The \nDeciders, which handle the execution logic and the worker are stateless and therefore fault \ntolerant. Whenever something goes wrong the Deciders and worker can be restarted and pick up their \nwork where they left off. Furthermore several Deciders and worker of the same kind can be run at \nthe same time without interference of the workflow execution or result which again leads to \nhigher resilience and scalability. Every step of a workflow execution is recorded by SWF and the \nhistory of events is provided to the Deciders when they are about to schedule tasks.\n\nThe process of implementing a SWF workflow can be somewhat tedious if you want to e.g. \nhandle complex execution logic and treat task failures and time-outs.\nfloto solves this problem by providing a Python package which allows you to easily define the \nexecution logic and activity worker.\nFor the impatient we provide a [\"Getting started example\"](examples/hello_world.py) of a\nsimple workflow.\nThe example shows the definition of a simple workflow with a single task. The task is defined and passed to the Decider. Furthermore an activity is defined so that the worker is able to executes the activity function on request. The Decider and the worker are started and the workflow execution is initiated. The single steps to define the components necessary to execute a workflow are discussed in more detail in the next sections.\n\n## Defining the Workflow's Logic\nThe business logic of your distributed application is handled by so called Deciders. Deciders act on events like workflow start, task completion or task failure and schedule tasks that are to be executed. The logic itself is defined by a list of tasks. The tasks are then [passed to the decider](#decider).\nLet's get started with a simple example of three activities as depicted in figure 1.  In this example ``ActivityA`` and ``ActivityB`` are scheduled after the workflow start. ``ActivityC`` is executed once they are completed.\n\n![alt tag](docs/images/decider_spec_01.png)\n\nThe definition of the activity tasks:\n```python\nfrom floto.specs.task import ActivityTask\n\nactivity_task_a = ActivityTask(name='ActivityA', version='v1')\nactivity_task_b = ActivityTask(name='ActivityB', version='v1')\nactivity_task_c = ActivityTask(name='ActivityC', version='v1', requires=[activity_task_a, activity_task_b])\n```\nfloto provides different kinds of tasks which can be used to define the workflow logic. In the aforementioned example objects of type ``ActivityTask`` have been used. Furthermore there are Timer, ChildWorkflow and Generator tasks which are described in the following sections.\n### Tasks\nThe tasks are the building blocks of the execution logic. All tasks implement the ``floto.specs.task.Task`` interface, which has the fields ``id_`` and ``requires``. The id of task must be unique on the workflow level. For ``Timer`` objects it has to be set explicitly. For the other tasks there is a default value which is derived by the object's properties, however it can be set explicitly. This is described in the corresponding sections. Dependencies of the tasks are defined by a list of required tasks. \n#### Activity Task\nActivity tasks are tasks which trigger the execution of activity function by the Decider. ``ActivityTask`` objects have the following properties:\n\n| Parameter | Type | Description |\n| :---         | :---           | :---          |\n| ``name`` [Required]   | ``str``        | The name of the activity. Corresponds to the name of the activity as defined by the [worker](#activity-worker).   |\n| ``version`` [Required]   | ``str``        | The version of the activity. Corresponds to the version of the activity as defined by the [worker](#activity-worker).    |\n| ``activity_id``   | ``str``        | The unique id of the task. Defaults to ``\u003cname:version:hash_id\u003e``. The ``hash_id`` is derived depending on the input and required tasks.    |\n| ``requires``   | ``list``        | List of ``floto.specs.task.Task`` objects, which defines the dependencies.    |\n| ``input``   | ``str``, ``obj``        | The input provided by the task definition. If an object is provided it must be JSON serializable, e.g. of type dict or list. For more information on inputs see section [Activity Context](#activity-context).    |\n| ``retry_strategy``   | ``floto.specs.retry_strategy.Strategy``        | The retry strategy which defines the behavior in case of task failure. See section [Retry Strategy](#retry-strategy)    |\n| ``task_list``   | ``str``        | The task list which is used when the task is scheduled. If not set the default activity task list of the decider is used.    |\n\n#### Generator\n``floto.specs.task.Generator`` inherits from ``ActivityTask`` and implements the same interface. Generators are activities which spawn tasks that are subsequently included in the execution logic. More on generators in section [Generator](#generators). \nTo see how generators work see the ``examples/s3_file_string_length`` example.\n#### ChildWorkflow\nDeciders can start child workflows during execution. See example ``examples/child_workflow``. The following table gives an overview over the child workflow task parameters:\n\n| Parameter | Type | Description |\n| :---         | :---           | :---          |\n| ``workflow_type_name`` [Required]   | ``str``        | The name of the workflow type.   |\n| ``workflow_type_version`` [Required]   | ``str``        | The version of the workflow type.    |\n| ``workflow_id``   | ``str``        | The unique id of the task. Defaults to ``\u003cworkflow_type_name:workflow_type_version:hash_id\u003e``. The ``hash_id`` is derived depending on the input and required tasks.    |\n| ``requires``   | ``list``        | List of ``floto.specs.task.Task`` objects, which defines the dependencies.    |\n| ``input``   | ``str``, ``obj``        | The input provided by the task definition. If an object is provided it must be JSON serializable, e.g. of type dict or list. For more information on inputs see section [Activity Context](#activity-context).    |\n| ``retry_strategy``   | ``floto.specs.retry_strategy.Strategy``        | The retry strategy which defines the behavior in case of task failure. See section [Retry Strategy](#retry-strategy)    |\n| ``task_list``   | ``str``        | The decider task list of the child workflow.    |\n\n#### Timer\nTimers are used to define time-outs. Time-outs can be used inside the execution graph to delay the execution of a subsequent task (figure 2). Secondly they can be used as independent task in order to delay the execution of a subsequent workflow execution (figure 3).\n\nExample task definitions for the delayed execution of ``ActivityB``:\n\n![alt tag](docs/images/decider_spec_02.png)\n\n```python\nactivity_task_a = ActivityTask(name='ActivityA', version='v1')\ntimer_30        = Timer(id_='Timer30', delay_in_seconds=30, requires=[activity_task_a])\nactivity_task_b = ActivityTask(name='ActivityB', version='v1', requires=[timer_30])\n```\n\nExample task definitions for a \"repeated workflow execution\" delay. In this case the workflow does not complete before the ``timer_3600`` times out after one hour.\n\n![alt tag](docs/images/decider_spec_03.png)\n\n```python\nactivity_task_a = ActivityTask(name='ActivityA', version='v1')\nactivity_task_b = ActivityTask(name='ActivityB', version='v1', requires=[activity_task_a])\ntimer_3600      = Timer(id_='Timer3600', delay_in_seconds=3600)\n```\n#### Retry Strategy\nSometimes activities fail or time out. A retry strategy can be defined for ``ActivityTask``, ``Generator`` and ``ChildWorkflow`` objects. In case a strategy is defined, the task is rescheduled after an execution failure. The following example shows a task definition which reschedules the task three times before the workflow fails.\n\n```python\nfrom floto.specs.retry_strategy import InstantRetry\n\nretry_strategy = InstantRetry(retries=3)\nactivity_task = ActivityTask(name='ActivityA', version='v1', retry_strategy=retry_strategy)\n```\n## Decider\nDeciders are the parts of your application which orchestrate the workflow execution. They are defined by means of Decider specifications:\n\n```python\nfrom floto.specs import DeciderSpec\nfrom floto.decider import Decider\n\ndecider_spec = DeciderSpec(domain='your_domain',\n                           task_list='your_decider_task_list',\n                           default_activity_task_list='your_activity_task_list',\n                           activity_tasks=[activity_task_a, activity_task_b, activity_task_c])\n\nDecider(decider_spec=decider_spec).run()\n```\n\nThe following table gives an overview over the decider spec parameters:\n\n| Parameter | Type | Description |\n| :---         | :---           | :---          |\n| ``domain`` [Required]   | ``str``        | The SWF domain.   |\n| ``task_list``   | ``str``        | The decider task list.    |\n| ``activity_tasks``   | ``list``        | List of ``floto.specs.task.Task`` objects, which defines the execution logic.      |\n| ``default_activity_task_list``   | ``str``        | The default task list of the activities. Used when not set explicitly by the task.    |\n| ``repeat_workflow``   | ``bool``        | If ``True``, the Decider restarts the workflow execution after completion.    |\n| ``terminate_workflow_after_completion``   | ``bool``        | If ``True``, the Decider terminates after workflow completion.    |\n\n### Dynamic Decider\nThe ``DynamicDecider`` reads the list of activity tasks from the workflow input. The activity tasks are not provided at the time of the Decider initialization. See example ``examples/dynamic_decider/``.\nThe following code shows the start of the workflow execution of the example. ``activity_tasks`` define the tasks to be executed. \n```python\nworkflow_args = {'domain': 'floto_test', \n                 'workflow_type_name': 's3_files_example',\n                 'workflow_type_version': '1',\n                 'task_list': 's3_files',\n                 'workflow_id': 's3_files',\n                 'input': {'activity_tasks':activity_tasks}}\n\nfloto.api.Swf().start_workflow_execution(**workflow_args)\n```\n### Decider Daemon\nfloto provides a \"daemonized\" service. It is described below how to start a \"Decider daemon\", which acts on signals sent to SWF. \n\nStart the Decider daemon \n```python\nimport floto.decider\nfloto.decider.Daemon(domain='floto_test', task_list='floto_daemon').run()\n```\nStart the \"daemon workflow\" once:\n```python\nimport floto.api\n\nfloto.api.Swf().start_workflow_execution(domain='floto_test', \n        workflow_type_name='floto_daemon_type', workflow_type_version='v1', \n        task_list='floto_daemon', workflow_id='floto_daemon') \n```\n\nThe Daemon acts on signals and starts child workflows and child deciders as specified in the \nDecider Specification.\n\n```python\nimport floto.api\nfrom floto.specs import DeciderSpec    \nfrom floto.specs.task import ActivityTask\n\nactivity_task_a = ActivityTask(name='ActivityA', version='v1') \n\ndecider_spec = floto.specs.DeciderSpec(domain='floto_test',\n             default_activity_task_list='my_activity_task_list',\n             activity_tasks=[activity_task_a])\n\n# Send a signal to the daemon and initiate a child workflow\nfloto.api.Swf().signal_workflow_execution(domain='floto_test', workflow_id='floto_daemon',\n                                          signal_name='startChildWorkflowExecution',\n                                          input={'decider_spec':decider_spec})\n```\nThe difference between the Decider daemon and the DynamicDecider is that the daemon consumes a complete decider spec which allows for the definition of all decider related parameter like task list and domain. On the other hand the workflow which is sent to the daemon is not part of the current workflow, which makes error handling more difficult. \n### JSON Representation of Decider Specifications\nDecider Specifications have a JSON representation, which alternatively can be passed to a \n``Decider``. To retrieve the JSON representation of a decider spec call the ``to_json()`` of the\nspec object.\n\n```JSON\n{\n    \"activity_tasks\": [\n        {\n            \"id_\": \"simple_activity:v1:2be88ca424\",\n            \"name\": \"simple_activity\",\n            \"type\": \"floto.specs.task.ActivityTask\",\n            \"version\": \"v1\"\n        }\n    ],\n    \"default_activity_task_list\": \"hello_world_atl\",\n    \"domain\": \"floto_test\",\n    \"repeat_workflow\": false,\n    \"task_list\": \"simple_decider\",\n    \"terminate_decider_after_completion\": true,\n    \"type\": \"floto.specs.DeciderSpec\"\n}\n```\n## Activities\nThe activity worker are the programs which perform the actual work, e.g. data cleansing, database updates or or data processing. In floto ``ActivityWorker`` objects are initiated and started. The worker are triggered by the scheduling of activity tasks by the Deciders. They poll for activity tasks and react with the execution of the corresponding activity. The activities which the worker can handle, react on and run are defined beforehand. The Activities are defined by means of ``@floto.activity`` and ``@floto.generator`` decorators. \n\n### Activity\nThe following code example show the definition of two activity functions:\n```python\nimport floto\n\n@floto.activity(name='ActivityA', version='v1')\ndef activity_a(context):\n    print('Running ActivityA')\n    print(context)\n    return {'your':'result_activity_a'}\n\n@floto.activity(name='ActivityB', version='v1')\ndef activity_b():\n    print('Running ActivityB')\n    return {'your':'result_activity_b'}\n```\n\n``name`` and ``version`` are handed over to the decorator and must correspond to ``name`` and ``version`` of the  ``ActivityTask`` defined in the Decider logic in order to get executed. The activity itself can have a ``context`` parameter which provides input to the function (See [Activity Context](#activity-context)). \n### Generator\nGenerators are special kinds of activities which must return a list of activity tasks. These activity tasks are subsequently included in the execution logic, i.e. a generator is able to spawn tasks which e.g. depend on the input of the activity function.\n\nThe following code shows the generator from the ``examples/s3_file_string_length`` example.\n```\n@floto.generator(name='weekDays', version='1')\ndef week_days(context):\n    from_date_iso = context['workflow']['from_date']\n    to_date_iso = context['workflow']['to_date']\n    from_date = dt.datetime.strptime(from_date_iso, '%Y-%m-%d').date()\n    to_date = dt.datetime.strptime(to_date_iso, '%Y-%m-%d').date()\n   \n    days = [from_date + dt.timedelta(days=n) for n in range(0, (to_date-from_date).days+1)]\n    week_days = [day for day in days if day.weekday()\u003c5]\n\n    def get_tasks(date):\n        rs = floto.specs.retry_strategy.InstantRetry(retries=3)\n        copy_file = ActivityTask(name='copyFile', version='1', input=date.isoformat(), \n                retry_strategy=rs)\n        length = ActivityTask(name='fileLength', version='1', requires=[copy_file], \n                retry_strategy=rs)\n        return [copy_file, length]\n\n    tasks = [get_tasks(date) for date in week_days]\n    tasks = [t for sublist in tasks for t in sublist] \n    return tasks\n```\n### Activity Context\nThe context variable provides information for the activities. The information that is sent around is limited in size. You should not think of it as real input data to a CPU intense process, but instead of e.g. paths to this data.\n\nThe inputs that activities get access to through the context objects originate from different\nsources:\n\n**Workflow start:** When an activity is scheduled after the start of a workflow it can access the\nworkflow input (See [Start the workflow](#start-the-workflow)) through ``context['workflow']``\n\n**Other activities:** The activities have access to the results of the activities they depend on. \nIf ``ActivityB`` requires ``ActivityA`` and ``ActivityA`` has returned a result it can access it \nthrough ``context['\u003cid of ActivityA\u003e']``\n\n**ActivityTask definition:** If an input has been defined at the time of the ``floto.specs.task.ActivityTask`` definition  it can be accessed by the activity through ``context['activity_task']``\n\n**ChildWorkflow definition:** If an input has been defined at the time of the ``floto.specs.task.ChildWorkflow`` definition  it can be accessed by the activity through ``context['child_workflow']``\n\n### Activity Result\nThe return values of activity functions are recorded as result of the activities. The result can be ``str`` or JSON serializable objects.\n\nAfter the **successful workflow completion** the results of the preceding activities are collected and recorded in the ``WorkflowExecutionCompleted`` event.\n\nAfter a **failed  worfklow execution** the error messages of the failed activities are collected and recorded in the ``WorkflowExecutionFailed`` event.\n## Activity Worker\nAfter the definition of activities and generators functions a worker is initiated and run with:\n```\nworker = floto.ActivityWorker(domain='floto_test', task_list='your_activity_task_list')\nworker.run()\n```\n### Activity Worker Heartbeats\nActivity workers send heartbeats to SWF. The heartbeat interval is set by:\n```\nfloto.ActivityWorker(domain='floto_test', task_list='your_activity_task_list', heartbeat_in_seconds=90)\n```\nThe default value is 90 seconds. If it is set to 0, no heartbeat is sent.\n## floto's simple SWF API\nFor easier access to the SWF API floto provides functionality throught the ``floto.api`` module.\n### Interface to SWF\nIn order to communicate with SWF create an ``swf`` object:\n```python\nimport floto.api\nswf = floto.api.Swf()\n```\n### Start the Workflow\n```python\nswf.start_workflow_execution(domain='floto_test',    \n                             workflow_type_name=workflow_type.name,    \n                             workflow_type_version=workflow_type.version,    \n                             task_list='decider_task_list',\n                             input='your_input')\n```\n\n### Register Domains, Workflow Type and Activity Type\n```python\n\n# Register a domain\nswf.domains.register_domain('floto_test')\n\n# Define and register a workflow type.\nworkflow_type = floto.api.WorkflowType(domain='floto_test', name='my_workflow_type', version='v1')\nswf.register_workflow_type(workflow_type)\n\n# Define and register an activity type\nactivity_type = floto.api.ActivityType(domain='floto_test', name='simple_activity', version='v1')\nswf.register_activity_type(activity_type)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbabbel%2Ffloto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbabbel%2Ffloto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbabbel%2Ffloto/lists"}