{"id":17434576,"url":"https://github.com/microsoft/durabletask-go","last_synced_at":"2025-05-14T13:05:54.817Z","repository":{"id":61623856,"uuid":"537192023","full_name":"microsoft/durabletask-go","owner":"microsoft","description":"The Durable Task Framework is a lightweight, embeddable engine for writing durable, fault-tolerant business logic (orchestrations) as ordinary code.","archived":false,"fork":false,"pushed_at":"2025-04-17T14:04:09.000Z","size":364,"stargazers_count":238,"open_issues_count":17,"forks_count":38,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-05-07T23:46:45.449Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microsoft.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-09-15T20:14:43.000Z","updated_at":"2025-04-25T04:22:00.000Z","dependencies_parsed_at":"2023-02-15T05:31:37.663Z","dependency_job_id":"8459daa9-a742-4834-a292-8ef38cdfb1b0","html_url":"https://github.com/microsoft/durabletask-go","commit_stats":{"total_commits":67,"total_committers":11,"mean_commits":6.090909090909091,"dds":0.4477611940298507,"last_synced_commit":"65c308bcd994d28175d9c23555bef557c01e2b6c"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fdurabletask-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fdurabletask-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fdurabletask-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fdurabletask-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microsoft","download_url":"https://codeload.github.com/microsoft/durabletask-go/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254149947,"owners_count":22022851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-17T09:07:11.929Z","updated_at":"2025-05-14T13:05:54.782Z","avatar_url":"https://github.com/microsoft.png","language":"Go","readme":"# Durable Task Framework for Go\n\n[![Build](https://github.com/microsoft/durabletask-go/actions/workflows/pr-validation.yml/badge.svg)](https://github.com/microsoft/durabletask-go/actions/workflows/pr-validation.yml)\n\nThe Durable Task Framework is a lightweight, embeddable engine for writing durable, fault-tolerant business logic (*orchestrations*) as ordinary code. The engine itself is written in Go and intended to be embedded into other Go-based processes. It exposes a gRPC endpoint to support writing durable flows in any language. There are currently SDKs that consume this gRPC endpoint for [.NET](https://github.com/microsoft/durabletask-dotnet) and [Java](https://github.com/microsoft/durabletask-java), with more to come. It's also possible to write orchestrations directly in Go and run them in the local process.\n\nThis project is largely a Go clone of the [.NET-based Durable Task Framework](https://github.com/Azure/durabletask), which is used by various cloud service teams at Microsoft for building reliable control planes and managing infrastructure. It also takes inspiration from the [Go Workflows](https://github.com/cschleiden/go-workflows) project, which itself is a Go project that borrows heavily from both the Durable Task Framework and [Temporal](https://github.com/temporalio/temporal). The main difference is that the Durable Task engine is designed to be used in sidecar architectures.\n\nThe Durable Task engine is also intended to be used as the basis for the [Dapr embedded workflow engine](https://github.com/dapr/dapr/issues/4576).\n\n\u003e This project is a work-in-progress and should not be used for production workloads. The public API surface is also not yet stable. The project itself is also in the very early stages and is missing some of the basics, such as contribution guidelines, etc.\n\n## Storage providers\n\nThis project includes a [sqlite](https://sqlite.org/) storage provider for persisting app state to disk.\n\n```go\n// Persists state to a file named test.sqlite3. Use \"\" for in-memory storage.\noptions := sqlite.NewSqliteOptions(\"test.sqlite3\")\nbe := sqlite.NewSqliteBackend(options, backend.DefaultLogger())\n```\n\nAdditional storage providers can be created by extending the `Backend` interface.\n\n## Creating the standalone gRPC sidecar\n\nSee the `main.go` file for an example of how to create a standalone gRPC sidecar that embeds the Durable Task engine. In short, you must create an `Backend` (for storage), an `Executor` (for executing user code), and host them as a `TaskHubWorker`.\n\nThe following code creates a `TaskHub` worker with sqlite `Backend` and a gRPC `Executor` implementations.\n\n```go\n// Use the default logger or provide your own\nlogger := backend.DefaultLogger()\n\n// Configure the sqlite backend that will store the runtime state\nsqliteOptions := sqlite.NewSqliteOptions(sqliteFilePath)\nbe := sqlite.NewSqliteBackend(sqliteOptions, logger)\n\n// Create a gRPC server that the language SDKs will connect to\ngrpcServer := grpc.NewServer()\nexecutor := backend.NewGrpcExecutor(grpcServer, be, logger)\n\n// Construct and start the task hub worker object, which polls the backend for new work\norchestrationWorker := backend.NewOrchestrationWorker(be, executor, logger)\nactivityWorker := backend.NewActivityTaskWorker(be, executor, logger)\ntaskHubWorker := backend.NewTaskHubWorker(be, orchestrationWorker, activityWorker, logger)\ntaskHubWorker.Start(context.Background())\n\n// Start listening.\nlis, _ := net.Listen(\"tcp\", \"localhost:4001\")\nfmt.Printf(\"server listening at %v\\n\", lis.Addr())\ngrpcServer.Serve(lis)\n```\n\nNote that the Durable Task gRPC service implementation is designed to serve one client at a time, just like with any sidecar architecture. Scale out is achieved by adding new pod replicas that contain both the app process and the sidecar (connected to a common database).\n\n### Language SDKs for gRPC\n\nThe Durable Task Framework for Go currently supports writing orchestrations in the following languages:\n\n| Language/Stack | Package | Project Home | Samples |\n| - | - | - | - |\n| .NET | [![NuGet](https://img.shields.io/nuget/v/Microsoft.DurableTask.Client.svg?style=flat)](https://www.nuget.org/packages/Microsoft.DurableTask.Client/) | [GitHub](https://github.com/microsoft/durabletask-dotnet) | [Samples](https://github.com/microsoft/durabletask-dotnet/tree/main/samples) |\n| Java | [![Maven Central](https://img.shields.io/maven-central/v/com.microsoft/durabletask-client?label=durabletask-client)](https://search.maven.org/artifact/com.microsoft/durabletask-client) | [GitHub](https://github.com/microsoft/durabletask-java) | [Samples](https://github.com/microsoft/durabletask-java/tree/main/samples/src/main/java/io/durabletask/samples) |\n| Python | [![PyPI version](https://badge.fury.io/py/durabletask.svg)](https://badge.fury.io/py/durabletask) | [GitHub](https://github.com/microsoft/durabletask-python) | [Samples](https://github.com/microsoft/durabletask-python/tree/main/examples) |\n\nMore language SDKs are planned to be added in the future. In particular, SDKs for Python and JavaScript/TypeScript. Anyone can theoretically create an SDK using a language that supports gRPC. However, there is not yet a guide for how to do this, so developers would need to reference existing SDK code as a reference. Starting with the Java implementation is recommended. The gRPC API is defined [here](https://github.com/microsoft/durabletask-protobuf).\n\n## Embedded orchestrations\n\nIt's also possible to create orchestrations in Go and run them in the local process. The full set of Durable Task features is not yet available as part of the Go SDK, but will be added over time.\n\n\u003e You can find code samples in the [samples](./samples/) directory.  \n\u003e To run them, get into the folder of each sample and run `go run .`\n\n### Activity sequence example\n\nActivity sequences like the following are the simplest and most common pattern used in the Durable Task Framework.\n\n```go\n// ActivitySequenceOrchestrator makes three activity calls in sequence and results the results\n// as an array.\nfunc ActivitySequenceOrchestrator(ctx *task.OrchestrationContext) (any, error) {\n\tvar helloTokyo string\n\tif err := ctx.CallActivity(SayHelloActivity, task.WithActivityInput(\"Tokyo\")).Await(\u0026helloTokyo); err != nil {\n\t\treturn nil, err\n\t}\n\tvar helloLondon string\n\tif err := ctx.CallActivity(SayHelloActivity, task.WithActivityInput(\"London\")).Await(\u0026helloLondon); err != nil {\n\t\treturn nil, err\n\t}\n\tvar helloSeattle string\n\tif err := ctx.CallActivity(SayHelloActivity, task.WithActivityInput(\"Seattle\")).Await(\u0026helloSeattle); err != nil {\n\t\treturn nil, err\n\t}\n\treturn []string{helloTokyo, helloLondon, helloSeattle}, nil\n}\n\n// SayHelloActivity can be called by an orchestrator function and will return a friendly greeting.\nfunc SayHelloActivity(ctx task.ActivityContext) (any, error) {\n\tvar input string\n\tif err := ctx.GetInput(\u0026input); err != nil {\n\t\treturn \"\", err\n\t}\n\treturn fmt.Sprintf(\"Hello, %s!\", input), nil\n}\n```\n\nYou can find the full sample [here](./samples/sequence).\n\n### Fan-out / fan-in execution example\n\nThe next most common pattern is \"fan-out / fan-in\" where multiple activities are run in parallel, as shown in the snippet below (note that the `GetDevicesToUpdate` and `UpdateDevice` activity definitions are left out of the snippet below for brevity):\n\n```go\n// UpdateDevicesOrchestrator is an orchestrator that runs activities in parallel\nfunc UpdateDevicesOrchestrator(ctx *task.OrchestrationContext) (any, error) {\n\t// Get a dynamic list of devices to perform updates on\n\tvar devices []string\n\tif err := ctx.CallActivity(GetDevicesToUpdate).Await(\u0026devices); err != nil {\n\t\treturn nil, err\n\t}\n\n\t// Start a dynamic number of tasks in parallel, not waiting for any to complete (yet)\n\ttasks := make([]task.Task, len(devices))\n\tfor i, id := range devices {\n\t\ttasks[i] = ctx.CallActivity(UpdateDevice, task.WithActivityInput(id))\n\t}\n\n\t// Now that all are started, wait for them to complete and then return the success rate\n\tsuccessCount := 0\n\tfor _, task := range tasks {\n\t\tvar succeeded bool\n\t\tif err := task.Await(\u0026succeeded); err == nil \u0026\u0026 succeeded {\n\t\t\tsuccessCount++\n\t\t}\n\t}\n\n\treturn float32(successCount) / float32(len(devices)), nil\n}\n```\n\nThe full sample can be found [here](./samples/parallel).\n\n### External orchestration inputs (events) example\n\nSometimes orchestrations need asynchronous input from external systems. For example, an approval workflow may require a manual approval signal from an authorized user. Or perhaps an orchestration pauses and waits for a command from an operator. The `WaitForSingleEvent` method can be used in an orchestrator function to pause execution and wait for such inputs. You an even specify a timeout value indicating how long to wait for the input before resuming execution (use `-1` to indicate infinite timeout).\n\n```go\n// ExternalEventOrchestrator is an orchestrator function that blocks for 30 seconds or\n// until a \"Name\" event is sent to it.\nfunc ExternalEventOrchestrator(ctx *task.OrchestrationContext) (any, error) {\n\tvar nameInput string\n\tif err := ctx.WaitForSingleEvent(\"Name\", 30*time.Second).Await(\u0026nameInput); err != nil {\n\t\t// Timeout expired\n\t\treturn nil, err\n\t}\n\n\treturn fmt.Sprintf(\"Hello, %s!\", nameInput), nil\n}\n```\n\nSending an event to a waiting orchestration can be done using the `RaiseEvent` method of the task hub client. These events are durably buffered in the orchestration state and are consumed as soon as the target orchestration calls `WaitForSingleEvent` with a matching event name. The following code shows how to use the `RaiseEvent` method to send an event with a payload to a running orchestration. See [Managing local orchestrations](#managing-local-orchestrations) for more information on how to interact with local orchestrations in Go.\n\n```go\nid, _ := client.ScheduleNewOrchestration(ctx, ExternalEventOrchestrator)\n\n// Prompt the user for their name and send that to the orchestrator\ngo func() {\n\tfmt.Println(\"Enter your first name: \")\n\tvar nameInput string\n\tfmt.Scanln(\u0026nameInput)\n\t\n\tclient.RaiseEvent(ctx, id, \"Name\", api.WithEventPayload(nameInput))\n}()\n```\n\nThe full sample can be found [here](./samples/externalevents).\n\n### Managing local orchestrations\n\nThe following code snippet provides an example of how you can configure and run orchestrations. The `TaskRegistry` type allows you to register orchestrator and activity functions, and the `TaskHubClient` allows you to start, query, terminate, suspend, resume, and wait for orchestrations to complete.\n\nThe code snippet below demonstrates how to register and start a new instance of the `ActivitySequenceOrchestrator` orchestrator and wait for it to complete. The initialization of the client and worker are left out for brevity.\n\n```go\nr := task.NewTaskRegistry()\nr.AddOrchestrator(ActivitySequenceOrchestrator)\nr.AddActivity(SayHelloActivity)\n\nctx := context.Background()\nclient, worker := Init(ctx, r)\ndefer worker.Shutdown(ctx)\n\nid, err := client.ScheduleNewOrchestration(ctx, ActivitySequenceOrchestrator)\nif err != nil {\n  panic(err)\n}\n\nmetadata, err := client.WaitForOrchestrationCompletion(ctx, id)\nif err != nil {\n  panic(err)\n}\n\nfmt.Printf(\"orchestration completed: %v\\n\", metadata)\n```\n\nEach sample linked above has a full implementation you can use as a reference.\n\n## Distributed tracing support\n\nThe Durable Task Framework for Go supports publishing distributed traces to any configured [Open Telemetry](https://opentelemetry.io/)-compatible exporter. Simply use [`otel.SetTracerProvider(tp)`](https://pkg.go.dev/go.opentelemetry.io/otel#SetTracerProvider) to register a global `TracerProvider` as part of your application startup and the task hub worker will automatically use it to emit OLTP trace spans.\n\nThe following example code shows how you can configure distributed trace collection with [Zipkin](https://zipkin.io/), a popular open source distributed tracing system. The example assumes Zipkin is running locally, as shown in the code.\n\n```go\nfunc ConfigureZipkinTracing() (*trace.TracerProvider, error) {\n\t// Inspired by this sample: https://github.com/open-telemetry/opentelemetry-go/blob/main/example/zipkin/main.go\n\texp, err := zipkin.New(\"http://localhost:9411/api/v2/spans\")\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// NOTE: The simple span processor is not recommended for production.\n\t//       Instead, the batch span processor should be used for production.\n\tprocessor := trace.NewSimpleSpanProcessor(exp)\n\t// processor := trace.NewBatchSpanProcessor(exp)\n\n\ttp := trace.NewTracerProvider(\n\t\ttrace.WithSpanProcessor(processor),\n\t\ttrace.WithSampler(trace.AlwaysSample()),\n\t\ttrace.WithResource(resource.NewWithAttributes(\n\t\t\t\"durabletask.io\",\n\t\t\tattribute.KeyValue{Key: \"service.name\", Value: attribute.StringValue(\"sample-app\")},\n\t\t)),\n\t)\n\totel.SetTracerProvider(tp)\n\treturn tp, nil\n}\n```\n\nYou can find this code in the [distributedtracing](./samples/distributedtracing) sample. The following is a screenshot showing the trace for the sample's orchestration, which calls an activity, creates a 2-second durable timer, and uses another activity to make an HTTP request to bing.com:\n\n![image](https://user-images.githubusercontent.com/2704139/205171291-8d12d6fe-5d4f-40c7-9a48-2586a4c4af49.png)\n\nNote that each orchestration is represented as a single span with activities, timers, and sub-orchestrations as child spans. The generated spans contain a variety of attributes that include information such as orchestration instance IDs, task names, task IDs, etc.\n\n## Cloning this repository\n\nThis repository contains submodules. Be sure to clone it with the option to include submodules. Otherwise you will not be able to generate the protobuf code.\n\n```bash\ngit clone --recurse-submodules https://github.com/microsoft/durabletask-go \n```\n\n## Building the project\n\nThis project requires go v1.19.x or greater. You can build a standalone executable by simply running `go build` at the project root.\n\n### Generating protobuf\n\nUse the following command to regenerate the protobuf from the submodule. Use this whenever updating the submodule reference.\n\n```bash\n# NOTE: assumes the .proto file defines: option go_package = \"/internal/protos\"\nprotoc --go_out=. --go-grpc_out=. -I submodules/durabletask-protobuf/protos orchestrator_service.proto\n```\n\n### Generating mocks for testing\n\nTest mocks were generated using [mockery](https://github.com/vektra/mockery). Use the following command at the project root to regenerate the mocks.\n\n```bash\nmockery --dir ./backend --name=\"^Backend|^Executor|^TaskWorker\" --output ./tests/mocks --with-expecter\n```\n\n## Running tests\n\nAll automated tests are under `./tests`. A separate test package hierarchy was chosen intentionally to prioritize [black box testing](https://en.wikipedia.org/wiki/Black-box_testing). This strategy also makes it easier to catch accidental breaking API changes.\n\nRun tests with the following command.\n\n```bash\ngo test ./tests/... -coverpkg ./api,./task,./client,./backend/...,./internal/helpers\n```\n\n## Running integration tests\n\nYou can run pre-built container images to run full integration tests against the durable task host over gRPC.\n\n### .NET Durable Task client SDK tests\n\nUse the following docker command to run tests against a running worker.\n\n```bash\ndocker run -e GRPC_HOST=\"host.docker.internal\" cgillum/durabletask-dotnet-tester:0.5.0-beta\n```\n\nNote that the test assumes the gRPC server can be reached over `localhost` on port `4001` on the host machine. These values can be overridden with the following environment variables:\n\n* `GRPC_HOST`: Use this to change from the default `127.0.0.1` to some other value, for example `host.docker.internal`.\n* `GRPC_PORT`: Set this environment variable to change the default port from `4001` to something else.\n\nIf successful, you should see output that looks like the following:\n\n```\nTest run for /root/out/bin/Debug/Microsoft.DurableTask.Tests/net6.0/Microsoft.DurableTask.Tests.dll (.NETCoreApp,Version=v6.0)\nMicrosoft (R) Test Execution Command Line Tool Version 17.3.1 (x64)\nCopyright (c) Microsoft Corporation.  All rights reserved.\n\nStarting test execution, please wait...\nA total of 1 test files matched the specified pattern.\n[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (64-bit .NET 6.0.10)\n[xUnit.net 00:00:00.82]   Discovering: Microsoft.DurableTask.Tests\n[xUnit.net 00:00:00.90]   Discovered:  Microsoft.DurableTask.Tests\n[xUnit.net 00:00:00.90]   Starting:    Microsoft.DurableTask.Tests\n  Passed Microsoft.DurableTask.Tests.OrchestrationPatterns.ExternalEvents(eventCount: 100) [6 s]\n  Passed Microsoft.DurableTask.Tests.OrchestrationPatterns.ExternalEvents(eventCount: 1) [309 ms]\n  Passed Microsoft.DurableTask.Tests.OrchestrationPatterns.LongTimer [8 s]\n  Passed Microsoft.DurableTask.Tests.OrchestrationPatterns.SubOrchestration [1 s]\n  ...\n  Passed Microsoft.DurableTask.Tests.OrchestrationPatterns.ActivityFanOut [914 ms]\n[xUnit.net 00:01:01.04]   Finished:    Microsoft.DurableTask.Tests\n  Passed Microsoft.DurableTask.Tests.OrchestrationPatterns.SingleActivity_Async [365 ms]\n\nTest Run Successful.\nTotal tests: 33\n     Passed: 33\n Total time: 1.0290 Minutes\n```\n\n## Running locally\n\nYou can run the engine locally by pressing `F5` in [Visual Studio Code](https://code.visualstudio.com/) (the recommended editor). You can also simply run `go run main.go` to start a local Durable Task gRPC server that listens on port 4001.\n\n```bash\ngo run main.go --port 4001 --db ./test.sqlite3\n```\n\nThe following is the expected output:\n\n```\n2022/09/14 17:26:50 backend started: sqlite::./test.sqlite3\n2022/09/14 17:26:50 server listening at 127.0.0.1:4001\n2022/09/14 17:26:50 orchestration-processor: waiting for new work items...\n2022/09/14 17:26:50 activity-processor: waiting for new work items...\n```\n\nAt this point you can use one of the [language SDKs](#language-sdks) mentioned earlier in a separate process to implement and execute durable orchestrations. Those SDKs will connect to port `4001` by default to interact with the Durable Task engine.\n\n## Contributing\n\nThis project welcomes contributions and suggestions.  Most contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.\n\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n\n## Trademarks\n\nThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark \u0026 Brand Guidelines](https://www.microsoft.com/legal/intellectualproperty/trademarks/usage/general).\nUse of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.\nAny use of third-party trademarks or logos are subject to those third-party's policies.\n","funding_links":[],"categories":["\u003ca name=\"Go\"\u003e\u003c/a\u003eGo"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fdurabletask-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrosoft%2Fdurabletask-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fdurabletask-go/lists"}