{"id":34151404,"url":"https://github.com/sublime-security/machinery","last_synced_at":"2026-04-02T21:50:25.995Z","repository":{"id":37799351,"uuid":"405164769","full_name":"sublime-security/machinery","owner":"sublime-security","description":"Manual fork of RichardKnop/machinery","archived":false,"fork":false,"pushed_at":"2026-02-24T16:01:44.000Z","size":21794,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2026-02-24T20:45:15.282Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sublime-security.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-09-10T17:40:32.000Z","updated_at":"2026-02-24T16:01:11.000Z","dependencies_parsed_at":"2024-05-22T02:27:37.840Z","dependency_job_id":"f9fcd4ba-13b8-45a5-9ec3-c98d669a0840","html_url":"https://github.com/sublime-security/machinery","commit_stats":null,"previous_names":[],"tags_count":42,"template":false,"template_full_name":null,"purl":"pkg:github/sublime-security/machinery","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sublime-security%2Fmachinery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sublime-security%2Fmachinery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sublime-security%2Fmachinery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sublime-security%2Fmachinery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sublime-security","download_url":"https://codeload.github.com/sublime-security/machinery/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sublime-security%2Fmachinery/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30414307,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-12T00:40:14.898Z","status":"online","status_checked_at":"2026-03-12T02:00:07.260Z","response_time":114,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-15T05:23:01.800Z","updated_at":"2026-03-12T03:32:18.402Z","avatar_url":"https://github.com/sublime-security.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[1]: https://raw.githubusercontent.com/RichardKnop/assets/master/machinery/example_worker.png\n[2]: https://raw.githubusercontent.com/RichardKnop/assets/master/machinery/example_worker_receives_tasks.png\n[3]: http://patreon_public_assets.s3.amazonaws.com/sized/becomeAPatronBanner.png\n\n## Sublime Fork\n\nThis fork exists to prototype some changes and allow us to iterate w/o waiting on the original maintainer.\n\nTo use this fork Go's opinionated directory setup it gets a little weird. Inspiration: https://newbedev.com/using-forked-package-import-in-go\n\nTo consume this package the `go.mod` should have the original `require` (e.g. referencing `RichardKnop`). Use `replace`\n(at the bottom) with:\n`replace github.com/RichardKnop/machinery =\u003e github.com/sublime-security/machinery v1.10.6`\n\nThe tagged version in the `replace` statement should be what's used.\n\nTo develop on this repo clone it to:\n`$GOPATH/src/github.com/RichardKnop/machinery` (even though it should be under `sublime-security`).\n\nAnd then update your `replace` statement to be:\n`replace github.com/RichardKnop/machinery =\u003e ../../RichardKnop/machinery`\n(and don't push that!)\n\n## Machinery\n\nMachinery is an asynchronous task queue/job queue based on distributed message passing.\n\n[![Travis Status for RichardKnop/machinery](https://travis-ci.org/RichardKnop/machinery.svg?branch=master\u0026label=linux+build)](https://travis-ci.org/RichardKnop/machinery)\n[![godoc for RichardKnop/machinery](https://godoc.org/github.com/nathany/looper?status.svg)](http://godoc.org/github.com/RichardKnop/machinery/v1)\n[![codecov for RichardKnop/machinery](https://codecov.io/gh/RichardKnop/machinery/branch/master/graph/badge.svg)](https://codecov.io/gh/RichardKnop/machinery)\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/RichardKnop/machinery)](https://goreportcard.com/report/github.com/RichardKnop/machinery)\n[![GolangCI](https://golangci.com/badges/github.com/RichardKnop/machinery.svg)](https://golangci.com)\n[![OpenTracing Badge](https://img.shields.io/badge/OpenTracing-enabled-blue.svg)](http://opentracing.io)\n\n[![Sourcegraph for RichardKnop/machinery](https://sourcegraph.com/github.com/RichardKnop/machinery/-/badge.svg)](https://sourcegraph.com/github.com/RichardKnop/machinery?badge)\n[![Donate Bitcoin](https://img.shields.io/badge/donate-bitcoin-orange.svg)](https://richardknop.github.io/donate/)\n\n---\n\n* [V2 Experiment](#v2-experiment)\n* [First Steps](#first-steps)\n* [Configuration](#configuration)\n  * [Lock](#lock)\n  * [Broker](#broker)\n  * [DefaultQueue](#defaultqueue)\n  * [ResultBackend](#resultbackend)\n  * [ResultsExpireIn](#resultsexpirein)\n  * [AMQP](#amqp-2)\n  * [DynamoDB](#dynamodb)\n  * [Redis](#redis-2)\n  * [GCPPubSub](#gcppubsub)\n* [Custom Logger](#custom-logger)\n* [Server](#server)\n* [Workers](#workers)\n* [Tasks](#tasks)\n  * [Registering Tasks](#registering-tasks)\n  * [Signatures](#signatures)\n  * [Supported Types](#supported-types)\n  * [Sending Tasks](#sending-tasks)\n  * [Delayed Tasks](#delayed-tasks)\n  * [Retry Tasks](#retry-tasks)\n  * [Get Pending Tasks](#get-pending-tasks)\n  * [Keeping Results](#keeping-results)\n* [Workflows](#workflows)\n  * [Groups](#groups)\n  * [Chords](#chords)\n  * [Chains](#chains)\n* [Periodic Tasks \u0026 Workflows](#periodic-tasks--workflows)\n  * [Periodic Tasks](#periodic-tasks)\n  * [Periodic Groups](#periodic-groups)\n  * [Periodic Chains](#periodic-chains)\n  * [Periodic Chords](#periodic-chords)\n* [Development](#development)\n  * [Requirements](#requirements)\n  * [Dependencies](#dependencies)\n  * [Testing](#testing)\n\n### V2 Experiment\n\nPlease be advised that V2 is work in progress and breaking changes can and will happen until it is ready.\n\nYou can use the current V2 in order to avoid having to import all dependencies for brokers and backends you are not using.\n\nInstead of factory, you will need to inject broker and backend objects to the server constructor:\n\n```go\nimport (\n  \"github.com/RichardKnop/machinery/v2\"\n  backendsiface \"github.com/RichardKnop/machinery/v2/backends/iface\"\n  brokersiface \"github.com/RichardKnop/machinery/v2/brokers/iface\"\n  locksiface \"github.com/RichardKnop/machinery/v2/locks/iface\"\n)\n\nvar broker brokersiface.Broker\nvar backend backendsiface.Backend\nvar lock locksiface.Lock\nserver := machinery.NewServer(cnf, broker, backend, lock)\n// server.NewWorker(\"machinery\", 10)\n```\n\n### First Steps\n\nAdd the Machinery library to your $GOPATH/src:\n\n```sh\ngo get github.com/RichardKnop/machinery/v1\n```\n\nOr to get experimental v2 release:\n\n```sh\ngo get github.com/RichardKnop/machinery/v2\n```\n\nFirst, you will need to define some tasks. Look at sample tasks in `example/tasks/tasks.go` to see a few examples.\n\nSecond, you will need to launch a worker process with one of these commands (v2 is recommended since it doesn't import dependencies for all brokers / backends, only those you actually need):\n\n```sh\ngo run example/amqp/main.go worker\ngo run example/redis/main.go worker\n\ngo run example/amqp/main.go worker\ngo run example/redis/main.go worker\n```\n\nYou can also try v2 examples.\n\n```sh\ncd v2/\ngo run example/amqp/main.go worker\ngo run example/redigo/main.go worker // Redis with redigo driver\ngo run example/go-redis/main.go worker // Redis with Go Redis driver\n\ngo run example/amqp/main.go worker\ngo run example/redis/main.go worker\n```\n\n![Example worker][1]\n\nFinally, once you have a worker running and waiting for tasks to consume, send some tasks with one of these commands (v2 is recommended since it doesn't import dependencies for all brokers / backends, only those you actually need):\n\n```sh\ngo run example/v2/amqp/main.go send\ngo run example/v2/redigo/main.go send // Redis with redigo driver\ngo run example/v2/go-redis/main.go send // Redis with Go Redis driver\n\ngo run example/v1/amqp/main.go send\ngo run example/v1/redis/main.go send\n```\n\nYou will be able to see the tasks being processed asynchronously by the worker:\n\n![Example worker receives tasks][2]\n\n### Configuration\n\nThe [config](/v1/config/config.go) package has convenience methods for loading configuration from environment variables or a YAML file. For example, load configuration from environment variables:\n\n```go\ncnf, err := config.NewFromEnvironment()\n```\n\nOr load from YAML file:\n\n```go\ncnf, err := config.NewFromYaml(\"config.yml\", true)\n```\n\nSecond boolean flag enables live reloading of configuration every 10 seconds. Use `false` to disable live reloading.\n\nMachinery configuration is encapsulated by a `Config` struct and injected as a dependency to objects that need it.\n\n#### Lock\n\n##### Redis\n\nUse Redis URL in one of these formats:\n\n```\nredis://[password@]host[port][/db_num]\n```\n\nFor example:\n\n1. `redis://localhost:6379`, or with password `redis://password@localhost:6379`\n\n#### Broker\n\nA message broker. Currently supported brokers are:\n\n##### AMQP\n\nUse AMQP URL in the format:\n\n```\namqp://[username:password@]@host[:port]\n```\n\nFor example:\n\n1. `amqp://guest:guest@localhost:5672`\n\nAMQP also supports multiples brokers urls. You need to specify the URL separator in the `MultipleBrokerSeparator` field.\n\n##### Redis\n\nUse Redis URL in one of these formats:\n\n```\nredis://[password@]host[port][/db_num]\nredis+socket://[password@]/path/to/file.sock[:/db_num]\n```\n\nFor example:\n\n1. `redis://localhost:6379`, or with password `redis://password@localhost:6379`\n2. `redis+socket://password@/path/to/file.sock:/0`\n\n##### AWS SQS\n\nUse AWS SQS URL in the format:\n\n```\nhttps://sqs.us-east-2.amazonaws.com/123456789012\n```\n\nSee [AWS SQS docs](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html) for more information.\nAlso, configuring `AWS_REGION` is required, or an error would be thrown.\n\nTo use a manually configured SQS Client:\n\n```go\nvar sqsClient = sqs.New(session.Must(session.NewSession(\u0026aws.Config{\n  Region:         aws.String(\"YOUR_AWS_REGION\"),\n  Credentials:    credentials.NewStaticCredentials(\"YOUR_AWS_ACCESS_KEY\", \"YOUR_AWS_ACCESS_SECRET\", \"\"),\n  HTTPClient:     \u0026http.Client{\n    Timeout: time.Second * 120,\n  },\n})))\nvar visibilityTimeout = 20\nvar cnf = \u0026config.Config{\n  Broker:          \"YOUR_SQS_URL\"\n  DefaultQueue:    \"machinery_tasks\",\n  ResultBackend:   \"YOUR_BACKEND_URL\",\n  SQS: \u0026config.SQSConfig{\n    Client: sqsClient,\n    // if VisibilityTimeout is nil default to the overall visibility timeout setting for the queue\n    // https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html\n    VisibilityTimeout: \u0026visibilityTimeout,\n    WaitTimeSeconds: 30,\n  },\n}\n```\n\n##### GCP Pub/Sub\n\nUse GCP Pub/Sub URL in the format:\n\n```\ngcppubsub://YOUR_GCP_PROJECT_ID/YOUR_PUBSUB_SUBSCRIPTION_NAME\n```\n\nTo use a manually configured Pub/Sub Client:\n\n```go\npubsubClient, err := pubsub.NewClient(\n    context.Background(),\n    \"YOUR_GCP_PROJECT_ID\",\n    option.WithServiceAccountFile(\"YOUR_GCP_SERVICE_ACCOUNT_FILE\"),\n)\n\ncnf := \u0026config.Config{\n  Broker:          \"gcppubsub://YOUR_GCP_PROJECT_ID/YOUR_PUBSUB_SUBSCRIPTION_NAME\"\n  DefaultQueue:    \"YOUR_PUBSUB_TOPIC_NAME\",\n  ResultBackend:   \"YOUR_BACKEND_URL\",\n  GCPPubSub: config.GCPPubSubConfig{\n    Client: pubsubClient,\n  },\n}\n```\n\n#### DefaultQueue\n\nDefault queue name, e.g. `machinery_tasks`.\n\n#### ResultBackend\n\nResult backend to use for keeping task states and results.\n\nCurrently supported backends are:\n\n##### Redis\n\nUse Redis URL in one of these formats:\n\n```\nredis://[password@]host[port][/db_num]\nredis+socket://[password@]/path/to/file.sock[:/db_num]\n```\n\nFor example:\n\n1. `redis://localhost:6379`, or with password `redis://password@localhost:6379`\n2. `redis+socket://password@/path/to/file.sock:/0`\n3. cluster `redis://host1:port1,host2:port2,host3:port3`\n4. cluster with password `redis://pass@host1:port1,host2:port2,host3:port3`\n\n##### Memcache\n\nUse Memcache URL in the format:\n\n```\nmemcache://host1[:port1][,host2[:port2],...[,hostN[:portN]]]\n```\n\nFor example:\n\n1. `memcache://localhost:11211` for a single instance, or\n2. `memcache://10.0.0.1:11211,10.0.0.2:11211` for a cluster\n\n##### AMQP\n\nUse AMQP URL in the format:\n\n```\namqp://[username:password@]@host[:port]\n```\n\nFor example:\n\n1. `amqp://guest:guest@localhost:5672`\n\n\u003e Keep in mind AMQP is not recommended as a result backend. See [Keeping Results](https://github.com/RichardKnop/machinery#keeping-results)\n\n##### MongoDB\n\nUse Mongodb URL in the format:\n\n```\nmongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]\n```\n\nFor example:\n\n1. `mongodb://localhost:27017/taskresults`\n\nSee [MongoDB docs](https://docs.mongodb.org/manual/reference/connection-string/) for more information.\n\n\n#### ResultsExpireIn\n\nHow long to store task results for in seconds. Defaults to `3600` (1 hour).\n\n#### AMQP\n\nRabbitMQ related configuration. Not necessary if you are using other broker/backend.\n\n* `Exchange`: exchange name, e.g. `machinery_exchange`\n* `ExchangeType`: exchange type, e.g. `direct`\n* `QueueBindingArguments`: an optional map of additional arguments used when binding to an AMQP queue\n* `BindingKey`: The queue is bind to the exchange with this key, e.g. `machinery_task`\n* `PrefetchCount`: How many tasks to prefetch (set to `1` if you have long running tasks)\n\n#### DynamoDB\n\nDynamoDB related configuration. Not necessary if you are using other backend.\n* `TaskStatesTable`: Custom table name for saving task states. Default one is `task_states`, and make sure to create this table in your AWS admin first, using `TaskUUID` as table's primary key.\n* `GroupMetasTable`: Custom table name for saving group metas. Default one is `group_metas`, and make sure to create this table in your AWS admin first, using `GroupUUID` as table's primary key.\nFor example:\n\n```\ndynamodb:\n  task_states_table: 'task_states'\n  group_metas_table: 'group_metas'\n```\nIf these tables are not found, an fatal error would be thrown.\n\nIf you wish to expire the records, you can configure the `TTL` field in AWS admin for these tables. The `TTL` field is set based on the `ResultsExpireIn` value in the Server's config. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html for more information.\n\n#### Redis\n\nRedis related configuration. Not necessary if you are using other backend.\n\nSee: [config](/v1/config/config.go) (TODO)\n\n#### GCPPubSub\n\nGCPPubSub related configuration. Not necessary if you are using other backend.\n\nSee: [config](/v1/config/config.go) (TODO)\n\n### Custom Logger\n\nYou can define a custom logger by implementing the following interface:\n\n```go\ntype Interface interface {\n  Print(...interface{})\n  Printf(string, ...interface{})\n  Println(...interface{})\n\n  Fatal(...interface{})\n  Fatalf(string, ...interface{})\n  Fatalln(...interface{})\n\n  Panic(...interface{})\n  Panicf(string, ...interface{})\n  Panicln(...interface{})\n}\n```\n\nThen just set the logger in your setup code by calling `Set` function exported by `github.com/RichardKnop/machinery/v1/log` package:\n\n```go\nlog.Set(myCustomLogger)\n```\n\n### Server\n\nA Machinery library must be instantiated before use. The way this is done is by creating a `Server` instance. `Server` is a base object which stores Machinery configuration and registered tasks. E.g.:\n\n```go\nimport (\n  \"github.com/RichardKnop/machinery/v1/config\"\n  \"github.com/RichardKnop/machinery/v1\"\n)\n\nvar cnf = \u0026config.Config{\n  Broker:        \"amqp://guest:guest@localhost:5672/\",\n  DefaultQueue:  \"machinery_tasks\",\n  ResultBackend: \"amqp://guest:guest@localhost:5672/\",\n  AMQP: \u0026config.AMQPConfig{\n    Exchange:     \"machinery_exchange\",\n    ExchangeType: \"direct\",\n    BindingKey:   \"machinery_task\",\n  },\n}\n\nserver, err := machinery.NewServer(cnf)\nif err != nil {\n  // do something with the error\n}\n```\n\n### Workers\n\nIn order to consume tasks, you need to have one or more workers running. All you need to run a worker is a `Server` instance with registered tasks. E.g.:\n\n```go\nworker := server.NewWorker(\"worker_name\", 10)\nerr := worker.Launch()\nif err != nil {\n  // do something with the error\n}\n```\n\nEach worker will only consume registered tasks. For each task on the queue the Worker.Process() method will be run\nin a goroutine. Use the second parameter of `server.NewWorker` to limit the number of concurrently running Worker.Process()\ncalls (per worker). Example: 1 will serialize task execution while 0 makes the number of concurrently executed tasks unlimited (default).\n\n### Tasks\n\nTasks are a building block of Machinery applications. A task is a function which defines what happens when a worker receives a message.\n\nEach task needs to return an error as a last return value. In addition to error tasks can now return any number of arguments.\n\nExamples of valid tasks:\n\n```go\nfunc Add(args ...int64) (int64, error) {\n  sum := int64(0)\n  for _, arg := range args {\n    sum += arg\n  }\n  return sum, nil\n}\n\nfunc Multiply(args ...int64) (int64, error) {\n  sum := int64(1)\n  for _, arg := range args {\n    sum *= arg\n  }\n  return sum, nil\n}\n\n// You can use context.Context as first argument to tasks, useful for open tracing\nfunc TaskWithContext(ctx context.Context, arg Arg) error {\n  // ... use ctx ...\n  return nil\n}\n\n// Tasks need to return at least error as a minimal requirement\nfunc DummyTask(arg string) error {\n  return errors.New(arg)\n}\n\n// You can also return multiple results from the task\nfunc DummyTask2(arg1, arg2 string) (string, string, error) {\n  return arg1, arg2, nil\n}\n```\n\n#### Registering Tasks\n\nBefore your workers can consume a task, you need to register it with the server. This is done by assigning a task a unique name:\n\n```go\nserver.RegisterTasks(map[string]interface{}{\n  \"add\":      Add,\n  \"multiply\": Multiply,\n})\n```\n\nTasks can also be registered one by one:\n\n```go\nserver.RegisterTask(\"add\", Add)\nserver.RegisterTask(\"multiply\", Multiply)\n```\n\nSimply put, when a worker receives a message like this:\n\n```json\n{\n  \"UUID\": \"48760a1a-8576-4536-973b-da09048c2ac5\",\n  \"Name\": \"add\",\n  \"RoutingKey\": \"\",\n  \"ETA\": null,\n  \"GroupUUID\": \"\",\n  \"GroupTaskCount\": 0,\n  \"Args\": [\n    {\n      \"Type\": \"int64\",\n      \"Value\": 1,\n    },\n    {\n      \"Type\": \"int64\",\n      \"Value\": 1,\n    }\n  ],\n  \"Immutable\": false,\n  \"RetryCount\": 0,\n  \"RetryTimeout\": 0,\n  \"OnSuccess\": null,\n  \"OnError\": null,\n  \"ChordCallback\": null\n}\n```\n\nIt will call Add(1, 1). Each task should return an error as well so we can handle failures.\n\nIdeally, tasks should be idempotent which means there will be no unintended consequences when a task is called multiple times with the same arguments.\n\n#### Signatures\n\nA signature wraps calling arguments, execution options (such as immutability) and success/error callbacks of a task so it can be sent across the wire to workers. Task signatures implement a simple interface:\n\n```go\n// Arg represents a single argument passed to invocation fo a task\ntype Arg struct {\n  Type  string\n  Value interface{}\n}\n\n// Headers represents the headers which should be used to direct the task\ntype Headers map[string]interface{}\n\n// Signature represents a single task invocation\ntype Signature struct {\n  UUID           string\n  Name           string\n  RoutingKey     string\n  ETA            *time.Time\n  GroupUUID      string\n  GroupTaskCount int\n  Args           []Arg\n  Headers        Headers\n  Immutable      bool\n  RetryCount     int\n  RetryTimeout   int\n  OnSuccess      []*Signature\n  OnError        []*Signature\n  ChordCallback  *Signature\n}\n```\n\n`UUID` is a unique ID of a task. You can either set it yourself or it will be automatically generated.\n\n`Name` is the unique task name by which it is registered against a Server instance.\n\n`RoutingKey` is used for routing a task to correct queue. If you leave it empty, the default behaviour will be to set it to the default queue's binding key for direct exchange type and to the default queue name for other exchange types.\n\n`ETA` is  a timestamp used for delaying a task. if it's nil, the task will be published for workers to consume immediately. If it is set, the task will be delayed until the ETA timestamp.\n\n`GroupUUID`, `GroupTaskCount` are useful for creating groups of tasks.\n\n`Args` is a list of arguments that will be passed to the task when it is executed by a worker.\n\n`Headers` is a list of headers that will be used when publishing the task to AMQP queue.\n\n`Immutable` is a flag which defines whether a result of the executed task can be modified or not. This is important with `OnSuccess` callbacks. Immutable task will not pass its result to its success callbacks while a mutable task will prepend its result to args sent to callback tasks. Long story short, set Immutable to false if you want to pass result of the first task in a chain to the second task.\n\n`RetryCount` specifies how many times a failed task should be retried (defaults to 0). Retry attempts will be spaced out in time, after each failure another attempt will be scheduled further to the future.\n\n`RetryTimeout` specifies how long to wait before resending task to the queue for retry attempt. Default behaviour is to use fibonacci sequence to increase the timeout after each failed retry attempt.\n\n`OnSuccess` defines tasks which will be called after the task has executed successfully. It is a slice of task signature structs.\n\n`OnError` defines tasks which will be called after the task execution fails. The first argument passed to error callbacks will be the error string returned from the failed task.\n\n`ChordCallback` is used to create a callback to a group of tasks.\n\n#### Supported Types\n\nMachinery encodes tasks to JSON before sending them to the broker. Task results are also stored in the backend as JSON encoded strings. Therefor only types with native JSON representation can be supported. Currently supported types are:\n\n* `bool`\n* `int`\n* `int8`\n* `int16`\n* `int32`\n* `int64`\n* `uint`\n* `uint8`\n* `uint16`\n* `uint32`\n* `uint64`\n* `float32`\n* `float64`\n* `string`\n* `[]bool`\n* `[]int`\n* `[]int8`\n* `[]int16`\n* `[]int32`\n* `[]int64`\n* `[]uint`\n* `[]uint8`\n* `[]uint16`\n* `[]uint32`\n* `[]uint64`\n* `[]float32`\n* `[]float64`\n* `[]string`\n\n#### Sending Tasks\n\nTasks can be called by passing an instance of `Signature` to an `Server` instance. E.g:\n\n```go\nimport (\n  \"github.com/RichardKnop/machinery/v1/tasks\"\n)\n\nsignature := \u0026tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n  },\n}\n\nasyncResult, err := server.SendTask(signature)\nif err != nil {\n  // failed to send the task\n  // do something with the error\n}\n```\n\n#### Delayed Tasks\n\nYou can delay a task by setting the `ETA` timestamp field on the task signature.\n\n```go\n// Delay the task by 5 seconds\neta := time.Now().UTC().Add(time.Second * 5)\nsignature.ETA = \u0026eta\n```\n\n#### Retry Tasks\n\nYou can set a number of retry attempts before declaring task as failed. Fibonacci sequence will be used to space out retry requests over time. (See `RetryTimeout` for details.)\n\n```go\n// If the task fails, retry it up to 3 times\nsignature.RetryCount = 3\n```\n\nAlternatively, you can return `tasks.ErrRetryTaskLater` from your task and specify duration after which the task should be retried, e.g.:\n\n```go\nreturn tasks.NewErrRetryTaskLater(\"some error\", 4 * time.Hour)\n```\n\n#### Get Pending Tasks\n\nTasks currently waiting in the queue to be consumed by workers can be inspected, e.g.:\n\n```go\nserver.GetBroker().GetPendingTasks(\"some_queue\")\n```\n\n\u003e Currently only supported by Redis broker.\n\n#### Keeping Results\n\nIf you configure a result backend, the task states and results will be persisted. Possible states:\n\n```go\nconst (\n\t// StatePending - initial state of a task\n\tStatePending = \"PENDING\"\n\t// StateReceived - when task is received by a worker\n\tStateReceived = \"RECEIVED\"\n\t// StateStarted - when the worker starts processing the task\n\tStateStarted = \"STARTED\"\n\t// StateRetry - when failed task has been scheduled for retry\n\tStateRetry = \"RETRY\"\n\t// StateSuccess - when the task is processed successfully\n\tStateSuccess = \"SUCCESS\"\n\t// StateFailure - when processing of the task fails\n\tStateFailure = \"FAILURE\"\n)\n```\n\n\u003e When using AMQP as a result backend, task states will be persisted in separate queues for each task. Although RabbitMQ can scale up to thousands of queues, it is strongly advised to use a better suited result backend (e.g. Memcache) when you are expecting to run a large number of parallel tasks.\n\n```go\n// TaskResult represents an actual return value of a processed task\ntype TaskResult struct {\n  Type  string      `bson:\"type\"`\n  Value interface{} `bson:\"value\"`\n}\n\n// TaskState represents a state of a task\ntype TaskState struct {\n  TaskUUID  string        `bson:\"_id\"`\n  State     string        `bson:\"state\"`\n  Results   []*TaskResult `bson:\"results\"`\n  Error     string        `bson:\"error\"`\n}\n\n// GroupMeta stores useful metadata about tasks within the same group\n// E.g. UUIDs of all tasks which are used in order to check if all tasks\n// completed successfully or not and thus whether to trigger chord callback\ntype GroupMeta struct {\n  GroupUUID      string   `bson:\"_id\"`\n  TaskUUIDs      []string `bson:\"task_uuids\"`\n  ChordTriggered bool     `bson:\"chord_triggered\"`\n  Lock           bool     `bson:\"lock\"`\n}\n```\n\n`TaskResult` represents a slice of return values of a processed task.\n\n`TaskState` struct will be serialized and stored every time a task state changes.\n\n`GroupMeta` stores useful metadata about tasks within the same group. E.g. UUIDs of all tasks which are used in order to check if all tasks completed successfully or not and thus whether to trigger chord callback.\n\n`AsyncResult` object allows you to check for the state of a task:\n\n```go\ntaskState := asyncResult.GetState()\nfmt.Printf(\"Current state of %v task is:\\n\", taskState.TaskUUID)\nfmt.Println(taskState.State)\n```\n\nThere are couple of convenient methods to inspect the task status:\n\n```go\nasyncResult.GetState().IsCompleted()\nasyncResult.GetState().IsSuccess()\nasyncResult.GetState().IsFailure()\n```\n\nYou can also do a synchronous blocking call to wait for a task result:\n\n```go\nresults, err := asyncResult.Get(time.Duration(time.Millisecond * 5))\nif err != nil {\n  // getting result of a task failed\n  // do something with the error\n}\nfor _, result := range results {\n  fmt.Println(result.Interface())\n}\n```\n\n#### Error Handling\n\nWhen a task returns with an error, the default behavior is to first attempty to retry the task if it's retriable, otherwise log the error and then eventually call any error callbacks.\n\nTo customize this, you can set a custom error handler on the worker which can do more than just logging after retries fail and error callbacks are trigerred:\n\n```go\nworker.SetErrorHandler(func (err error) {\n  customHandler(err)\n})\n```\n\n### Workflows\n\nRunning a single asynchronous task is fine but often you will want to design a workflow of tasks to be executed in an orchestrated way. There are couple of useful functions to help you design workflows.\n\n#### Groups\n\n`Group` is a set of tasks which will be executed in parallel, independent of each other. E.g.:\n\n```go\nimport (\n  \"github.com/RichardKnop/machinery/v1/tasks\"\n  \"github.com/RichardKnop/machinery/v1\"\n)\n\nsignature1 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n  },\n}\n\nsignature2 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n  },\n}\n\ngroup, _ := tasks.NewGroup(\u0026signature1, \u0026signature2)\nasyncResults, err := server.SendGroup(group, 0) //The second parameter specifies the number of concurrent sending tasks. 0 means unlimited.\nif err != nil {\n  // failed to send the group\n  // do something with the error\n}\n```\n\n`SendGroup` returns a slice of `AsyncResult` objects. So you can do a blocking call and wait for the result of groups tasks:\n\n```go\nfor _, asyncResult := range asyncResults {\n  results, err := asyncResult.Get(time.Duration(time.Millisecond * 5))\n  if err != nil {\n    // getting result of a task failed\n    // do something with the error\n  }\n  for _, result := range results {\n    fmt.Println(result.Interface())\n  }\n}\n```\n\n#### Chords\n\n`Chord` allows you to define a callback to be executed after all tasks in a group finished processing, e.g.:\n\n```go\nimport (\n  \"github.com/RichardKnop/machinery/v1/tasks\"\n  \"github.com/RichardKnop/machinery/v1\"\n)\n\nsignature1 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n  },\n}\n\nsignature2 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n  },\n}\n\nsignature3 := tasks.Signature{\n  Name: \"multiply\",\n}\n\ngroup := tasks.NewGroup(\u0026signature1, \u0026signature2)\nchord, _ := tasks.NewChord(group, \u0026signature3)\nchordAsyncResult, err := server.SendChord(chord, 0) //The second parameter specifies the number of concurrent sending tasks. 0 means unlimited.\nif err != nil {\n  // failed to send the chord\n  // do something with the error\n}\n```\n\nThe above example executes task1 and task2 in parallel, aggregates their results and passes them to task3. Therefore what would end up happening is:\n\n```\nmultiply(add(1, 1), add(5, 5))\n```\n\nMore explicitly:\n\n```\n(1 + 1) * (5 + 5) = 2 * 10 = 20\n```\n\n`SendChord` returns `ChordAsyncResult` which follows AsyncResult's interface. So you can do a blocking call and wait for the result of the callback:\n\n```go\nresults, err := chordAsyncResult.Get(time.Duration(time.Millisecond * 5))\nif err != nil {\n  // getting result of a chord failed\n  // do something with the error\n}\nfor _, result := range results {\n  fmt.Println(result.Interface())\n}\n```\n\n#### Chains\n\n`Chain` is simply a set of tasks which will be executed one by one, each successful task triggering the next task in the chain. E.g.:\n\n```go\nimport (\n  \"github.com/RichardKnop/machinery/v1/tasks\"\n  \"github.com/RichardKnop/machinery/v1\"\n)\n\nsignature1 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n  },\n}\n\nsignature2 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n  },\n}\n\nsignature3 := tasks.Signature{\n  Name: \"multiply\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 4,\n    },\n  },\n}\n\nchain, _ := tasks.NewChain(\u0026signature1, \u0026signature2, \u0026signature3)\nchainAsyncResult, err := server.SendChain(chain)\nif err != nil {\n  // failed to send the chain\n  // do something with the error\n}\n```\n\nThe above example executes task1, then task2 and then task3. When a task is completed successfully, the result is appended to the end of list of arguments for the next task in the chain. Therefore what would end up happening is:\n\n```\nmultiply(4, add(5, 5, add(1, 1)))\n```\n\nMore explicitly:\n\n```\n  4 * (5 + 5 + (1 + 1))   # task1: add(1, 1)        returns 2\n= 4 * (5 + 5 + 2)         # task2: add(5, 5, 2)     returns 12\n= 4 * (12)                # task3: multiply(4, 12)  returns 48\n= 48\n```\n\n`SendChain` returns `ChainAsyncResult` which follows AsyncResult's interface. So you can do a blocking call and wait for the result of the whole chain:\n\n```go\nresults, err := chainAsyncResult.Get(time.Duration(time.Millisecond * 5))\nif err != nil {\n  // getting result of a chain failed\n  // do something with the error\n}\nfor _, result := range results {\n  fmt.Println(result.Interface())\n}\n```\n\n### Periodic Tasks \u0026 Workflows\n\nMachinery now supports scheduling periodic tasks and workflows. See examples bellow.\n\n#### Periodic Tasks\n\n```go\nimport (\n  \"github.com/RichardKnop/machinery/v1/tasks\"\n)\n\nsignature := \u0026tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n  },\n}\nerr := server.RegisterPeriodicTask(\"0 6 * * ?\", \"periodic-task\", signature)\nif err != nil {\n  // failed to register periodic task\n}\n```\n\n#### Periodic Groups\n\n```go\nimport (\n  \"github.com/RichardKnop/machinery/v1/tasks\"\n  \"github.com/RichardKnop/machinery/v1\"\n)\n\nsignature1 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n  },\n}\n\nsignature2 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n  },\n}\n\ngroup, _ := tasks.NewGroup(\u0026signature1, \u0026signature2)\nerr := server.RegisterPeriodicGroup(\"0 6 * * ?\", \"periodic-group\", group)\nif err != nil {\n  // failed to register periodic group\n}\n```\n\n#### Periodic Chains\n\n```go\nimport (\n  \"github.com/RichardKnop/machinery/v1/tasks\"\n  \"github.com/RichardKnop/machinery/v1\"\n)\n\nsignature1 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n  },\n}\n\nsignature2 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n  },\n}\n\nsignature3 := tasks.Signature{\n  Name: \"multiply\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 4,\n    },\n  },\n}\n\nchain, _ := tasks.NewChain(\u0026signature1, \u0026signature2, \u0026signature3)\nerr := server.RegisterPeriodicChain(\"0 6 * * ?\", \"periodic-chain\", chain)\nif err != nil {\n  // failed to register periodic chain\n}\n```\n\n#### Chord\n\n```go\nimport (\n  \"github.com/RichardKnop/machinery/v1/tasks\"\n  \"github.com/RichardKnop/machinery/v1\"\n)\n\nsignature1 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n    {\n      Type:  \"int64\",\n      Value: 1,\n    },\n  },\n}\n\nsignature2 := tasks.Signature{\n  Name: \"add\",\n  Args: []tasks.Arg{\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n    {\n      Type:  \"int64\",\n      Value: 5,\n    },\n  },\n}\n\nsignature3 := tasks.Signature{\n  Name: \"multiply\",\n}\n\ngroup := tasks.NewGroup(\u0026signature1, \u0026signature2)\nchord, _ := tasks.NewChord(group, \u0026signature3)\nerr := server.RegisterPeriodicChord(\"0 6 * * ?\", \"periodic-chord\", chord)\nif err != nil {\n  // failed to register periodic chord\n}\n```\n\n### Development\n\n#### Requirements\n\n* Go\n* RabbitMQ (optional)\n* Redis\n* Memcached (optional)\n* MongoDB (optional)\n\nOn OS X systems, you can install requirements using [Homebrew](http://brew.sh/):\n\n```sh\nbrew install go\nbrew install rabbitmq\nbrew install redis\nbrew install memcached\nbrew install mongodb\n```\n\nOr optionally use the corresponding [Docker](http://docker.io/) containers:\n\n```\ndocker run -d -p 5672:5672 rabbitmq\ndocker run -d -p 6379:6379 redis\ndocker run -d -p 11211:11211 memcached\ndocker run -d -p 27017:27017 mongo\ndocker run -d -p 6831:6831/udp -p 16686:16686 jaegertracing/all-in-one:latest\n```\n\n#### Dependencies\n\nSince Go 1.11, a new recommended dependency management system is via [modules](https://github.com/golang/go/wiki/Modules).\n\nThis is one of slight weaknesses of Go as dependency management is not a solved problem. Previously Go was officially recommending to use the [dep tool](https://github.com/golang/dep) but that has been abandoned now in favor of modules.\n\n#### Testing\n\nEasiest (and platform agnostic) way to run tests is via `docker-compose`:\n\n```sh\nmake ci\n```\n\nThis will basically run docker-compose command:\n\n```sh\n(docker-compose -f docker-compose.test.yml -p machinery_ci up --build -d) \u0026\u0026 (docker logs -f machinery_sut \u0026) \u0026\u0026 (docker wait machinery_sut)\n```\n\nAlternative approach is to setup a development environment on your machine.\n\nIn order to enable integration tests, you will need to install all required services (RabbitMQ, Redis, Memcache, MongoDB) and export these environment variables:\n\n```sh\nexport AMQP_URL=amqp://guest:guest@localhost:5672/\nexport REDIS_URL=localhost:6379\nexport MEMCACHE_URL=localhost:11211\nexport MONGODB_URL=localhost:27017\n```\n\nTo run integration tests against an SQS instance, you will need to create a \"test_queue\" in SQS and export these environment variables:\n\n```sh\nexport SQS_URL=https://YOUR_SQS_URL\nexport AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID\nexport AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY\nexport AWS_DEFAULT_REGION=YOUR_AWS_DEFAULT_REGION\n```\n\nThen just run:\n\n```sh\nmake test\n```\n\nIf the environment variables are not exported, `make test` will only run unit tests.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsublime-security%2Fmachinery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsublime-security%2Fmachinery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsublime-security%2Fmachinery/lists"}