{"id":27253431,"url":"https://github.com/marselester/gopher-celery","last_synced_at":"2025-04-11T01:27:33.653Z","repository":{"id":42674867,"uuid":"490861300","full_name":"marselester/gopher-celery","owner":"marselester","description":"A tool to place/process Celery tasks in Go.","archived":false,"fork":false,"pushed_at":"2024-12-27T22:17:22.000Z","size":81,"stargazers_count":45,"open_issues_count":0,"forks_count":11,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-12-27T23:35:21.209Z","etag":null,"topics":["celery","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marselester.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-10T21:09:40.000Z","updated_at":"2024-12-27T22:17:26.000Z","dependencies_parsed_at":"2024-06-20T19:06:14.884Z","dependency_job_id":null,"html_url":"https://github.com/marselester/gopher-celery","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marselester%2Fgopher-celery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marselester%2Fgopher-celery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marselester%2Fgopher-celery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marselester%2Fgopher-celery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marselester","download_url":"https://codeload.github.com/marselester/gopher-celery/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248324231,"owners_count":21084669,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","golang"],"created_at":"2025-04-11T01:27:31.346Z","updated_at":"2025-04-11T01:27:33.646Z","avatar_url":"https://github.com/marselester.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Gopher Celery 🥬\n\n[![Documentation](https://godoc.org/github.com/marselester/gopher-celery?status.svg)](https://pkg.go.dev/github.com/marselester/gopher-celery)\n[![Go Report Card](https://goreportcard.com/badge/github.com/marselester/gopher-celery)](https://goreportcard.com/report/github.com/marselester/gopher-celery)\n\nThe objective of this project is to provide\nthe very basic mechanism to efficiently produce and consume Celery tasks on Go side.\nTherefore there are no plans to support all the rich features the Python version provides,\nsuch as tasks chains, etc.\nEven task result backend has no practical value in the context of Gopher Celery,\nso it wasn't taken into account.\nNote, Celery has [no result backend](https://docs.celeryq.dev/en/stable/userguide/tasks.html?#result-backends)\nenabled by default (it incurs overhead).\n\nTypically one would want to use Gopher Celery when certain tasks on Python side\ntake too long to complete or there is a big volume of tasks requiring lots of Python workers\n(expensive infrastructure).\n\nThis project offers a little bit more convenient API of https://github.com/gocelery/gocelery\nincluding support for Celery protocol v2.\n\n## Usage\n\nThe Celery app can be used as either a producer or consumer (worker).\nTo send tasks to a queue for a worker to consume, use `Delay` method.\nIn order to process a task you should register it using `Register` method.\n\n```python\ndef mytask(a, b):\n    print(a + b)\n```\n\nFor example, whenever a task `mytask` is popped from `important` queue,\nthe Go function is executed with args and kwargs obtained from the task message.\nBy default Redis broker (localhost) is used with json task message serialization.\n\n```go\napp := celery.NewApp()\napp.Register(\n\t\"myproject.apps.myapp.tasks.mytask\",\n\t\"important\",\n\tfunc(ctx context.Context, p *celery.TaskParam) error {\n\t\tp.NameArgs(\"a\", \"b\")\n\t\t// Methods prefixed with Must panic if they can't find an argument name\n\t\t// or can't cast it to the corresponding type.\n\t\t// The panic doesn't affect other tasks execution; it's logged.\n\t\tfmt.Println(p.MustInt(\"a\") + p.MustInt(\"b\"))\n\t\t// Non-nil errors are logged.\n\t\treturn nil\n\t},\n)\nif err := app.Run(context.Background()); err != nil {\n\tlog.Printf(\"celery worker error: %v\", err)\n}\n```\n\nHere is an example of sending `mytask` task to `important` queue with `a=2`, `b=3` arguments.\nIf a task is processed on Python side,\nyou don't need to register the task or run the app.\n\n```go\napp := celery.NewApp()\nerr := app.Delay(\n\t\"myproject.apps.myapp.tasks.mytask\",\n\t\"important\",\n\t2,\n\t3,\n)\nif err != nil {\n\tlog.Printf(\"failed to send mytask: %v\", err)\n}\n```\n\nMore examples can be found in [the examples](examples) dir.\nNote, you'll need a Redis server to run them.\n\n```sh\n$ redis-server\n$ cd ./examples/\n```\n\n\u003cdetails\u003e\n\n\u003csummary\u003eSending tasks from Go and receiving them on Python side.\u003c/summary\u003e\n\n```sh\n$ go run ./producer/\n{\"err\":null,\"msg\":\"task was sent using protocol v2\"}\n{\"err\":null,\"msg\":\"task was sent using protocol v1\"}\n$ celery --app myproject worker --queues important --loglevel=debug --without-heartbeat --without-mingle\n...\n[... WARNING/ForkPoolWorker-1] received a=fizz b=bazz\n[... WARNING/ForkPoolWorker-8] received a=fizz b=bazz\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\n\u003csummary\u003eSending tasks from Python and receiving them on Go side.\u003c/summary\u003e\n\n```sh\n$ python producer.py\n$ go run ./consumer/\n{\"msg\":\"waiting for tasks...\"}\nreceived a=fizz b=bazz\nreceived a=fizz b=bazz\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\nMost likely your Redis server won't be running on localhost when the service is deployed,\nso you would need to pass a connection pool to the broker.\n\n\u003csummary\u003eRedis connection pool.\u003c/summary\u003e\n\n```sh\n$ go run ./producer/\n{\"err\":null,\"msg\":\"task was sent using protocol v2\"}\n{\"err\":null,\"msg\":\"task was sent using protocol v1\"}\n$ go run ./redis/\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\n\u003csummary\u003ePrometheus task metrics.\u003c/summary\u003e\n\n```sh\n$ go run ./producer/\n$ go run ./metrics/\n$ curl http://0.0.0.0:8080/metrics\n# HELP task_duration_seconds How long it took in seconds to process a task.\n# TYPE task_duration_seconds histogram\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"0.016\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"0.032\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"0.064\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"0.128\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"0.256\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"0.512\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"1.024\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"2.048\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"4.096\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"8.192\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"16.384\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"32.768\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"60\"} 2\ntask_duration_seconds_bucket{task=\"myproject.mytask\",le=\"+Inf\"} 2\ntask_duration_seconds_sum{task=\"myproject.mytask\"} 7.2802e-05\ntask_duration_seconds_count{task=\"myproject.mytask\"} 2\n# HELP tasks_total How many Celery tasks processed, partitioned by task name and error.\n# TYPE tasks_total counter\ntasks_total{error=\"false\",task=\"myproject.mytask\"} 2\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\nAlthough there is no built-in support for task retries (publishing a task back to Redis),\nyou can still retry the operation within the same goroutine.\n\n\u003csummary\u003eTask retries.\u003c/summary\u003e\n\n```sh\n$ go run ./retry/\n...\n{\"attempt\":1,\"err\":\"uh oh\",\"msg\":\"request failed\",\"ts\":\"2022-08-07T23:42:23.401191Z\"}\n{\"attempt\":2,\"err\":\"uh oh\",\"msg\":\"request failed\",\"ts\":\"2022-08-07T23:42:28.337204Z\"}\n{\"attempt\":3,\"err\":\"uh oh\",\"msg\":\"request failed\",\"ts\":\"2022-08-07T23:42:37.279873Z\"}\n```\n\n\u003c/details\u003e\n\n## Testing\n\nTests require a Redis server running locally.\n\n```sh\n$ go test -v -count=1 ./...\n```\n\nBenchmarks help to spot performance changes as the project evolves\nand also compare performance of serializers.\nFor example, based on the results below the protocol v2 is faster than v1 when encoding args:\n\n- 350 nanoseconds mean time, 3 allocations (248 bytes) with 0% variation across the samples\n- 1.21 microseconds mean time, 4 allocations (672 bytes) with 0% variation across the samples\n\nIt is recommended to run benchmarks multiple times and check\nhow stable they are using [Benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat) tool.\n\n```sh\n$ go test -bench=. -benchmem -count=10 ./internal/... | tee bench-new.txt\n```\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\n\n```sh\n$ benchstat bench-old.txt\n```\n\n\u003c/summary\u003e\n\n```\nname                                  time/op\nJSONSerializerEncode_v2NoParams-12    2.97ns ± 1%\nJSONSerializerEncode_v2Args-12         350ns ± 0%\nJSONSerializerEncode_v2Kwargs-12       582ns ± 0%\nJSONSerializerEncode_v2ArgsKwargs-12   788ns ± 1%\nJSONSerializerEncode_v1NoParams-12    1.12µs ± 1%\nJSONSerializerEncode_v1Args-12        1.21µs ± 0%\nJSONSerializerEncode_v1Kwargs-12      1.68µs ± 0%\nJSONSerializerEncode_v1ArgsKwargs-12  1.77µs ± 0%\n\nname                                  alloc/op\nJSONSerializerEncode_v2NoParams-12     0.00B\nJSONSerializerEncode_v2Args-12          248B ± 0%\nJSONSerializerEncode_v2Kwargs-12        472B ± 0%\nJSONSerializerEncode_v2ArgsKwargs-12    528B ± 0%\nJSONSerializerEncode_v1NoParams-12      672B ± 0%\nJSONSerializerEncode_v1Args-12          672B ± 0%\nJSONSerializerEncode_v1Kwargs-12      1.00kB ± 0%\nJSONSerializerEncode_v1ArgsKwargs-12  1.00kB ± 0%\n\nname                                  allocs/op\nJSONSerializerEncode_v2NoParams-12      0.00\nJSONSerializerEncode_v2Args-12          3.00 ± 0%\nJSONSerializerEncode_v2Kwargs-12        7.00 ± 0%\nJSONSerializerEncode_v2ArgsKwargs-12    8.00 ± 0%\nJSONSerializerEncode_v1NoParams-12      4.00 ± 0%\nJSONSerializerEncode_v1Args-12          4.00 ± 0%\nJSONSerializerEncode_v1Kwargs-12        10.0 ± 0%\nJSONSerializerEncode_v1ArgsKwargs-12    10.0 ± 0%\n```\n\n\u003c/details\u003e\n\nThe old and new stats are compared as follows.\n\n```sh\n$ benchstat bench-old.txt bench-new.txt\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarselester%2Fgopher-celery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarselester%2Fgopher-celery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarselester%2Fgopher-celery/lists"}