https://github.com/taylorchu/work

gocraft/work v2 prototype
https://github.com/taylorchu/work

Last synced: 4 days ago
JSON representation

gocraft/work v2 prototype

• Host: GitHub
• URL: https://github.com/taylorchu/work
• Owner: taylorchu
• License: mit
• Created: 2018-10-29T19:38:30.000Z (about 6 years ago)
• Default Branch: master
• Last Pushed: 2024-08-19T21:48:53.000Z (3 months ago)
• Last Synced: 2024-08-20T01:44:29.334Z (3 months ago)
• Language: Go
• Size: 265 KB
• Stars: 145
• Watchers: 11
• Forks: 22
• Open Issues: 14
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-ccamel - taylorchu/work - gocraft/work v2 prototype (Go)

README

        
# gocraft/work v2

[![GoDoc](https://godoc.org/github.com/taylorchu/work?status.png)](https://godoc.org/github.com/taylorchu/work)

[![Go Report Card](https://goreportcard.com/badge/github.com/taylorchu/work)](https://goreportcard.com/report/github.com/taylorchu/work)

[![FOSSA Status](https://app.fossa.com/api/projects/custom%2B4257%2Fgit%40github.com%3Ataylorchu%2Fwork.git.svg?type=shield)](https://app.fossa.com/projects/custom%2B4257%2Fgit%40github.com%3Ataylorchu%2Fwork.git?ref=badge_shield)

[![CircleCI](https://circleci.com/gh/taylorchu/work.svg?style=svg)](https://circleci.com/gh/taylorchu/work)

Please see `cmd/` for enqueuer and worker demo.

## Improvements

- [x] queue backend abstraction

  - redis is still the default, but the new design allows custom queue implementation.

- [x] simplify the keyspace design of redis queue backend

  - The new design uses 1 redis hash per job, and 1 redis sorted set for queue.

  - [Interesting read](https://kirshatrov.com/posts/redis-job-queue)

- [x] modular

  - The core only catches panics, retries on failure, and waits if a queue is empty.

  - All other [functionalities](https://kirshatrov.com/posts/state-of-background-jobs)

    are either removed or moved to separate middlewares.

- [x] support binary payload/args with message pack.

- [x] replace built-in UI with prometheus metrics (use grafana if you want dashboard).

- [x] additional optimizations (alloc + bulk queue ops)

  ```go

  BenchmarkWorkerRunJob/work_v1_1-8         	    3000	    515957 ns/op

  BenchmarkWorkerRunJob/work_v2_1-8         	    5000	    284516 ns/op

  BenchmarkWorkerRunJob/work_v1_10-8        	    1000	   2136546 ns/op

  BenchmarkWorkerRunJob/work_v2_10-8        	    5000	    367997 ns/op

  BenchmarkWorkerRunJob/work_v1_100-8       	     100	  18234023 ns/op

  BenchmarkWorkerRunJob/work_v2_100-8       	    1000	   1759186 ns/op

  BenchmarkWorkerRunJob/work_v1_1000-8      	      10	 162110100 ns/op

  BenchmarkWorkerRunJob/work_v2_1000-8      	     100	  12646080 ns/op

  BenchmarkWorkerRunJob/work_v1_10000-8     	       1	1691287122 ns/op

  BenchmarkWorkerRunJob/work_v2_10000-8     	      10	 144923087 ns/op

  BenchmarkWorkerRunJob/work_v1_100000-8    	       1	17515722574 ns/op

  BenchmarkWorkerRunJob/work_v2_100000-8    	       1	1502468637 ns/op

  PASS

  ok  	github.com/taylorchu/work	87.901s

  ```

- [x] http server

  - [x] delete job

  - [x] create job

  - [x] get job status

  - [x] get queue metrics (kubernetes autoscaler integration with [keda metrics api scaler](https://keda.sh/docs/2.5/scalers/metrics-api/))

  - [ ] OpenAPI spec

## License

[![FOSSA Status](https://app.fossa.com/api/projects/custom%2B4257%2Fgit%40github.com%3Ataylorchu%2Fwork.git.svg?type=large)](https://app.fossa.com/projects/custom%2B4257%2Fgit%40github.com%3Ataylorchu%2Fwork.git?ref=badge_large)