https://github.com/cydave/jobq
A simple job queue for golang
https://github.com/cydave/jobq
go golang queue
Last synced: over 1 year ago
JSON representation
A simple job queue for golang
- Host: GitHub
- URL: https://github.com/cydave/jobq
- Owner: cydave
- Created: 2025-01-19T16:15:40.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2025-01-20T09:33:31.000Z (over 1 year ago)
- Last Synced: 2025-01-27T13:48:00.329Z (over 1 year ago)
- Topics: go, golang, queue
- Language: Go
- Homepage:
- Size: 13.7 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# jobq
jobq is a simple job queue.
jobq starts out with a set of initial jobs. As the initial jobs are processed by
the workers, new jobs can be submitted from within the workers if needed.
Think of it as a webcrawler-esque job queue:
1. Fetch from initial URL
2. Extract all URLs
3. For each found URL:
- enqueue URL for crawling
**Note:** by default, jobq de-duplicates jobs. Jobs that have been processed are
indexed in an internal map. Jobs with the same ID are
skipped if they are fed back in the job queue. If you have no need for
deduplication, use `jobq.NewDup[T]()` instead of `jobq.New[T]()`.
## Examples
Currently there are two simple examples to get started with jobq:
- [de-duplicating jobq](examples/dedup/example.go)
- [non-de-duplicating jobq](examples/dup/example.go)
## Credits
The implementation of jobq is based off of the algorithm by [qvl/httpsyet](https://github.com/qvl/httpsyet).