https://github.com/arquivei/goduck
Simple and extensible stream/pool consumer
https://github.com/arquivei/goduck
hacktoberfest kafka serverless
Last synced: 2 months ago
JSON representation
Simple and extensible stream/pool consumer
- Host: GitHub
- URL: https://github.com/arquivei/goduck
- Owner: arquivei
- License: bsd-3-clause
- Created: 2019-10-09T19:33:05.000Z (over 6 years ago)
- Default Branch: main
- Last Pushed: 2026-01-12T15:57:17.000Z (3 months ago)
- Last Synced: 2026-01-12T20:50:12.489Z (3 months ago)
- Topics: hacktoberfest, kafka, serverless
- Language: Go
- Homepage:
- Size: 403 KB
- Stars: 6
- Watchers: 16
- Forks: 4
- Open Issues: 14
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
## goDuck
This project's purpose is to be an engine that abstract message dispatching for workers
that deals with the concept of either streams or pools.
In other words, reading a message from a stream or a pool, and delivering that message
through the **Processor** Interface and interpreting its return value.
>It is important to note that, if the Process function returns an error, the engine wont
Ack the message, thus, not removing it from the queue or stream. The main idea for this,
is that the engine guarantees that every message will be processed at least once, without errors.
Sample of a stream processor
```go
import(
"github.com/arquivei/goduck"
"github.com/arquivei/goduck/engine/streamengine"
)
// The engine requires a type that implements the Process function
type processor struct{}
// Process func will receive the pulled message from the engine.
func (p processor) Process(ctx context.Context, message []byte) error {
...
err := serviceCall(args)
...
return err
}
func main {
// call below returns a kafka abstraction (interface)
kafka := NewKafkaStream()
engine := streamengine.New(processor{}, []goduck.Stream{kafka})
engine.Run(context.Background())
}
```
Sample of a pool processor
```go
import(
"github.com/arquivei/goduck"
"github.com/arquivei/goduck/engine/streamengine"
)
// The engine requires a type that implements the Process function
type processor struct{}
// Process func will receive the pulled message from the engine.
func (p processor) Process(ctx context.Context, message []byte) error {
...
err := serviceCall(args)
...
return err
}
func main {
// call below returns a pubsub abstraction (interface)
pubsub, err := NewPubsubQueue()
if err != nil {
}
engine := jobpoolengine.New(pubsub, processor{}, 1)
engine.Run(context.Background())
}
```
## Important configuration
### Kafka
* Commit interval:
Set this to allow asynchronous message processing between commits.
Without a value, defaults to every message having to be acknowledged before a new one
is retrieved.
This is bad to have when you should avoid message reprocessing. Suppose there is a failure
and the engine stops executing while processing. The larger the commit interval is, higher
is the chance of duplicating messages
To terminate the engine execution, a simple context cancellation will perform a shutdown
of the application.