https://github.com/fogfish/swarm
Go channels for distributed queueing and event-driven systems
https://github.com/fogfish/swarm
aws-dynamodb aws-eventbridge aws-s3 aws-sqs aws-websockets event-driven event-sourcing golang golang-channels serverless
Last synced: 8 months ago
JSON representation
Go channels for distributed queueing and event-driven systems
- Host: GitHub
- URL: https://github.com/fogfish/swarm
- Owner: fogfish
- License: apache-2.0
- Created: 2021-08-03T15:38:30.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2024-10-07T16:30:51.000Z (over 1 year ago)
- Last Synced: 2024-10-20T10:23:52.770Z (over 1 year ago)
- Topics: aws-dynamodb, aws-eventbridge, aws-s3, aws-sqs, aws-websockets, event-driven, event-sourcing, golang, golang-channels, serverless
- Language: Go
- Homepage:
- Size: 2.38 MB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
swarm
Go channels for distributed queueing and event-driven systems
sub-moduledocfeaturesabout
API & Kernel
AWS EventBridge
AWS DynamoDB Stream
AWS S3 Event
AWS SQS Events
AWS SQS and SQS FIFO
AWS WebSocket API
AWS SNS
AWS Kinesis Events
AWS Kinesis
AWS ElastiCache
MQTT
---
# From Chaos to Channels
> *"Finally, messaging that feels like Go!"* — Distributed systems, done right.
Writing distributed, event-driven systems in Go today is harder than it should be - a lesson learned from a decade building such systems in Erlang. Vendor APIs are clunky, non-idiomatic, and tightly coupled to specific messaging brokers.
`swarm` makes asynchronous, distributed messaging in Go **idiomatic, testable, and portable** by expressing queueing/event-driven systems through Go channels instead of vendor-specific APIs.
[User Guide](./doc/user-guide.md) |
[Playground](https://go.dev/play/p/B4C9fcNmWYW) |
[Getting started](#getting-started) |
[Examples](./broker/) |
[Philosophy](./doc/pattern.md)
## Quick Start
The example below shows the simplest way to enqueue and dequeue messages using AWS SQS.
```go
package main
import (
"github.com/fogfish/swarm"
"github.com/fogfish/swarm/broker/sqs"
"github.com/fogfish/swarm/emit"
"github.com/fogfish/swarm/listen"
)
// Example message
type Order struct {
ID string `json:"id"`
Amount float64 `json:"amount"`
}
func main() {
// create broker for AWS SQS
q := sqs.Endpoint().Build("aws-sqs-queue-name")
// create Golang channels
rcv, ack := listen.Typed[Order](q.Listener)
out := swarm.LogDeadLetters(emit.Typed[Order](q.Emitter))
// Send messages
out <- Order{ID: "123", Amount: 100.0}
// use Golang channels for I/O
go func() {
for order := range rcv {
processOrder(order.Object)
ack <- order // acknowledge processing
}
}()
q.Await()
}
func processOrder(order Order) {
// Your business logic here
}
```
**That's it!** You're now using distributed messaging with native Go channels.
- Try this code in [playground](https://go.dev/play/p/B4C9fcNmWYW)
- Continue to [Getting Started](#getting-started) for advanced configuration
- See more examples by browsing [/broker/*/examples/](./broker/)
- Check the design pattern [Distributed event-driven Golang channels](./doc/pattern.md) for deep-dive into library philosophy.
Continue reading to understand the purpose of the library and why it exists.
## What is `swarm`?
`swarm` is a Go library that solves the complexity of distributed, event-driven systems by abstracting external messaging queues (like AWS SQS, AWS EventBridge, RabbitMQ, Kafka, etc.) behind **type-safe Go channels**.
By aligning with Go's native concurrency model, `swarm` encourages developers to think in terms of **message flows rather than request-response cycles**, which reflects the asynchronous and unreliable reality of distributed systems. This mindset shift leads to more resilient, scalable, and maintainable architectures — where complexity is not hidden, but made explicit and manageable.
* **Immediate Productivity**: Use Go channel patterns you already know, no need to learn new APIs.
* **Smooth Testing**: Write unit tests with in-memory channels; switch to real systems for integration.
* **Future-Proof**: Swap messaging technologies without changing your business logic.
* **Serverless-Ready**: Designed for both long-running services and ephemeral serverless functions.
* **Scale-Out**: Idiomatic architecture to build distributed topologies and scale-out Golang applications.
Think of `swarm` as net.Conn **for distributed messaging**: a universal, idiomatic interface you can depend on, regardless of the underlying transport.
Below we discussed why `swarm` exists and what problems it solves.
## Why Choose `swarm`?
Traditional messaging libraries have fundamental issues
| Traditional libraries | With `swarm` |
| ----------------------------- | ------------------------------- |
| Different SDK for each broker | One API for all brokers |
| Complex mocks for testing | In-memory channels for tests |
| Vendor lock-in | Portable across technologies |
| Sync APIs for async systems | Native async with Go channels |
| Manual error handling | Built-in retries & dead letters |
`swarm` solves these problems by providing a universal, idiomatic interface built on Go's concurrency model — embracing the asynchronous nature of distributed systems, making message flows explicit, and hiding vendor-specific complexity behind familiar Go channels.
See practical scenarios for `swarm` in [Storytelling](#storytelling---why-we-built-swarm).
## Getting started
The library requires **Go 1.24** or later due to usage of [generic alias types](https://go.dev/blog/alias-names#generic-alias-types).
The latest version of the library is available at `main` branch of this repository. All development, including new features and bug fixes, take place on the `main` branch using forking and pull requests as described in contribution guidelines. The stable version is available via Golang modules.
Use `go get` to retrieve the library and add it as dependency to your application.
```bash
go get -u github.com/fogfish/swarm
```
### When to Use `swarm`
**Perfect for:**
- **Event-driven architectures** - Decouple services with async messaging
- **Serverless applications** - Zero-boilerplate event consumers on AWS Lambda
- **Microservices** - Replace fragile HTTP calls with resilient messaging
- **High-throughput systems** - Scale message processing horizontally
- **Multi-cloud applications** - Abstract away broker differences
**You'll love it if you:**
- Want type-safe messaging without vendor lock-in
- Need to unit test message-driven code easily
- Prefer Go channels over learning broker-specific APIs
- Value clean, testable architecture patterns
## Advanced Usage and Next steps
* Learn [why we built `swarm`](#storytelling---why-we-built-swarm).
* [User guide](./doc/user-guide.md) help you with adopting the library.
* The library supports a variety of brokers out of the box. See the table at the beginning of this document for details. If you need to implement a broker that isn’t supported yet, refer to [Bring Your Own Broker](./doc/bring-your-own-broker.md) for guidance.
## Real-World Success Stories
Here's how `swarm` solves actual production problems that teams face every day.
### E-commerce Order Processing: The Cascading Failure Problem
Imagine you're building an e-commerce platform that processes thousands of orders per minute during Black Friday sales. Your order processing system needs to:
* Validate payment information
* Update inventory levels
* Send confirmation emails
* Update analytics systems
* Trigger fulfillment processes
What are problems with the traditional approach:
* Blocking Operations: Each service call blocks until completion, making the system slow during peak loads
* Cascading Failures: If the email service is down, the entire order process fails, even though the core business logic succeeded
* Poor Scalability: Cannot scale individual components independently
```go
func ProcessOrder(order *Order) error {
// Synchronous calls create cascading failures
if err := paymentService.Charge(order.Payment); err != nil {
return err // Customer sees error immediately
}
if err := inventoryService.Reserve(order.Items); err != nil {
return err // Payment charged but inventory failed
}
if err := emailService.SendConfirmation(order); err != nil {
return err // Everything fails if email is slow
}
return nil
}
```
Using the `swarm` library, we benefit from
* Immediate Response: Customers get instant order confirmation
* Fault Tolerance: If email service is down, orders still process successfully
* Independent Scaling: Each service can scale based on its own load patterns
```go
func ProcessOrder(order *Order) error {
// Immediate response to customer
if err := validateOrder(order); err != nil {
return err
}
// Async processing via channels
orderEvents <- OrderCreated{Order: order}
return nil // Customer gets immediate confirmation
}
// Background processing
go func() {
for event := range orderEvents {
paymentQueue <- PaymentRequest{Order: event.Order}
inventoryQueue <- InventoryUpdate{Order: event.Order}
emailQueue <- EmailNotification{Order: event.Order}
}
}()
```
### Serverless Event Processing: The SDK Complexity Problem
You're building a serverless application on AWS that needs to process events from multiple sources - SQS queues, S3 bucket notifications, EventBridge events, and DynamoDB streams. Each service has its own SDK with different patterns.
```go
// Different patterns for each service
func handleSQSEvent(event events.SQSEvent) error {
for _, record := range event.Records {
// SQS-specific parsing and acknowledgment
body := record.Body
receiptHandle := record.ReceiptHandle
// ... SQS-specific error handling
}
}
func handleS3Event(event events.S3Event) error {
for _, record := range event.Records {
// S3-specific parsing
bucket := record.S3.Bucket.Name
key := record.S3.Object.Key
// ... S3-specific error handling
}
}
```
Using the `swarm` library, we benefit from
* Unified Interface: Same code patterns work across all AWS services
* Easy Testing: Mock implementations for local development
* Technology Agnostic: Can switch from SQS to EventBridge without code changes
```go
events := eventbridge.Must(eventbridge.Listener().Build())
upload := s3.Must(s3.Listener().Build())
orders := sqs.Must(sqs.Listener().Build())
listen.Typed[UserEvent](events)
listen.Typed[DataUploaded](upload)
listen.Typed[Order](orders)
```
### Microservices Communication: The Synchronous Trap
You have a microservices architecture where the user service, notification service, and analytics service need to communicate. Traditional REST APIs create tight coupling and cascading failures.
```go
// Synchronous REST calls create brittleness
func CreateUser(user *User) error {
// Save user
if err := userDB.Save(user); err != nil {
return err
}
// Synchronous calls to other services
if err := notificationService.SendWelcomeEmail(user); err != nil {
return err // User creation fails if email fails
}
if err := analyticsService.TrackUserCreated(user); err != nil {
return err // User creation fails if analytics fails
}
return nil
}
```
Using the `swarm` library, we benefit from
* Resilient: Core functionality works even if downstream services are down
* Scalable: Services can process at their own pace
* Evolvable: Easy to add new services without changing existing code
```go
// Async communication via channels
func CreateUser(user *User) error {
// Core business logic
if err := userDB.Save(user); err != nil {
return err
}
// Async notifications
userEvents <- UserCreated{User: user}
return nil // Immediate success
}
// Services consume events independently
go func() {
for event := range userEvents {
notificationQueue <- WelcomeEmail{User: event.User}
analyticsQueue <- UserCreatedEvent{User: event.User}
}
}()
```
## How To Contribute
The library is [Apache Version 2.0](LICENSE) licensed and accepts contributions via GitHub pull requests:
1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Added some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request
The build and testing process requires [Go](https://golang.org) version 1.24 or later.
**build** and **test** library.
```bash
git clone https://github.com/fogfish/swarm
cd swarm
go test ./...
```
### commit message
The commit message helps us to write a good release note, speed-up review process. The message should address two question what changed and why. The project follows the template defined by chapter [Contributing to a Project](http://git-scm.com/book/ch5-2.html) of Git book.
### bugs
If you experience any issues with the library, please let us know via [GitHub issues](https://github.com/fogfish/swarm/issues). We appreciate detailed and accurate reports that help us to identity and replicate the issue.
## Community & Support
- **Documentation**: [User Guide](./doc/user-guide.md) • [API Reference](https://pkg.go.dev/github.com/fogfish/swarm)
- **Questions**: [GitHub Discussions](https://github.com/fogfish/swarm/discussions) for Q&A and ideas
- **Issues**: [GitHub Issues](https://github.com/fogfish/swarm/issues) for bug reports and feature requests
- **Examples**: [Live examples](./broker/) for all supported brokers
## License
[](LICENSE)