{"id":49584992,"url":"https://github.com/robertwhurst/conduit","last_synced_at":"2026-05-03T22:04:59.903Z","repository":{"id":320622495,"uuid":"1082418614","full_name":"RobertWHurst/Conduit","owner":"RobertWHurst","description":"A lightweight, transport-agnostic messaging framework for Go. Build distributed systems with multiple encoding formats, streaming support, and pluggable transports like NATS.","archived":false,"fork":false,"pushed_at":"2025-10-28T00:15:00.000Z","size":85,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-12-26T20:30:51.704Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RobertWHurst.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-24T08:01:44.000Z","updated_at":"2025-10-28T00:14:47.000Z","dependencies_parsed_at":"2025-10-24T21:28:38.574Z","dependency_job_id":null,"html_url":"https://github.com/RobertWHurst/Conduit","commit_stats":null,"previous_names":["robertwhurst/conduit"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/RobertWHurst/Conduit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobertWHurst%2FConduit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobertWHurst%2FConduit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobertWHurst%2FConduit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobertWHurst%2FConduit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RobertWHurst","download_url":"https://codeload.github.com/RobertWHurst/Conduit/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobertWHurst%2FConduit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32586189,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T06:36:36.687Z","status":"ssl_error","status_checked_at":"2026-05-03T06:36:09.306Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-03T22:04:58.606Z","updated_at":"2026-05-03T22:04:59.892Z","avatar_url":"https://github.com/RobertWHurst.png","language":"Go","funding_links":["https://github.com/sponsors/RobertWHurst"],"categories":[],"sub_categories":[],"readme":"# Conduit\n\nA lightweight, transport-agnostic messaging framework for Go. Build distributed systems with multiple encoding formats, streaming support, and pluggable transports like NATS.\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/RobertWHurst/conduit.svg)](https://pkg.go.dev/github.com/RobertWHurst/conduit)\n[![Go Report Card](https://goreportcard.com/badge/github.com/RobertWHurst/conduit)](https://goreportcard.com/report/github.com/RobertWHurst/conduit)\n[![License](https://img.shields.io/github/license/RobertWHurst/conduit)](LICENSE)\n[![Sponsor](https://img.shields.io/static/v1?label=Sponsor\u0026message=%E2%9D%A4\u0026logo=GitHub\u0026color=%23fe8e86)](https://github.com/sponsors/RobertWHurst)\n\n## Table of Contents\n\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Core Concepts](#core-concepts)\n  - [Conduit](#conduit)\n  - [Service Communication](#service-communication)\n  - [Message Encoding](#message-encoding)\n  - [Transports](#transports)\n- [Messaging Patterns](#messaging-patterns)\n  - [Send (Fire and Forget)](#send-fire-and-forget)\n  - [Request/Reply](#requestreply)\n  - [Event Binding](#event-binding)\n  - [Queue Binding](#queue-binding)\n- [Message Handling](#message-handling)\n  - [Decoding Messages](#decoding-messages)\n  - [Reading Raw Data](#reading-raw-data)\n  - [Replying to Messages](#replying-to-messages)\n- [Built-in Encoders](#built-in-encoders)\n  - [JSON Encoder](#json-encoder)\n  - [MessagePack Encoder](#messagepack-encoder)\n  - [Protocol Buffers Encoder](#protocol-buffers-encoder)\n- [Built-in Transports](#built-in-transports)\n  - [NATS Transport](#nats-transport)\n  - [Chunked Streaming](#chunked-streaming)\n- [Advanced Usage](#advanced-usage)\n  - [Custom Encoders](#custom-encoders)\n  - [Custom Transports](#custom-transports)\n- [Help Welcome](#help-welcome)\n- [License](#license)\n- [Related Projects](#related-projects)\n\n## Features\n\n- 📡 **Event-Driven Messaging** - Broadcast events across services with automatic routing\n- 🌊 **Streaming Support** - Send messages of any size without loading into memory\n- 🔌 **Pluggable Transports** - Built-in NATS support, easily add RabbitMQ, Kafka, or Redis\n- 📦 **Multiple Encoders** - JSON, MessagePack, and Protocol Buffers included\n- ⚖️ **Load Balancing** - Queue bindings distribute work across service instances\n- 🔄 **Request/Reply** - Synchronous request/reply pattern when you need it\n\n## Installation\n\n```bash\ngo get github.com/RobertWHurst/conduit\n```\n\nFor NATS transport:\n\n```bash\ngo get github.com/nats-io/nats.go\n```\n\n## Quick Start\n\nHere's a simple example showing event-driven communication between services over NATS:\n\n**User Service** (broadcasts events):\n\n```go\n// Connect to NATS\nnc, _ := natsgo.Connect(\"nats://localhost:4222\")\n\n// Create conduit\nconduit := conduit.New(\"user-service\", natstransport.New(nc), jsonencoder.New())\n\n// Broadcast user.created event when a user signs up\nconduit.Service(\"notification-service\").Send(\"user.created\", UserCreatedEvent{\n    UserID: 123,\n    Email:  \"alice@example.com\",\n    Name:   \"Alice\",\n})\n```\n\n**Notification Service** (listens for events):\n\n```go\n// Connect to NATS\nnc, _ := natsgo.Connect(\"nats://localhost:4222\")\n\n// Create conduit\nconduit := conduit.New(\"notification-service\", natstransport.New(nc), jsonencoder.New())\n\n// Listen for user.created events\nconduit.Bind(\"user.created\").To(func(msg *conduit.Message) {\n    var event UserCreatedEvent\n    msg.Into(\u0026event)\n    \n    // Send welcome email\n    sendWelcomeEmail(event.Email, event.Name)\n})\n```\n\n## Core Concepts\n\n### Conduit\n\nA conduit allows sending and receiving data from different services. Each conduit takes the name of the service it represents, a transport for facilitating communication, and an encoder for encoding and decoding structs.\n\nEach service should have one conduit instance. The service name identifies your service to others in the distributed system. The transport handles the underlying message delivery (like NATS). The encoder serializes and deserializes your data structures.\n\n```go\nconduit := conduit.New(\n    \"my-service\",\n    natstransport.New(natsConn),\n    jsonencoder.New(),\n)\ndefer conduit.Close()\n```\n\n### Service Communication\n\nTo send messages to another service, create a service client using `conduit.Service()` with the target service name. This returns a `ServiceClient` that provides methods for sending one-way messages or making request/reply calls.\n\n```go\n// Create a service client for \"notification-service\"\nnotificationService := conduit.Service(\"notification-service\")\n\n// Send one-way message\nnotificationService.Send(\"email.send\", EmailRequest{\n    To:      \"user@example.com\",\n    Subject: \"Welcome!\",\n})\n\n// Request with reply (blocks until response or timeout)\nvar result EmailResult\nnotificationService.Request(\"email.send\", emailReq).Into(\u0026result)\n\n// Custom timeout\nnotificationService.RequestWithTimeout(\"email.send\", emailReq, 5*time.Second).Into(\u0026result)\n\n// Context-based cancellation\nctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)\ndefer cancel()\nnotificationService.RequestWithCtx(ctx, \"email.send\", emailReq).Into(\u0026result)\n\n// Communicate with instances of the same service\nselfClient := conduit.Self()\nselfClient.Send(\"cache.invalidate\", CacheKey{Key: \"user:123\"})\n```\n\n### Message Encoding\n\nEncoders serialize and deserialize messages automatically. Send strongly-typed Go structs without manual marshaling.\n\nEncoders handle five types of values:\n\n- **Structs** - Marshaled using the encoder (JSON, MessagePack, Protocol Buffers)\n- **Strings** - Sent as-is without encoding\n- **Byte slices** - Sent as-is without encoding\n- **io.Reader** - Streamed directly without buffering\n- **nil** - Send events without payload data\n\n```go\n// Send a struct\nconduit.Service(\"notification-service\").Send(\"user.created\", User{ID: 123, Name: \"Alice\"})\n\n// Send a string\nconduit.Service(\"log-service\").Send(\"log.info\", \"User logged in\")\n\n// Send bytes\nconduit.Service(\"analytics-service\").Send(\"event.track\", []byte{0x01, 0x02, 0x03})\n\n// Stream a file\nfile, _ := os.Open(\"report.pdf\")\nconduit.Service(\"storage-service\").Send(\"file.upload\", file)\n\n// Send event without payload\nconduit.Service(\"cache-service\").Send(\"cache.invalidate\", nil)\n```\n\n### Transports\n\nTransports handle the underlying message delivery between services. Conduit includes a NATS transport with support for reliable messaging and streaming.\n\nThe transport interface is simple, making it straightforward to add support for other brokers like RabbitMQ, Redis, or Kafka.\n\nThe NATS transport uses a chunked streaming protocol to send messages of any size without loading them into memory. Messages are split into 16KB chunks and streamed between services.\n\n```go\nnc, _ := natsgo.Connect(\"nats://localhost:4222\")\nconduit := conduit.New(\"my-service\", natstransport.New(nc), jsonencoder.New())\n```\n\n## Messaging Patterns\n\n### Send (Fire and Forget)\n\nSend delivers one-way messages without waiting for a reply. This is ideal for broadcasting events, logging, and notifications where you don't need confirmation of processing.\n\n```go\n// Broadcast login event\nconduit.Service(\"analytics-service\").Send(\"user.login\", LoginEvent{\n    UserID:    123,\n    Timestamp: time.Now(),\n    IPAddress: \"192.168.1.1\",\n})\n\n// Send log message\nconduit.Service(\"log-service\").Send(\"log.info\", \"User 123 logged in\")\n```\n\n### Request/Reply\n\nRequest/Reply is a synchronous pattern where the sender waits for a response. Use this when you need a result back from another service.\n\n```go\n// Make request and wait for reply (30 second default timeout)\nvar result ProcessResult\nif err := conduit.Service(\"worker-service\").Request(\"job.process\", job).Into(\u0026result); err != nil {\n    log.Fatal(err)\n}\n\n// Custom timeout\nconduit.Service(\"worker-service\").RequestWithTimeout(\"job.process\", job, 5*time.Second).Into(\u0026result)\n\n// Context-based cancellation\nctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)\ndefer cancel()\nconduit.Service(\"worker-service\").RequestWithCtx(ctx, \"job.process\", job).Into(\u0026result)\n```\n\n### Event Binding\n\nBind to subjects to receive messages. All service instances with the same binding will receive every message (broadcast).\n\nUse `Next()` to process messages in a loop, or `To()` to handle messages with a callback:\n\n```go\n// Option 1: Process messages in a loop\nbinding := conduit.Bind(\"user.created\")\ngo func() {\n    for {\n        var event UserCreatedEvent\n        if err := binding.Next().Into(\u0026event); err != nil {\n            if err == conduit.ErrBindingClosed {\n                break\n            }\n            log.Printf(\"Failed to decode: %v\", err)\n            continue\n        }\n        updateCache(event)\n    }\n}()\n\n// Option 2: Use a handler function\nbinding := conduit.Bind(\"user.created\").To(func(msg *conduit.Message) {\n    var event UserCreatedEvent\n    msg.Into(\u0026event)\n    updateCache(event)\n})\n```\n\nUnbind when done (safe to call multiple times):\n\n```go\nbinding := conduit.Bind(\"user.created\")\ndefer binding.Unbind()\n```\n\nBind with automatic cleanup using context:\n\n```go\n// Binding automatically unbinds when context is cancelled\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\n\nbinding := conduit.Bind(\"user.created\").WithCtx(ctx).To(func(msg *conduit.Message) {\n    var event UserCreatedEvent\n    msg.Into(\u0026event)\n    updateCache(event)\n})\n\n// No need to manually unbind - happens automatically when ctx is cancelled\n```\n\n**Note:** `WithCtx` can only be called once per binding and will panic if called multiple times.\n\n### Queue Binding\n\nQueue bindings distribute messages across service instances - only one instance receives each message. Use this for load balancing work.\n\n```go\n// Each message goes to only one instance\nconduit.QueueBind(\"job.process\").To(func(msg *conduit.Message) {\n    var job Job\n    msg.Into(\u0026job)\n    processJob(job)\n})\n```\n\n**When to use:**\n\n- **Bind()** - All instances receive the message (cache invalidation, config updates)\n- **QueueBind()** - One instance receives the message (job processing, work distribution)\n\nBoth can be used on the same subject simultaneously.\n\n## Message Handling\n\n### Decoding Messages\n\nUse `Into()` to decode messages into Go structs. The decoder respects `MaxDecodeSize` (default 5MB) to prevent memory exhaustion.\n\n```go\nconduit.Bind(\"order.created\").To(func(msg *conduit.Message) {\n    var event OrderCreatedEvent\n    if err := msg.Into(\u0026event); err != nil {\n        log.Printf(\"Failed to decode: %v\", err)\n        return\n    }\n    \n    processOrder(event)\n})\n```\n\nChange the decode size limit:\n\n```go\nconduit.MaxDecodeSize = 10 * 1024 * 1024 // 10MB\n```\n\n### Reading Raw Data\n\nMessages implement `io.Reader` for streaming large data without loading it into memory.\n\n```go\nconduit.Bind(\"file.upload\").To(func(msg *conduit.Message) {\n    file, _ := os.Create(\"/tmp/upload\")\n    defer file.Close()\n    \n    io.Copy(file, msg)\n    log.Println(\"File uploaded\")\n})\n```\n\n### Replying to Messages\n\nUse `Reply()` to respond to requests. Only messages sent via `Request()` have reply subjects - messages from `Send()` cannot be replied to.\n\n```go\nconduit.Bind(\"job.process\").To(func(msg *conduit.Message) {\n    var job Job\n    msg.Into(\u0026job)\n    \n    result := processJob(job)\n    \n    if err := msg.Reply(result); err != nil {\n        log.Printf(\"Failed to reply: %v\", err)\n    }\n})\n```\n\n`Reply()` accepts structs, strings, byte slices, and `io.Reader` values.\n\n## Built-in Encoders\n\n### JSON Encoder\n\nJSON encoding is human-readable and widely supported. Good for development and debugging.\n\n```go\nimport \"github.com/RobertWHurst/conduit/encoder/jsonencoder\"\n\nconduit := conduit.New(\"my-service\", transport, jsonencoder.New())\n```\n\nUse JSON when:\n\n- Human-readable messages are important\n- Broad compatibility is needed\n- Performance is not critical\n\n### MessagePack Encoder\n\nMessagePack is a fast, compact binary format - approximately 5x faster than JSON with smaller message sizes.\n\n```go\nimport \"github.com/RobertWHurst/conduit/encoder/msgpackencoder\"\n\nconduit := conduit.New(\"my-service\", transport, msgpackencoder.New())\n```\n\nUse MessagePack when:\n\n- High throughput is needed\n- Bandwidth or memory is constrained\n- All services are under your control\n\n### Protocol Buffers Encoder\n\nProtocol Buffers provides type safety, schema validation, and cross-language compatibility.\n\n**Define a schema:**\n\n```protobuf\nsyntax = \"proto3\";\n\nmessage UserCreatedEvent {\n  int64 user_id = 1;\n  string email = 2;\n  string name = 3;\n}\n```\n\n**Generate Go code:**\n\n```bash\nprotoc --go_out=. --go_opt=paths=source_relative events.proto\n```\n\n**Use with Conduit:**\n\n```go\nimport (\n    \"github.com/RobertWHurst/conduit/encoder/protobufencoder\"\n    pb \"github.com/myuser/myapp/proto\"\n)\n\nconduit := conduit.New(\"my-service\", transport, protobufencoder.New())\n\n// Send protobuf message\nconduit.Service(\"notification-service\").Send(\"user.created\", \u0026pb.UserCreatedEvent{\n    UserId: 123,\n    Email:  \"alice@example.com\",\n    Name:   \"Alice\",\n})\n```\n\nUse Protocol Buffers when:\n\n- Type-safe schemas are needed\n- Supporting multiple languages\n- Backward/forward compatibility is important\n\n## Built-in Transports\n\n### NATS Transport\n\nThe NATS transport provides reliable, high-performance messaging with support for streaming, request/reply, and events.\n\n```go\nimport (\n    \"github.com/RobertWHurst/conduit/transport/natstransport\"\n    natsgo \"github.com/nats-io/nats.go\"\n)\n\nnc, _ := natsgo.Connect(\"nats://localhost:4222\")\ntransport := natstransport.New(nc)\nconduit := conduit.New(\"my-service\", transport, jsonencoder.New())\n```\n\nFeatures:\n\n- At-most-once delivery\n- Subject-based routing\n- Clustering for high availability\n- TLS encryption and authentication\n\n### Chunked Streaming\n\nThe NATS transport streams messages of any size without loading them into memory. Data is sent in 16KB chunks.\n\n```go\n// Stream large file\nfile, _ := os.Open(\"large-file.dat\")\nconduit.Service(\"storage-service\").Send(\"file.store\", file)\n```\n\nReceiver streams directly to disk:\n\n```go\nconduit.Bind(\"file.store\").To(func(msg *conduit.Message) {\n    outFile, _ := os.Create(\"uploaded-file.dat\")\n    defer outFile.Close()\n    io.Copy(outFile, msg)\n})\n```\n\nProtocol details:\n\n- Chunk size: 16KB (configurable via `nats.ChunkSize`)\n- Send timeout: 5 seconds (configurable via `nats.SendTimeout`)\n- Subject format: `conduit.\u003cservice-name\u003e`\n\n## Advanced Usage\n\n### Custom Encoders\n\nImplement the `Encoder` interface to add support for other formats:\n\n```go\ntype Encoder interface {\n    Encode(v any) ([]byte, error)\n    Decode(data []byte, v any) error\n}\n\ntype MyEncoder struct{}\n\nfunc (e *MyEncoder) Encode(v any) ([]byte, error) {\n    // Your encoding logic\n    return encoded, nil\n}\n\nfunc (e *MyEncoder) Decode(data []byte, v any) error {\n    // Your decoding logic\n    return nil\n}\n\nconduit := conduit.New(\"my-service\", transport, \u0026MyEncoder{})\n```\n\n### Custom Transports\n\nImplement the `Transport` interface to add support for other message brokers:\n\n```go\ntype Transport interface {\n    Send(serviceName, subject, sourceServiceName, replySubject string, reader io.Reader) error\n    Handle(serviceName string, handler func(sourceServiceName, subject, replySubject string, reader io.Reader))\n    HandleQueue(serviceName string, handler func(sourceServiceName, subject, replySubject string, reader io.Reader))\n    Close() error\n}\n\ntype MyTransport struct{}\n\nfunc (t *MyTransport) Send(serviceName, subject, sourceServiceName, replySubject string, reader io.Reader) error {\n    data, _ := io.ReadAll(reader)\n    // Send to your message broker\n    return nil\n}\n\nfunc (t *MyTransport) Handle(serviceName string, handler func(sourceServiceName, subject, replySubject string, reader io.Reader)) {\n    // Subscribe to broadcast messages\n}\n\nfunc (t *MyTransport) HandleQueue(serviceName string, handler func(sourceServiceName, subject, replySubject string, reader io.Reader)) {\n    // Subscribe to queue messages\n}\n\nfunc (t *MyTransport) Close() error {\n    return nil\n}\n```\n\n## Help Welcome\n\nIf you want to support this project with coffee money, it's greatly appreciated.\n\n[![sponsor](https://img.shields.io/static/v1?label=Sponsor\u0026message=%E2%9D%A4\u0026logo=GitHub\u0026color=%23fe8e86)](https://github.com/sponsors/RobertWHurst)\n\nIf you're interested in providing feedback or would like to contribute, please feel free to do so. I recommend first [opening an issue][feature-request] expressing your feedback or intent to contribute a change. From there we can consider your feedback or guide your contribution efforts. Any and all help is greatly appreciated.\n\nThank you!\n\n[feature-request]: https://github.com/RobertWHurst/Conduit/issues/new?template=feature_request.md\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n## Related Projects\n\n- [Navaros](https://github.com/RobertWHurst/Navaros) - HTTP framework for Go with powerful pattern matching\n- [Velaros](https://github.com/RobertWHurst/Velaros) - WebSocket framework for Go with message routing\n- [Zephyr](https://github.com/TelemetryTV/Zephyr) - Microservice framework built on Navaros with service discovery and streaming\n- Eurus - WebSocket API gateway framework (upcoming, integrates with Velaros)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobertwhurst%2Fconduit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frobertwhurst%2Fconduit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobertwhurst%2Fconduit/lists"}