{"id":38769386,"url":"https://github.com/sreekar2307/queue","last_synced_at":"2026-01-17T12:03:29.857Z","repository":{"id":285528931,"uuid":"958342639","full_name":"sreekar2307/queue","owner":"sreekar2307","description":"Distributed queue","archived":false,"fork":false,"pushed_at":"2025-06-29T18:29:17.000Z","size":320,"stargazers_count":28,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-29T19:32:27.822Z","etag":null,"topics":["boltdb","distributed-systems","dragonboat-raft","golang","grpc","persistent","protobuf","queue","raft"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sreekar2307.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-01T03:27:36.000Z","updated_at":"2025-06-29T18:29:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"b47a955e-bd69-4d15-ad5d-9e1bd74c9b81","html_url":"https://github.com/sreekar2307/queue","commit_stats":null,"previous_names":["sreekar2307/queue"],"tags_count":29,"template":false,"template_full_name":null,"purl":"pkg:github/sreekar2307/queue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sreekar2307%2Fqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sreekar2307%2Fqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sreekar2307%2Fqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sreekar2307%2Fqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sreekar2307","download_url":"https://codeload.github.com/sreekar2307/queue/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sreekar2307%2Fqueue/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28508464,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T11:50:55.898Z","status":"ssl_error","status_checked_at":"2026-01-17T11:50:55.569Z","response_time":85,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["boltdb","distributed-systems","dragonboat-raft","golang","grpc","persistent","protobuf","queue","raft"],"created_at":"2026-01-17T12:03:29.765Z","updated_at":"2026-01-17T12:03:29.841Z","avatar_url":"https://github.com/sreekar2307.png","language":"Go","readme":"[![Go Report Card](https://goreportcard.com/badge/github.com/sreekar2307/queue)](https://goreportcard.com/report/github.com/sreekar2307/queue) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/gojp/goreportcard/blob/master/LICENSE)\n\n# Queue\n\nA fault-tolerant distributed message queue inspired by Kafka — written in Go.\nIt uses Raft (via [Dragonboat](https://github.com/lni/dragonboat)) for consensus,\nBoltDB for per-partition storage, and features consumer groups with sticky re-balancing.\n\n---\n\n## ✨ Features\n\n### ✅ Core (V0)\n- **Topic/Partition model** with persistent on-disk storage (BoltDB per partition)\n- **Consumer Groups** with sticky partition assignments\n- **At-least-once delivery**\n- **gRPC API** for Producer/Consumer interactions\n- **Distributed replication using Dragonboat Raft**\n- **IOnDiskStateMachine** integration for crash-safe FSM state\n- **Heartbeats and Sticky Rebalancing** for dynamic consumer group coordination\n- **Manual `Ack` and `Poll` APIs** to control delivery semantics\n\n### 🧠 Under the Hood\n- Raft Shards map 1:1 with queue partitions (one Raft group per partition)\n- Storage engine is decoupled using interface-based abstractions\n- Custom `MessageService` orchestrates routing logic and partition calculations\n- No centralized metadata — each broker stores partition state locally\n\n---\n\n## 🚀 Getting Started\n\n### 🛠 Requirements\n- Go 1.24\n- Protobuf Compiler (`protoc`) if you want to generate gRPC code\n\n### 🔧 Running a Single Node Broker\n\n```bash\n$ go build -o queue . \u0026\u0026 ./queue\n```\nthis will start a single broker instance with grpc enabled on 0.0.0.0:8000,\ncheck out [queue.proto](https://buf.build/sreekar2307/queue/docs) for the proto definition\n\n\n---\n### 🧩 Architecture Overview\n\n```text\n                        +----------------+\n                        |     Client     |\n                        | (Producer/Cons)|\n                        +-------+--------+\n                                |\n                                v\n                     +----------+----------+\n                     |        Broker        |   \u003c-- Handles API (Send, Poll, Ack, etc.)\n                     |  (Multiple instances)|   \u003c-- Maps topic-partitions, maintains metadata\n                     +----------+----------+\n                                |\n       +------------------------+-------------------------+\n       |                        |                         |\n       v                        v                         v\n+---------------+      +---------------+         +---------------+\n|   Partition 0 |      |   Partition 1 |   ...   |   Partition N |\n| (TopicX-P0)   |      | (TopicX-P1)   |         | (TopicY-PM)   |\n+------+--------+      +------+--------+         +------+--------+\n       |                      |                         |\n       | Raft Group (Shard)   | Raft Group (Shard)      | Raft Group (Shard)\n       |                      |                         |\n+------+------+        +------+------+           +------+------+\n| Broker A    |        | Broker B    |           | Broker C    |\n| (Leader)    |        | (Leader)    |           | (Leader)    |\n| BoltDB P0   |        | BoltDB P1   |           | BoltDB PM   |\n+------+------+        +------+------+           +------+------+\n       |                      |                         |\n+------+------+        +------+------+           +------+------+\n| Broker B    |        | Broker C    |           | Broker A    |\n| (Follower)  |        | (Follower)  |           | (Follower)  |\n+-------------+        +-------------+           +-------------+\n\n                    \u003c\u003c Metadata Raft Group \u003e\u003e\n                    \u003c\u003c Replicates metadata to all brokers \u003e\u003e\n+---------------------------------------------------------------+\n|          Each broker has its own `metadata.bolt` file         |\n|     - Topic definitions                                       |\n|     - Partition assignments                                   |\n|     - Consumer group membership \u0026 offsets                     |\n|     Metadata is kept in sync via a Raft shard across brokers  |\n+---------------------------------------------------------------+\n```\n\n\n## 🔧 Component Breakdown\n\n### 1. Transport\n\nHandles incoming gRPC and HTTP requests, and translates them into internal commands.\n\n- `grpc/server.go`: gRPC transport layer\n- `http/server.go`: Optional HTTP transport layer\n\n---\n\n### 2. Queue\n\nMain orchestration logic:\n- Topic and partition creation\n- Routing messages\n- Interacting with FSMs and services\n\nActs as the glue between transports, services, and Raft.\n\n---\n\n### 3. FSMs (Finite State Machines)\n\nImplements Dragonboat's `IOnDiskStateMachine`:\n\n- `BrokerFSM`: Tracks broker, topic, consumer metadata \n- `MessageFSM`: Handles message persistence, ack, polling\n\n---\n\n### 4. Service Layer\n\nDomain logic for various operations:\n\n- `BrokerService`: Broker registration, shard info for each partition, entire cluster metadata\n- `ConsumerService`: Consumer group registration, re-balancing\n- `TopicService`: Topic creation and introspection\n- `MessageService`: Append, poll, ack, etc.\n\n---\n\n### 5. Storage Layer\n\nEach partition’s finite state machine (FSM) persists its data in its own BoltDB file.\nso in a n-node cluster every message in a partition is stored on its leader plus n-1 additional nodes.\nDragonboat takes care of Raft log replication for these partition shards. Separately,\nall cluster metadata—topic definitions, consumer group state, and shard membership \nis kept in sync across every broker via a dedicated Raft shard.\n\n---\n\n### 6. Partition Assignors\n\nImplements partition assignment strategies (currently hardcoded to sticky partition assignment):\n\n- Assigns partitions to consumers\n- Triggers rebalance on consumer join/leave\n- Can be extended with more strategies (e.g., round-robin)\n\n---\n\n## ⚙️ Configuration\n\nThis system uses [Viper](https://github.com/spf13/viper) for configuration loading.\n\n### Supported Sources:\n\n- **Flags** (via `pflag`)\n- **Environment Variables**\n- **All config types supported by viper**\n\n---\n\n## 🛰️ gRPC Proxy Layer\n\nTo simplify client-broker interaction, a gRPC proxy is introduced at the client side.\n\n- The proxy determines the **correct broker** using metadata (Topic, Partition).\n- Routing is abstracted from application logic.\n- Built using a **custom gRPC resolver + load balancer** that:\n  - Fetches broker topology via lightweight **HTTP endpoints**.\n  - Performs **refreshes periodically** to keep metadata in sync.\n- Enables clean and scalable partition-level request routing.\n\nThis ensures that the application client simply calls `Poll`, `Ack`, or `Append`, without needing to know broker details.\n\n---\n\n## ➕ Dynamic Broker Addition\n\nThe queue system supports **adding brokers dynamically** to the cluster.\n\n- New brokers **can be added without leader restart**\n- New **partitions can be assigned** to the new brokers.\n- Existing partitions are **not rebalanced** — no reshuffling or migration is performed.\n\nThis makes horizontal scaling straightforward without introducing complexity or data shuffling overhead.\n","funding_links":[],"categories":["\u003ca name=\"Go\"\u003e\u003c/a\u003eGo"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsreekar2307%2Fqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsreekar2307%2Fqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsreekar2307%2Fqueue/lists"}