{"id":22511279,"url":"https://github.com/varthanv/pub-sub","last_synced_at":"2025-06-18T17:37:46.997Z","repository":{"id":265901864,"uuid":"895879273","full_name":"VarthanV/pub-sub","owner":"VarthanV","description":"A toy real-time Pub-Sub messaging system with durable queues, dynamic routing, and WebSocket support for seamless communication. 🌐🚀","archived":false,"fork":false,"pushed_at":"2024-12-01T12:05:59.000Z","size":612,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-26T08:08:09.724Z","etag":null,"topics":["concurrency","golang","message-queues","pub-sub"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VarthanV.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-29T05:19:28.000Z","updated_at":"2024-12-01T12:06:02.000Z","dependencies_parsed_at":"2024-12-01T13:19:13.132Z","dependency_job_id":"2c6f6b5e-fba2-4bbd-bee0-efd9703cd894","html_url":"https://github.com/VarthanV/pub-sub","commit_stats":null,"previous_names":["varthanv/pub-sub"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/VarthanV/pub-sub","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VarthanV%2Fpub-sub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VarthanV%2Fpub-sub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VarthanV%2Fpub-sub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VarthanV%2Fpub-sub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VarthanV","download_url":"https://codeload.github.com/VarthanV/pub-sub/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VarthanV%2Fpub-sub/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260600258,"owners_count":23034676,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","golang","message-queues","pub-sub"],"created_at":"2024-12-07T02:10:40.178Z","updated_at":"2025-06-18T17:37:41.983Z","avatar_url":"https://github.com/VarthanV.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🚀 Pub-Sub : A toy Messaging System\n\nA toy real-time Pub-Sub messaging system with durable queues, dynamic routing, and WebSocket support for seamless communication. 🌐🚀\n\n# Architecture\n\n![](./assets/architecture.png)\n\n# Pipeline\n\n- Minimal frontend (WIP)\n\n- Client and server libraries in Go to interact with this stack\n\n- Lock-free datastructures to reduce contention and increase throughput.\n\n\n## ⚡ Project Layout\n\n```bash\n├── README.md\n├── assets\n│   └── working.png\n├── binding\n│   └── binding.go\n├── broker\n│   ├── helpers.go\n│   └── main.go\n├── errors\n│   └── main.go\n├── exchange\n│   └── main.go\n├── go.mod\n├── go.sum\n├── image.png\n├── main.go\n├── messages\n│   └── message.go\n├── models\n│   ├── base.go\n│   ├── binding.go\n│   ├── exchange.go\n│   ├── message.go\n│   └── queue.go\n├── pkg\n│   ├── config\n│   │   └── config.go\n│   └── database\n│       └── database.go\n├── pub-sub.db\n├── queue\n│   ├── main.go\n│   └── main_test.go\n└── server\n    ├── binding.go\n    ├── controller.go\n    ├── exchange.go\n    ├── main.go\n    ├── publish.go\n    ├── queue.go\n    ├── routes.go\n    └── subscription.go\n\n13 directories, 30 files\n```\n\n## 🌟 Features\n\n### 🛠️ Core Functionalities\n- **Exchange Management**: Supports multiple exchange types, including `FanOut` and `Direct`, to route messages effectively.\n- **Queue Management**: Durable and non-durable queues for persistent or transient message storage.\n- **Dynamic Bindings**: Bind queues to exchanges with flexible routing keys.\n- **WebSocket Subscriptions**: Real-time updates with WebSocket-based subscription mechanisms.\n- **Message Persistence**: Persist messages for durable queues to ensure reliability and fault tolerance.\n\n### ⚡ Built for Performance\n- Lightweight and highly concurrent architecture using **Go's goroutines** and **sync primitives**.\n- **Semaphore-based worker control** for concurrent operations like real-time updates and database syncing.\n- Optimized database interactions with **GORM**, including bulk inserts and conflict handling.\n\n### 💾 Database-Driven\n- Persistent state management with a relational database for exchanges, bindings, and queues.\n- Preloaded relationships (`Preload(\"Bindings.Queues\")`) to minimize query overhead.\n- Many-to-many relationships with **GORM's advanced association handling**.\n\n### 🔄 Real-Time Data Flow\n- Publish-Subscribe pattern implemented with efficient routing algorithms.\n- FanOut messages to all bound queues or Direct messages to specific routing keys.\n- Seamless WebSocket integration to notify subscribers in real time.\n\n## 📚 Tech Stack\n- **Go**: A blazing-fast, statically typed language perfect for concurrent systems.\n- **GORM**: ORM for seamless database interactions and relationship management.\n- **WebSocket**: Enables real-time communication between broker and clients.\n- **PostgreSQL/MySQL**: Supported as the database backend for reliable persistence.\n- **Logrus**: Enhanced logging for debugging and system monitoring.\n\n## 🔧 How It Works\n\n### 🎯 Core Models\n1. **Exchange**: Acts as a router for messages. Each exchange can have:\n   - A `FanOut` type for broadcasting messages to all bound queues.\n   - A `Direct` type for routing messages based on a key.\n   \n2. **Queue**: Stores messages for consumers, with options for:\n   - Durability to survive broker restarts.\n   - Non-durability for lightweight, temporary storage.\n\n3. **Binding**: Connects an exchange to one or more queues with a routing key.\n\n### 🌀 Broker Lifecycle\n1. **Startup**:\n   - Loads exchanges, bindings, and queues from the database.\n   - Rebuilds in-memory mappings for fast runtime operations.\n\n2. **Publish**:\n   - Routes messages based on exchange type and routing key.\n   - Supports concurrent publishing with **Go routines**.\n\n3. **Subscribe**:\n   - Real-time updates via WebSocket connections for bound queues.\n\n4. **Checkpointing**:\n   - Periodic database synchronization to persist messages for durable queues.\n\n## 🎉 Why You'll Love This\n- 🌐 **Scalable**: Handles high throughput and concurrency with ease.\n- 🔒 **Reliable**: Durable message storage ensures no data loss.\n- 🧩 **Extensible**: Add new exchange types or integrate with other services effortlessly.\n- 🛠️ **Developer-Friendly**: Clean architecture and logs make debugging a breeze.\n\n## 🏁 Getting Started\n\n### 🔨 Prerequisites\n- **Go** (1.18+)\n\n\n\n## Environment Variables  \n\n### Server Configuration  \n- **`HTTP_PORT`**  \n  The port on which the application listens for incoming HTTP requests.  \n  - **Default:** `8060`  \n  - Example: `http://localhost:8060`  \n\n- **`ALLOWED_HOSTS`**  \n  Specifies the hosts allowed to access the server. Use `*` to allow all hosts (for development purposes).  \n  - **Default:** `*`  \n  - Note: Restrict this in production for security.  \n\n### Database Configuration  \n- **`DATABASE_NAME`**  \n  The name of the database used by the application.  \n  - **Default:** `pub-sub`  \n\n### Migration Configuration  \n- **`DO_MIGRATIONS`**  \n  Indicates whether to run database migrations automatically at application startup.  \n  - **Default:** `true`  \n  - Values:  \n    - `true` - Run migrations  \n    - `false` - Skip migrations  \n\n### Real-Time Updates Configuration  \n- **`CHECKPOINT_IN_SECONDS`**  \n  The interval, in seconds, for checkpointing or saving the application state.  \n  - **Default:** `10`  \n\n- **`MAX_WORKERS_ALLOWED_CONCURRENTLY_FOR_REAL_TIME_UPDATES`**  \n  The maximum number of worker threads/processes allocated for handling real-time updates.  \n  - **Default:** `10`  \n\n### Synchronization Configuration  \n- **`WORKERS_ALLOWED_FOR_SYNC`**  \n  The maximum number of worker threads/processes allocated for synchronization tasks.  \n  - **Default:** `20`  \n\n## Notes  \n- Adjust these variables as per your environment and performance requirements.  \n- Ensure `ALLOWED_HOSTS` is set appropriately in production to avoid unauthorized access.  \n\n- Check ``.env.example`` for same env\n\n### 🚀 Run the Broker\n```bash\ngit clone https://github.com/VarthanV/pub-sub.git\ncd pub-sub\ngo run main.go\n```\n\n\n## Screenshots\n\n![alt text](./assets/working.png)\n\n\n\n\n## 📄 License\nThis project is licensed under the **MIT License**. Feel free to use, modify, and distribute as needed.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvarthanv%2Fpub-sub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvarthanv%2Fpub-sub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvarthanv%2Fpub-sub/lists"}