{"id":22562142,"url":"https://github.com/opplieam/bb-dist-noti","last_synced_at":"2025-03-28T12:42:55.070Z","repository":{"id":264690960,"uuid":"893180682","full_name":"opplieam/bb-dist-noti","owner":"opplieam","description":"A stateful distributed notification system written in Go, leveraging Raft for resilient, strong consistency, and fully Kubernetes-ready","archived":false,"fork":false,"pushed_at":"2025-02-27T14:40:05.000Z","size":285,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-27T20:44:18.075Z","etag":null,"topics":["distributed-systems","go","golang","helm","kubernetes","kubernetes-operator","nats-jetstream","raft-consensus-algorithm","serf","server-sent-events"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opplieam.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-23T18:38:51.000Z","updated_at":"2025-02-27T14:40:08.000Z","dependencies_parsed_at":"2024-12-17T13:34:18.206Z","dependency_job_id":"2ccb8969-a6fe-45ae-bfe4-3843eb2e4eca","html_url":"https://github.com/opplieam/bb-dist-noti","commit_stats":null,"previous_names":["opplieam/bb-dist-noti"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opplieam%2Fbb-dist-noti","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opplieam%2Fbb-dist-noti/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opplieam%2Fbb-dist-noti/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opplieam%2Fbb-dist-noti/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opplieam","download_url":"https://codeload.github.com/opplieam/bb-dist-noti/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246034275,"owners_count":20712851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","go","golang","helm","kubernetes","kubernetes-operator","nats-jetstream","raft-consensus-algorithm","serf","server-sent-events"],"created_at":"2024-12-07T22:11:50.125Z","updated_at":"2025-03-28T12:42:55.065Z","avatar_url":"https://github.com/opplieam.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Distributed Notification System\n\n![img](https://github.com/opplieam/bb-dist-noti/blob/main/bb-dist-noti.drawio.png?raw=true)\n\nA high-performance, fault-tolerant distributed notification system designed to deliver real-time notifications \nwith unparalleled speed and reliability. This system receives events from pipelines like NATS (Pub/Sub) \nand seamlessly notifies connected users through Server-Sent Events (SSE). Built for scalability, \nit also caches notifications in memory with an efficient eviction policy, ensuring rapid access to recent messages \nwithout querying external services.\n\n## Key Features\n\n- **Decentralized Service Discovery**  \n  Leverages **Serf** with the gossip protocol for robust and decentralized node discovery.\n\n- **State Consistency with Raft**  \n  Implements **Raft consensus** to coordinate nodes, ensuring a single leader model and reliable replication across the cluster.\n\n- **Efficient Log Management**  \n  Uses Write-Ahead Logs (WAL) for both log and stable store, delivering superior performance.\n\n- **Pipeline Connection**  \n  Only the leader node establishes a secure subscription to the event stream pipeline (e.g., NATS JetStream).\n\n- **Security First**  \n  Supports **mutual TLS** for secure communication across all nodes.\n\n- **Multiplexed RPC Connections**  \n  Utilizes **gRPC** and **Raft RPC** for ease of use.\n\n- **In-Memory Caching with Eviction**  \n  Built-in append-only memory caching stores the most recent notifications (configurable, e.g., 500 or 1000 entries) \n  with a robust eviction policy.\n\n- **Kubernetes Ready**  \n  Comes with Helm charts for easy deployment on Kubernetes clusters.\n\n- **Dynamic Load Balancing with Ingress Nginx**  \n  Uses Ingress Nginx to load balance traffic only to follower nodes.\n  Includes a custom Kubernetes operator for node labeling to ensure proper traffic routing.\n\n## The Heart of the System: `agent.go`\n\nAt the core of this system lies the `agent` package. It orchestrates all critical components, including:\n\n1. **Client Connection Management**  \n   Maintains active connections with clients, handling reconnections gracefully during failures.\n\n2. **HTTP and SSE Handling**  \n   Manages HTTP requests and delivers real-time notifications to users via Server-Sent Events (SSE).\n\n3. **gRPC and Membership Services**  \n   Sets up a gRPC server for internal server-to-server communication. Leveraging Serf for node discovery and failure detection.\n\n4. **Consensus and Coordination**  \n   Ensures state consistency across nodes through the Raft algorithm, enabling leader election and state replication.\n\n5. **Fault-Tolerant Shutdown**  \n   Implements graceful shutdown procedures to protect the stability of the cluster and the WAL.\n\n---\n\n## Installation\n\n### Prerequisites\n- **Go 1.22+** \n- **Docker** and **Kubernetes**\n- **minikube** (or equivalent local cluster solution)\n- **Helm 3.x+**\n- **cfssl** (for mTLS certificate generation)\n- **protoc** (for protocol buffer compilation)\n\n### Test\n\n1. Generate certificate. Please check `tls/` for configuration\n    ```bash\n    make gencert    # Generates certificates in $HOME/.bb-noti/ \n    ```\n2. Run Test\n    ```bash\n    make test       # Includes end-to-end testing via agent_test.go \n    ```\n\n### Local deployment without cluster\n\n1. Start Dependencies\n   ```bash\n    # Terminal 1: Start NATS JetStream\n    make run-jet-stream\n\n    # Terminal 2: Start mock publisher\n    make run-mock-pub \n    ```\n   \n2. Launch Cluster Nodes\n    ```bash\n    # Terminal 3-5: Start three nodes\n    make run-node-1    # Leader node (HTTP port 8402)\n    make run-node-2    # Follower node (HTTP port 8502)\n    make run-node-3    # Follower node (HTTP port 8602)\n    ```\n\n3. Verify Node Status\n   ```bash\n    curl http://localhost:8502/readiness    # Check Node 2\n    curl http://localhost:8602/readiness    # Check Node 3\n    ``` \n4. Connect to Service\n   ```bash\n    curl http://localhost:8502/category     # Connect via Node 2\n    curl http://localhost:8602/category     # Connect via Node 3\n    ```\n5. Kill Node 1 or try to play around.\n\n### Kubernetes Deployment (Local Cluster)\n\n#### 0. Install Nginx Ingress Controller\nNginx Ingress Controller is required for HTTP protocol routing.\n\n```bash\nmake install-nginx\n```\n\n#### 1. Build and Load Docker Image\nBuild and load the application image into your local cluster. Choose the appropriate command based on your environment:\n\n```bash\nmake docker-build-dev-minikube    # For Minikube environment\nmake docker-build-dev-kind        # For Kind environment\n```\n\n#### 2. Deploy Dependencies\nEnsure that required dependencies are running before deploying the application:\n\n```bash\nmake run-jet-stream   # Starts NATS JetStream\nmake run-mock-pub     # Starts mock publisher for testing\n```\n\n#### 3. Deploy Application\nDeploy the main application using Helm. The configuration is defined in `deploy/bb-noti/values.yaml`:\n\n```bash\nmake helm\n```\n\n#### 4. Deploy Custom Kubernetes Operators\nFor dynamic node labeling, deploy the custom Kubernetes operator:\n[bb-dist-noti-operator](https://github.com/opplieam/bb-dist-noti-operator)\n\n#### 5. Expose Application\nFor Kind: Port-forward Nginx to expose the application.\n```bash\nkubectl port-forward svc/nginx-ingress-controller -n ingress-nginx 8080:80\n```\nFor Minikube: Use the Minikube tunnel.\n\n```bash\nminikube tunnel\n```\n\nAlso, update /etc/hosts with:\n```\n127.0.0.1 bb-noti.localhost\n```\nAccess the application at http://bb-noti.localhost.\n\n---\n## Configuration\n\n### Application Configuration\nRun `make help` to view available configuration flags\n\n### Key Configuration Files\n- `deploy/bb-noti/values.yaml`: Kubernetes deployment configuration\n- `tls/`: TLS certificate configuration\n\n---\n\n## Project structure\n\n```bash\n.\n├── cmd/\n│   ├── mockpub/          # Mock publisher implementation\n│   └── noti/             # Main application entry point\n├── deploy/               # Kubernetes and Helm configurations\n├── internal/\n│   └── agent/            # Core system implementation\n│       └── agent_test.go # End-to-end tests\n├── pkg/                  # Shared libraries\n├── proto/                # Protocol buffer definitions\n├── protogen/             # Generated protocol buffer code\n└── tls/                  # TLS configuration files\n```\n\n---\n## Note about Kubernetes Operator\n\nWhile there may be simpler and more efficient solutions than using a Kubernetes operator, I chose to explore \nthe operator model specifically to learn how to build one using Kubebuilder.\n\n---\n## Future Improvements\n\n- [x] Add golangci-lint\n- [ ] Upgrade gRPC to Opaque\n- [x] Implement more efficient data serialization\n- [ ] Add support for real user ID authentication\n- [x] Enhance start-join-addrs configuration\n- [x] Enhance Serf cluster joining mechanism\n- [x] Implement load balancing for follower node connections\n- [ ] Add comprehensive monitoring and metrics\n- [x] Improve cache eviction policies\n- [] Handle full outage. \n- [] Improve Makefile and handle go tools. ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopplieam%2Fbb-dist-noti","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopplieam%2Fbb-dist-noti","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopplieam%2Fbb-dist-noti/lists"}