Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/connor4312/hubsub

Last synced: 1 day ago
JSON representation

Host: GitHub
URL: https://github.com/connor4312/hubsub
Owner: connor4312
License: gpl-3.0
Created: 2015-07-18T07:48:28.000Z (over 9 years ago)
Default Branch: master
Last Pushed: 2015-07-29T03:24:04.000Z (over 9 years ago)
Last Synced: 2024-10-25T22:19:42.507Z (18 days ago)
Language: Java
Size: 203 KB
Stars: 1
Watchers: 3
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: readme.md
- License: LICENSE

Awesome Lists containing this project

README

# HubSub

HubSub is a new pubsub server, aimed at making distributed systems better.

## The Problem & Goals

At [Beam](https://github.com/WatchBeam), a big part of our service includes chat servers. Right now, we're using Redis for this. While Redis is a great system, it's flawed in that all pubsub happens _globally_. As scale increases, network traffic (gossip) increases exponentially, to the eventual point where adding more servers slows things down more than anything.

The goal of hubsub is for situations where pubsub is needed at a large scale but in relatively small increments, such as for chat rooms. Users in one chat room could care less what's happening in the other ninety nine thousands. HubSub aims to solve this by introducing "micro" clusters as rooms (think of them like keyspaces).

```
Client Asks
for Room
│
│
▼
┌────────────────┐
│ Already a room │
│ created? │───────────Yes┐
└────────────────┘ │
│ │
No │
▼ ▼
┌────────────────┐ ┌───────────────────┐ ┌────────────────┐
│Find the n least│ │ Tell the client │ │Client connects │
│ busy servers │ │ which servers to │────▶│like to a Redis │
└────────────────┘ │ connect to. │ │ server. │
│ └───────────────────┘ └────────────────┘
│ ▲
▼ │
┌────────────────┐ │
│Assign them the │ │
│ room │──────────────┘
└────────────────┘
```

A clients connects to some central server, which queries the pubsub cluster. If there room they ask for has already been created, great, just respond with the servers to connect to and move on. Otherwise, find the `n` least busy servers, confirm the room assignment with them, and send it back to the client. Afterwards, the client can connect to any of the HubSub servers just like a Redis server, using any standard Redis client library.

The goal is to get as close to linear scalability as possible. There will be some gossip, of course, to obtain consensus in the cluster. However, traffic related to pubsub is isolated within the room. The room is assigned to one or more HubSub instances (more would be better for failover!), and afterwards traffic in that room is communicated only between the instances. Scaling in this manner should be within `O(N)`.

```

Chatter about Chatter about
EventA only EventB only

│ │
┌ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐│
EventA │ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐
│ │ EventB ││
│ │ │ │
│ ┌─────────────┐ │ ┌─────────────┐ ││ ┌─────────────┐
│ HubSub │ ▼ │ │ HubSub │ ▼ │ HubSub │ │
│ │ Instance │◀━━━━▶│ Instance │◀─┼──▶│ Instance │
└─────────────┘ │ └─────────────┘ └─────────────┘ │
│ ▲ ▲ │ ▲
┌──┴────┐ │ ┌───┴───┐ ┌─────┴───┐ │
│ ┌───┴──┐┌───┴──┐ ┌───┴──┐┌───┴──┐ │ ┌──┴───┐ ┌──┴───┐
│Client││Client│ │ │Client││Client│ │Client│ │Client│ │
│ └──────┘└──────┘ └──────┘└──────┘ │ └──────┘ └──────┘
│ │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─

```

## Progress

Currently HubSub's functionality is on par with a (somewhat inefficient, functions at about 80% of the speed speed) standalone Redis server.

* [x] Implement Redis protocol parser and such
* [x] Implement local pubsub pools
* [x] Implement connection acceptors and lifecycle
* [x] Get Akka cluster going
* [x] Implement room assignment (Akka's really making this too easy...)
* [ ] Implement room maintenance, maybe a little REST API **(in progress)**
* [ ] Implement room failovers, reassignments due to shifting load
* [ ] Make fasterer
* More to come?

### License