{"id":15663072,"url":"https://github.com/itzmeanjan/pub0sub","last_synced_at":"2025-05-06T02:49:21.423Z","repository":{"id":49598653,"uuid":"364118386","full_name":"itzmeanjan/pub0sub","owner":"itzmeanjan","description":"Fast, Lightweight Pub/Sub over TCP, QUIC - powered by Async I/O","archived":false,"fork":false,"pushed_at":"2021-06-15T08:50:11.000Z","size":1325,"stargazers_count":20,"open_issues_count":1,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-06T02:49:15.308Z","etag":null,"topics":["golang","pubsub","pubsub-publisher","pubsub-subscriber","quic","tcp","tcp-client","tcp-server","topics"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/itzmeanjan/pub0sub","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/itzmeanjan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-04T02:29:54.000Z","updated_at":"2025-01-31T16:53:42.000Z","dependencies_parsed_at":"2022-09-19T01:51:17.839Z","dependency_job_id":null,"html_url":"https://github.com/itzmeanjan/pub0sub","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itzmeanjan%2Fpub0sub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itzmeanjan%2Fpub0sub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itzmeanjan%2Fpub0sub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itzmeanjan%2Fpub0sub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/itzmeanjan","download_url":"https://codeload.github.com/itzmeanjan/pub0sub/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252612452,"owners_count":21776253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","pubsub","pubsub-publisher","pubsub-subscriber","quic","tcp","tcp-client","tcp-server","topics"],"created_at":"2024-10-03T13:35:24.388Z","updated_at":"2025-05-06T02:49:21.408Z","avatar_url":"https://github.com/itzmeanjan.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pub0sub\nFast, Lightweight Pub/Sub over TCP, QUIC - **powered by Async I/O**\n\n## Motivation\n\nFew days back I worked on an _Embeddable, Fast, Light-weight Pub/Sub System for Go Projects_, called `pubsub`, which is built only using native Go functionalities \u0026 as the title suggests, you can embed that system in your application for doing in-app message passing using any of following patterns\n\n- Single Publisher Single Subscriber\n- Single Publisher Multiple Subscriber\n- Multiple Publisher Single Subscriber\n- Multiple Publisher Multiple Subscriber\n\nThat enables making multiple go routines talk to each other over **topics**. Also there's no involvement of network I/O, so all operations are quite low-latency.\n\n\u003e If you're interested in taking a look at [`pubsub`](https://github.com/itzmeanjan/pubsub)\n\nNow I'm interested in extending aforementioned `pubsub` architecture to a more generic form so that clients i.e. _{publishers, subscribers}_ can talk to **Pub/Sub Hub** over network i.e. TCP, QUIC.\n\nWhat it gives us is, ability to publish messages to topics over network, where **Pub/Sub Hub** might sit somewhere else; subscribe to topics of interest \u0026 keep receiving messages as soon as they're published, over network.\n\n\u003e QUIC to be preferred choice of network I/O, due to benefits it brings on table.\n\n\u003e ⭐️ Primary implementation is on top of TCP.\n\n## Architecture\n\n![architecture](./sc/architecture.jpg)\n\n![async-io-model](./sc/async-io.jpg)\n\n![wire-format](./sc/wire-format.jpg)\n\n## Install\n\nAdd `pub0sub` into your project _( **GOMOD** enabled )_\n\n```bash\ngo get -u github.com/itzmeanjan/pub0sub/...\n```\n\n## Usage\n\n`pub0sub` has three components\n\n- [Hub](#hub)\n- [Publisher](#publisher)\n- [Subscriber](#subscriber)\n\n### Hub\n\nYou probably would like to use `0hub` for this purpose.\n\n---\nDefault Port | Default Interface\n--- | ---\n13000 | 127.0.0.1\n---\n\nBuild using \n\n```bash\nmake build_hub\n```\n\nRun using\n\n```bash\n./0hub -help\n./0hub # run\n```\n\n\u003e Single step build-and-run with `make hub`\n\n\u003e If interested, you can check `0hub` implementation [here](./cli/hub/0hub.go)\n\n---\n\n**Dockerised `0hub`**\n\nYou may prefer Docker-ised setup for running Hub\n\n```bash\nmake docker_hub # assuming docker daemon running\n```\n\nList images\n\n```bash\ndocker images # find `0hub` in list\n```\n\nRemove `\u003cnone\u003e` tagged images [ **optional** ]\n\n```bash\ndocker images -a | grep none | awk '{ print $3}' | xargs docker rmi --force\n```\n\nRun `0hub` as container. *You may want to change ENV variables, check defaults in `0hub.env`*\n\n```bash\nmake run_hub\ndocker ps # must list `hub`\n```\n\n\u003e Or, if you want `0hub` to be reachable on `localhost:13000`\n\n```bash\ndocker run --name hub -p 13000:13000 --env-file 0hub.env -d 0hub # reachable at localhost:13000\n```\n\nCheck container log\n\n```bash\ndocker logs hub\n```\n\nIf you want to inspect container\n\n```bash\ndocker inspect hub\n```\n\nStop when done\n\n```bash\nmake stop_hub\nmake remove_hub\n```\n\n### Publisher\n\nYou can interact with Hub, using minimalistic publisher CLI client `0pub`. Implementation can be found [here](./cli/publisher/0pub.go)\n\nBuild using\n\n```bash\nmake build_pub\n```\n\nRun using\n\n```bash\n./0pub -help\n./0pub # run\n```\n\n\u003e Single step build-and-run with `make pub`, using defaults\n\nYou're probably interested in publishing messages programmatically. \n\n- Let's first create a publisher, which will establish TCP connection with Hub\n\n```go\nctx, cancel := context.WithCancel(context.Background())\n\npub, err := publisher.New(ctx, \"tcp\", \"127.0.0.1:13000\")\nif err != nil {\n\treturn\n}\n```\n\n- Construct message you want to publish\n\n```go\ndata := []byte(\"hello\")\ntopics := []string{\"topic_1\", \"topic_2\"}\n\nmsg := ops.Msg{Topics: topics, Data: data}\n```\n\n- Publish message\n\n```go\nn, err := pub.Publish(\u0026msg)\nif err != nil {\n\treturn\n}\n\nlog.Printf(\"Approximate %d receiver(s)\\n\", n)\n```\n\n- When done using publisher instance, cancel context, which will tear down network connection gracefully\n\n```go\ncancel()\n\u003c-time.After(time.Second) // just wait a second\n```\n\n- You can always check whether network connection with Hub in unaffected or not\n\n```go\nif pub.Connected() {\n    log.Println(\"Yes, still connected\")\n}\n```\n\n### Subscriber\n\nYou're encouraged to first test out `0sub` - minimalistic CLI subscriber client for interacting with Hub.\n\nBuild using\n\n```bash\nmake build_sub\n```\n\nRun using\n\n```bash\n./0sub -help\n./0sub # run\n```\n\nTake a look at implementation [here](./cli/subscriber/0sub.go)\n\n\u003e Single step build-and-run using `make sub`, runs with default config\n\nBut probably you want to programmatically interact with Hub for subscribing to topics of interest \u0026 receive messages as soon as they're published\n\n\n- Start by creating subscriber instance, which will establish a long-lived TCP connection with Hub \u0026 subscribe initially to topics provided with\n\n```go\nctx, cancel := context.WithCancel(context.Background())\n\ncapacity := 128 // pending message inbox capacity\ntopics := []string{\"topic_1\", \"topic_2\"}\n\nsub, err := subscriber.New(ctx, \"tcp\", \"127.0.0.1:13000\", capacity, topics...)\nif err != nil {\n\treturn\n}\n```\n\n- As soon as new message is available for consumption **( queued in inbox )**, subscriber process to be notified over go channel. It's better to listen \u0026 pull message from inbox\n\n```go\nfor {\n    select {\n        case \u003c-ctx.Done():\n            return\n        \n        // watch to get notified\n        case \u003c-sub.Watch():\n            if msg := sub.Next(); msg != nil {\n                // consume message\n            }\n\t}\n}\n```\n\n- You can add more on-the-fly topic subscriptions\n\n```go\nn, err := sub.AddSubscription(\"topic_3\")\nif err != nil {\n    return\n}\n\nlog.Printf(\"Subscribed to %d topic(s)\\n\", n)\n```\n\n- You might need to unsubscribe from topics\n\n```go\nn, err := sub.Unsubscribe(\"topic_1\")\nif err != nil {\n    return\n}\n\nlog.Printf(\"Unsubscribed from %d topic(s)\\n\", n)\n```\n\n- You can unsubscribe from all topics\n\n```go\nn, err := sub.UnsubscribeAll()\nif err != nil {\n    return\n}\n\nlog.Printf(\"Unsubscribed from %d topic(s)\\n\", n)\n```\n\n- Any time you can check existence of unconsumed bufferred messages in inbox\n\n```go\nif sub.Queued() {\n    log.Println(\"We've messages to consume\")\n\n    if msg := sub.Next(); msg != nil {\n        // act on message\n    }\n}\n```\n\n- Or you may need to check whether client is still connected to Hub over TCP\n\n```go\nif sub.Connected() {\n    log.Println(\"Yes, still connected\")\n}\n```\n\n- When done using, it's better to gracefully tear down TCP connection\n\n```go\nif err := sub.Disconnect(); err != nil {\n    // may happen when already teared down\n    log.Println(err.Error())\n}\n```\n\n## Test\n\nFor running all test cases\n\n```bash\ngo test -v -race -covermode=atomic ./... # excludes `stress` testing, check 👇\n```\n\nFor running **stress** testing with **1k, 2k, 4k, 8k** simultaneous TCP connections\n\n```bash\ngo test -v -tags stress -run=8k # also try 1k/ 2k/ 4k\n```\n\n![stress-testing](./sc/stress-testing.png)\n\n\u003e Make sure your system is able to open **\u003e 8k** file handles at a time or you'll get `too many open files`\n\n## Benchmarking\n\nPublisher's message publish flow\n\n```bash\ngo test -run=XXX -tags stress -bench Publisher\n```\n\n![publisher](sc/benchmark-publiser.png)\n\nSubscriber's message consumption flow\n\n```bash\ngo test -run=XXX -tags stress -bench Subscriber\n```\n\n![subscriber](sc/benchmark-subscriber.png)\n\nSubscriber's topic subscription flow\n\n```bash\ngo test -run=XXX -tags stress -bench TopicSubscription\n```\n\n![topic_subscription](sc/benchmark-topic-subscription.png)\n\n## Simulator\n\nI wrote one simulator-visualiser tool --- which can be used for testing `0hub` with flexible configuration. You can ask it to use **N** -concurrent clients, all sending/ receiving messages to/ from **M** -topics. Finally it generates some simple visuals depicting performance.\n\nCheck [here](https://github.com/itzmeanjan/gClient)\n\n**More coming soon**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitzmeanjan%2Fpub0sub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fitzmeanjan%2Fpub0sub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitzmeanjan%2Fpub0sub/lists"}