{"id":13413408,"url":"https://github.com/smancke/guble","last_synced_at":"2026-01-23T05:44:48.226Z","repository":{"id":57496865,"uuid":"46234623","full_name":"smancke/guble","owner":"smancke","description":"websocket based messaging server written in golang","archived":false,"fork":false,"pushed_at":"2017-10-31T19:15:41.000Z","size":9340,"stargazers_count":158,"open_issues_count":5,"forks_count":23,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-07-31T20:52:18.613Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/smancke.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-11-15T20:32:42.000Z","updated_at":"2024-06-13T00:02:09.000Z","dependencies_parsed_at":"2022-08-28T11:51:34.102Z","dependency_job_id":null,"html_url":"https://github.com/smancke/guble","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smancke%2Fguble","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smancke%2Fguble/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smancke%2Fguble/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smancke%2Fguble/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/smancke","download_url":"https://codeload.github.com/smancke/guble/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243635421,"owners_count":20322935,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:01:39.653Z","updated_at":"2026-01-23T05:44:43.197Z","avatar_url":"https://github.com/smancke.png","language":"Go","readme":"# Guble Messaging Server\n\nGuble is a simple user-facing messaging and data replication server written in Go.\n\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/f3b9a351201b416db4fe6df8faea363b)](https://www.codacy.com/app/cosminrentea/guble?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=smancke/guble\u0026utm_campaign=badger)\n[![Release](https://img.shields.io/github/release/smancke/guble.svg)](https://github.com/smancke/guble/releases/latest)\n[![Docker](https://img.shields.io/docker/pulls/smancke/guble.svg)](https://hub.docker.com/r/smancke/guble/)\n[![Build Status](https://api.travis-ci.org/smancke/guble.svg?branch=master)](https://travis-ci.org/smancke/guble)\n[![Go Report Card](https://goreportcard.com/badge/github.com/smancke/guble)](https://goreportcard.com/report/github.com/smancke/guble)\n[![codebeat badge](https://codebeat.co/badges/7f317892-0a7b-4e31-97f4-a530cf779889)](https://codebeat.co/projects/github-com-smancke-guble)\n[![Coverage Status](https://coveralls.io/repos/smancke/guble/badge.svg?branch=master\u0026service=github)](https://coveralls.io/github/smancke/guble?branch=master)\n[![GoDoc](https://godoc.org/github.com/smancke/guble?status.svg)](https://godoc.org/github.com/smancke/guble)\n[![Awesome-Go](https://camo.githubusercontent.com/13c4e50d88df7178ae1882a203ed57b641674f94/68747470733a2f2f63646e2e7261776769742e636f6d2f73696e647265736f726875732f617765736f6d652f643733303566333864323966656437386661383536353265336136336531353464643865383832392f6d656469612f62616467652e737667)](https://awesome-go.com)\n\n# Overview\nGuble is in an early state (release 0.4). \nIt is already working well and is very useful, but the protocol, API and storage formats \nmay still change (until reaching 0.7). \nIf you intend to use guble, please get in contact with us.\n\nThe goal of guble is to be a simple and fast message bus for user interaction and replication of data between multiple devices:\n* Very easy consumption of messages with web and mobile clients\n* Fast realtime messaging, as well as playback of messages from a persistent commit log\n* Reliable and scalable over multiple nodes\n* User-aware semantics to easily support messaging scenarios between people using multiple devices\n* Batteries included: usable as front-facing server, without the need of a proxy layer\n* Self-contained: no mandatory dependencies to other services\n\n## Working Features (0.4)\n\n* Publishing and subscription of messages to topics and subtopics\n* Persistent message store with transparent live and offline fetching\n* WebSocket and REST APIs for message publishing\n* Commandline client and Go client library\n* Firebase Cloud Messaging (FCM) adapter: delivery of messages as FCM push notifications\n* Docker images for server and client\n* Simple Authentication and Access-Management\n* Clean shutdown\n* Improved logging using [logrus](https://github.com/Sirupsen/logrus) and logstash formatter\n* Health-Check with Endpoint\n* Collection of Basic Metrics, with Endpoint\n* Added Postgresql as KV Backend\n* Load testing with 5000 messages per instance\n* Support for Apple Push Notification services (a new connector alongside Firebase)\n* Upgrade, cleanup, abstraction, documentation, and test coverage of the Firebase connector\n* GET list of subscribers / list of topics per subscriber (userID , deviceID) \n* Support for SMS-sending using Nexmo (a new connector alongside Firebase)\n\n## Throughput\nMeasured on an old notebook with i5-2520M, dual core and SSD. Message payload was 'Hello Word'.\nLoad driver and server were set up on the same machine, so 50% of the cpu was allocated to the load driver.\n\n* End-2-End: Delivery of ~35.000 persistent messages per second\n* Fetching: Receive of ~70.000 persistent messages per second\n\nDuring the tests, the memory consumption of the server was around ~25 MB.\n\n## Table of Contents\n\n- [Roadmap](#roadmap)\n  - [Roadmap Release 0.5](#roadmap-release-05)\n  - [Roadmap Release 0.6](#roadmap-release-06)\n  - [Roadmap Release 0.7](#roadmap-release-07)\n- [Guble Docker Image](#guble-docker-image)\n  - [Start the Guble Server](#start-the-guble-server)\n  - [Connecting with the Guble Client](#connecting-with-the-guble-client)\n- [Build and Run](#build-and-run)\n  - [Build and Start the Server](#build-and-start-the-server)\n    - [Configuration](#configuration)\n  - [Run All Tests](#run-all-tests)\n- [Clients](#clients)\n- [Protocol Reference](#protocol-reference)\n  - [REST API](#rest-api)\n    - [Headers](#headers)\n  - [WebSocket Protocol](#websocket-protocol)\n    - [Message Format](#message-format)\n    - [Client Commands](#client-commands)\n    - [Server Status Messages](#server-status-messages)\n  - [Topics](#topics)\n    - [Subtopics](#subtopics)\n\n# Roadmap\nThis is the current (and fast changing) roadmap and todo list:\n\n## Roadmap Release 0.5\n* Replication across multiple servers (in a Guble cluster)\n* Acknowledgement of message delivery for connectors\n* Storing the sequence-Id of topics in KV store, if we turn off persistence\n* Filtering of messages in guble server (e.g. sent by the REST client) according to URL parameters: UserID, DeviceID, Connector name\n* Updating README to show subscribe/unsubscribe/get/posting, health/metrics \n\n## Roadmap Release 0.6\n* Make notification messages optional by client configuration\n* Correct behaviour of receive command with `maxCount` on subtopics\n* Cancel of fetch in the message store and multiple concurrent fetch commands for the same topic\n* Configuration of different persistence strategies for topics\n* Delivery semantics: user must read on one device / deliver only to one device / notify if not connected, etc.\n* User-specific persistent subscriptions across all clients of the user\n* Client: (re-)setup of subscriptions after client reconnect\n* Message size limit configurable by the client with fetching by URL\n\n## Roadmap Release 0.7\n* HTTPS support in the service\n* Minimal example: chat application\n* Stable JavaScript client: https://github.com/smancke/guble-js\n* (TBD) Improved authentication and access-management\n* (TBD) Add Consul as KV Backend\n* (TBD) Index-based search of messages using [GoLucene](https://github.com/balzaczyy/golucene)\n\n# Guble Docker Image\nWe are providing Docker images of the server and client for your convenience.\n\n## Start the Guble Server\nThere is an automated Docker build for the master at the Docker Hub.\nTo start the server with Docker simply type:\n```\ndocker run -p 8080:8080 smancke/guble\n```\n\nTo see available configuration options:\n```\ndocker run smancke/guble --help\n```\n\nAll options can be supplied on the commandline or by a corresponding environment variable with the prefix `GUBLE_`.\nSo to let guble be more verbose, you can either use:\n```\ndocker run smancke/guble --log=info\n```\nor\n```\ndocker run -e GUBLE_LOG=info smancke/guble\n```\n\nThe Docker image has a volume mount point at `/var/lib/guble`, so if you want to bind-mount the persistent storage from your host you should use:\n```\ndocker run -p 8080:8080 -v /host/storage/path:/var/lib/guble smancke/guble\n```\n\n## Connecting with the Guble Client\nThe Docker image includes the guble commandline client `guble-cli`.\nYou can execute it within a running guble container and connect to the server:\n```\ndocker run -d --name guble smancke/guble\ndocker exec -it guble /usr/local/bin/guble-cli\n```\nVisit the [`guble-cli` documentation](https://github.com/smancke/guble/tree/master/guble-cli) for more details.\n\n# Build and Run\nSince Go makes it very easy to build from source, you can compile guble using a single command.\nA prerequisite is having an installed Go environment and an empty directory:\n```\nsudo apt-get install golang\nmkdir guble \u0026\u0026 cd guble\nexport GOPATH=`pwd`\n```\n\n## Build and Start the Server\nBuild and start guble with the following commands (assuming that directory `/var/lib/guble` is already created with read-write rights for the current user):\n```\ngo get github.com/smancke/guble\nbin/guble --log=info\n```\n\n### Configuration\n\n|CLI Option|Env Variable|Values|Default|Description|\n|--- |--- |--- |--- |--- |\n|`--env`|GUBLE_ENV|development \u0026#124; integration \u0026#124; preproduction \u0026#124; production|development|Name of the environment on which the application is running. Used mainly for logging|\n|`--health-endpoint`|GUBLE_HEALTH_ENDPOINT|resource/path/to/healthendpoint|/admin/healthcheck|The health endpoint to be used by the HTTP server.Can be disabled by setting the value to \"\"|\n|`--http`|GUBLE_HTTP_LISTEN|format: [host]:port||The address to for the HTTP server to listen on|\n|`--kvs`|GUBLE_KVS|memory \u0026#124; file \u0026#124; postgres|file|The storage backend for the key-value store to use|\n|`--log`|GUBLE_LOG|panic \u0026#124; fatal \u0026#124; error \u0026#124; warn \u0026#124; info \u0026#124; debug|error|The log level in which the process logs|\n|`--metrics-endpoint`|GUBLE_METRICS_ENDPOINT|resource/path/to/metricsendpoint|/admin/metrics|The metrics endpoint to be used by the HTTP server.Can be disabled by setting the value to \"\"|\n|`--ms`|GUBLE_MS|memory \u0026#124; file|file|The message storage backend|\n|`--profile`|GUBLE_PROFILE|cpu \u0026#124; mem \u0026#124; block||The profiler to be used|\n|`--storage-path`|GUBLE_STORAGE_PATH|path/to/storage|/var/lib/guble|The path for storing messages and key-value data like subscriptions if defined.The path must exists!|\n\n\n#### APNS\n\n|CLI Option|Env Variable|Values|Default|Description|\n|--- |--- |--- |--- |--- |\n|`--apns`|GUBLE_APNS|true \u0026#124; false|false|Enable the APNS module in general as well as the connector to the development endpoint|\n|`--apns-production`|GUBLE_APNS_PRODUCTION|true \u0026#124; false|false|Enables the connector to the apns production endpoint, requires the apns option to be set|\n|`--apns-cert-file`|GUBLE_APNS_CERT_FILE|path/to/cert/file||The APNS certificate file name, use this as an alternative to the certificate bytes option|\n|`--apns-cert-bytes`|GUBLE_APNS_CERT_BYTES|cert-bytes-as-hex-string||The APNS certificate bytes, use this as an alternative to the certificate file option|\n|`--apns-cert-password`|GUBLE_APNS_CERT_PASSWORD|password||The APNS certificate password|\n|`--apns-app-topic`|GUBLE_APNS_APP_TOPIC|topic||The APNS topic (as used by the mobile application)|\n|`--apns-prefix`|GUBLE_APNS_PREFIX|prefix|/apns/|The APNS prefix / endpoint|\n|`--apns-workers`|GUBLE_APNS_WORKERS|number of workers|Number of CPUs|The number of workers handling traffic with APNS (default: number of CPUs)|\n\n\n#### SMS\n\n|CLI Option|Env Variable|Values|Default |Description|\n|--- |--- |--- |--- |--- |\n|`sms`|GUBLE_SMS|true \u0026#124; false|false |Enable the SMS gateway|\n|`sms_api_key`|GUBLE_SMS_API_KEY|api key||The Nexmo API Key for Sending sms|\n|`sms_api_secret`|GUBLE_SMS_API_SECRET|api secret||The Nexmo API Secret for Sending sms|\n|`sms_topic`|GUBLE_SMS_TOPIC|topic|/sms|The topic for sms route|\n|`sms_workers`|GUBLE_SMS_WORKERS|number of workers|Number of CPUs|The number of workers handling traffic with Nexmo sms endpoint|\n\n#### FCM\n\n|CLI Option|Env Variable|Values|Default|Description|\n|--- |--- |--- |--- |--- |\n|`--fcm|GUBLE_FCM`|true \u0026#124; false|false|Enable the Google Firebase Cloud Messaging connector|\n|`--fcm-api-key`|GUBLE_FCM_API_KEY|api key||The Google API Key for Google Firebase Cloud Messaging|\n|`--fcm-workers`|GUBLE_FCM_WORKERS|number of workers|Number of CPUs|The number of workers handling traffic with Firebase Cloud Messaging|\n|`--fcm-endpoint`|GUBLE_FCM_ENDPOINT|format: url-schema|https://fcm.googleapis.com/fcm/send|The Google Firebase Cloud Messaging endpoint|\n|`--fcm-prefix`|GUBLE_FCM_PREFIX|prefix|/fcm/|The FCM prefix / endpoint|\n\n#### Postgres\n\n|CLI Option|Env Variable|Values|Default|Description|\n|--- |--- |--- |--- |--- |\n|`--pg-host`|GUBLE_PG_HOST|hostname|localhost|The PostgreSQL hostname|\n|`--pg-port`|GUBLE_PG_PORT|port|5432|The PostgreSQL port|\n|`--pg-user`|GUBLE_PG_USER|user|guble|The PostgreSQL user|\n|`--pg-password`|GUBLE_PG_PASSWORD|password|guble|The PostgreSQL password|\n|`--pg-dbname`|GUBLE_PG_DBNAME|database|guble|The PostgreSQL database name|\n\n\n## Run All Tests\n```\ngo get -t github.com/smancke/guble/...\ngo test github.com/smancke/guble/...\n```\n\n# Clients\nThe following clients are available:\n* __Commandline Client__: https://github.com/smancke/guble/tree/master/guble-cli\n* __Go client library__: https://github.com/smancke/guble/tree/master/client\n* __JavaScript library__: (in early stage) https://github.com/smancke/guble-js\n\n# Protocol Reference\n\n## REST API\nCurrently there is a minimalistic REST API, just for publishing messages.\n\n```\nPOST /api/message/\u003ctopic\u003e\n```\nURL parameters:\n* __userId__: The PublisherUserId\n* __messageId__: The PublisherMessageId\n\n### Headers\nYou can set fields in the header JSON of the message by providing the corresponding HTTP headers with the prefix `X-Guble-`.\n\nCurl example with the resulting message:\n```\ncurl -X POST -H \"x-Guble-Key: Value\" --data Hello 'http://127.0.0.1:8080/api/message/foo?userId=marvin\u0026messageId=42'\n```\nResults in:\n```\n16,/foo,marvin,VoAdxGO3DBEn8vv8,42,1451236804\n{\"Key\":\"Value\"}\nHello\n```\n\n## WebSocket Protocol\nThe communication with the guble server is done by ordinary WebSockets, using a binary encoding.\n\n### Message Format\nAll payload messages sent from the server to the client are using the following format:\n```\n\u003cpath:string\u003e,\u003csequenceId:int64\u003e,\u003cpublisherUserId:string\u003e,\u003cpublisherApplicationId:string\u003e,\u003cpublisherMessageId:string\u003e,\u003cmessagePublishingTime:unix-timestamp\u003e\\n\n[\u003capplication headers json\u003e]\\n\n\u003cbody\u003e\n\nexample 1:\n/foo/bar,42,user01,phone1,id123,1420110000\n{\"Content-Type\": \"text/plain\", \"Correlation-Id\": \"7sdks723ksgqn\"}\nHello World\n\nexample 2:\n/foo/bar,42,user01,54sdcj8sd7,id123,1420110000\n\nanyByteData\n```\n\n* All text formats are assumed to be UTF-8 encoded.\n* Message `sequenceId`s are `int64`, and distinct within a topic.\n  The message `sequenceId`s are strictly monotonically increasing depending on the message age, but there is no guarantee for the right order while transmitting.\n\n### Client Commands\nThe client can send the following commands.\n\n#### Send\nPublish a message to a topic:\n```\n\u003e \u003cpath\u003e [\u003cpublisherMessageId\u003e]\\n\n[\u003cheader\u003e\\n]..\n\\n\n\u003cbody\u003e\n\nexample:\n\u003e /foo\n\nHello World\n```\n\n#### Subscribe/Receive\nReceive messages from a path (e.g. a topic or subtopic).\nThis command can be used to subscribe for incoming messages on a topic,\nas well as for replaying the message history.\n```\n+ \u003cpath\u003e [\u003cstartId\u003e[,\u003cmaxCount\u003e]]\n```\n* `path`: the topic to receive the messages from\n* `startId`: the message id to start the replay\n** If no `startId` is given, only future messages will be received (simple subscribe).\n** If the `startId` is negative, it is interpreted as relative count of last messages in the history.\n* `maxCount`: the maximum number of messages to replay\n\n__Note__: Currently, the fetching of stored messages does not recognize subtopics.\n\nExamples:\n```\n+ /foo         # Subscribe to all future messages matching /foo\n+ /foo/bar     # Subscribe to all future messages matching /foo/bar\n\n+ /foo 0       # Receive all message from the topic and subscribe for further incoming messages.\n\n+ /foo 42      # Receive all message with message ids \u003e= 42\n               # from the topic and subscribe for further incoming messages.\n\n+ /foo 0 20    # Receive the first (oldest) 20 messages within the topic and stop.\n               # (If the topic has less messages, it will stop after receiving all existing ones.)\n\n+ /foo -20     # Receive the last (newest) 20 messages from the topic and then\n               # subscribe for further incoming messages.\n\n+ /foo -20 20  # Receive the last (newest) 20 messages within the topic and stop.\n               # (If the topic has less messages, it will stop after receiving all existing ones.)\n```\n\n#### Unsubscribe/Cancel\nCancel further receiving of messages from a path (e.g. a topic or subtopic).\n\n```\n- \u003cpath\u003e\n\nexample:\n- /foo\n- /foo/bar\n```\n\n### Server Status Messages\nThe server sends status messages to the client. All positive status messages start with `\u003e`.\nStatus messages reporting an error start with `!`. Status messages are in the following format.\n\n```\n'#'\u003cmsgType\u003e \u003cExplanation text\u003e\\n\n\u003cjson data\u003e\n```\n\n#### Connection Message\n```\n#ok-connected You are connected to the server.\\n\n{\"ApplicationId\": \"the app id\", \"UserId\": \"the user id\", \"Time\": \"the server time as unix timestamp \"}\n```\n\nExample:\n```\n#connected You are connected to the server.\n{\"ApplicationId\": \"phone1\", \"UserId\": \"user01\", \"Time\": \"1420110000\"}\n```\n\n#### Send Success Notification\nThis notification confirms, that the messaging system has successfully received the message and now starts transmitting it to the subscribers:\n\n```\n#send \u003cpublisherMessageId\u003e\n{\"sequenceId\": \"sequence id\", \"path\": \"/foo\", \"publisherMessageId\": \"publishers message id\", \"messagePublishingTime\": \"unix-timestamp\"}\n```\n\n#### Receive Success Notification\nDepending on the type of `+` (receive) command, up to three different notification messages will be sent back.\nBe aware, that a server may send more receive notifications that you would have expected in first place, e.g. when:\n* Additional messages are stored, while the first fetching is in progress\n* The server decides to meanwhile stop the online subscription and change to fetching,\n  because your client is too slow to read all incoming messages.\n\n1. When the fetch operation starts:\n\n    ```\n    #fetch-start \u003cpath\u003e \u003ccount\u003e\n    ```\n    * `path`: the topic path\n    * `count`: the number of messages that will be returned\n\n2. When the fetch operation is done:\n\n    ```\n    #fetch-done \u003cpath\u003e\n    ```\n    * `path`: the topic path\n3. When the subscription to new messages was taken:\n\n    ```\n    #subscribed-to \u003cpath\u003e\n    ```\n    * `path`: the topic path\n\n#### Unsubscribe Success Notification\nAn unsubscribe/cancel operation is confirmed by the following notification:\n```\n#canceled \u003cpath\u003e\n```\n\n#### Send Error Notification\nThis message indicates, that the message could not be delivered.\n```\n!error-send \u003cpublisherMessageId\u003e \u003cerror text\u003e\n{\"sequenceId\": \"sequence id\", \"path\": \"/foo\", \"publisherMessageId\": \"publishers message id\", \"messagePublishingTime\": \"unix-timestamp\"}\n```\n\n#### Bad Request\nThis notification has the same meaning as the http 400 Bad Request.\n```\n!error-bad-request unknown command 'sdcsd'\n```\n\n#### Internal Server Error\nThis notification has the same meaning as the http 500 Internal Server Error.\n```\n!error-server-internal this computing node has problems\n```\n\n## Topics\n\nMessages can be hierarchically routed by topics, so they are represented by a path, separated by `/`.\nThe server takes care, that a message only gets delivered once, even if it is matched by multiple\nsubscription paths.\n\n### Subtopics\nThe path delimiter gives the semantic of subtopics. \nWith this, a subscription to a parent topic (e.g. `/foo`)\nalso results in receiving all messages of the subtopics (e.g. `/foo/bar`).\n","funding_links":[],"categories":["Messaging","消息","消息系统","机器学习","Relational Databases","消息传递","\u003cspan id=\"消息-messaging\"\u003e消息 Messaging\u003c/span\u003e"],"sub_categories":["Advanced Console UIs","Search and Analytic Databases","检索及分析资料库","SQL 查询语句构建库","高級控制台界面","交流","\u003cspan id=\"高级控制台用户界面-advanced-console-uis\"\u003e高级控制台用户界面 Advanced Console UIs\u003c/span\u003e","高级控制台界面"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmancke%2Fguble","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmancke%2Fguble","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmancke%2Fguble/lists"}