{"id":19973255,"url":"https://github.com/zeromq/zeps","last_synced_at":"2025-06-24T01:06:32.163Z","repository":{"id":21395034,"uuid":"24712856","full_name":"zeromq/zeps","owner":"zeromq","description":"ZeroMQ Enterprise Publish-Subscribe (ZEPS) **DEPRECATED**","archived":false,"fork":false,"pushed_at":"2015-04-25T20:13:13.000Z","size":188,"stargazers_count":12,"open_issues_count":0,"forks_count":7,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-05-04T02:38:52.147Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zeromq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-10-02T09:02:57.000Z","updated_at":"2021-12-03T04:20:54.000Z","dependencies_parsed_at":"2022-07-30T04:47:55.631Z","dependency_job_id":null,"html_url":"https://github.com/zeromq/zeps","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zeromq/zeps","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeromq%2Fzeps","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeromq%2Fzeps/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeromq%2Fzeps/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeromq%2Fzeps/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zeromq","download_url":"https://codeload.github.com/zeromq/zeps/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeromq%2Fzeps/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261583821,"owners_count":23180671,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T03:10:48.263Z","updated_at":"2025-06-24T01:06:32.139Z","avatar_url":"https://github.com/zeromq.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ZeroMQ Enterprise Publish-Subscribe - ZEPS\n\n## DEPRECATED\n\nThis project is deprecated by zeromq/Malamute.\n\n## Overview\n\nZEPS is a broker-based publish-and-subscribe design that offers both high-speed low-latency pub-sub, and persistent journalled pub-sub, automatically switching clients from low gear to high gear according to their capabilities. ZEPS is compatible with multicast protocols like NORM and PGM, and has extensible subscription matching.\n\nNote that this is a concept wire-frame. There is no ZEPS broker at this stage. ZEPS is intended to be packaged into the ZeroMQ broker project (zbroker), which is an embeddable broker library that can be wrapped in arbitrary languages for deployment.\n\n## Problem Statement\n\nThe built-in ZeroMQ PUB-SUB pattern is tuned for high frequency, low-latency, transient data (the radio broadcast model). This model is extremely scalable, and requires no mediation (brokers). It also maps directly to multicast protocols like PGM and NORM.\n\nHowever, the PUB-SUB pattern has two scalability features that become anti-features, when scale, volume, and low-latency are not so important:\n\n* The pattern keeps no message state, so subscribers will lose arbitrary numbers of messages when they start-up. This is called the \"late joiner syndrome\".\n\n* The pattern deals with slow subscribers, or congested networks, by dropping messages. This can lead to gaps that are hard to detect, and very hard to recover from.\n\nIt is thus a frequent demand from users: \"how does one do reliable publish-subscribe with ZeroMQ?\"\n\nZEPS is an answer to this question.\n\n## Overall Design\n\nZEPS is a broker-based design. It connects a set of clients, which can both subscribe and publish, independently.\n\nThe message flow operates at two levels, real-time and historical:\n\n* During normal operation, messages are routed in real-time to subscribers, held in outgoing socket pipes, and delivered without any processing, disk I/O, or other work except subscription resolution and network I/O.\n* Concurrently, all messages are logged to a journal, which is a sequential disk file.\n* When a client subscribes, it can request messages from a certain past point (a time period, or a message sequence). The broker will then replay its journal, filtering messages according to the subscription pattern. When it has finished replaying historical data, the broker switches the client to receiving the real-time flow.\n\nIn the same way, a client that is congested will stop receiving real-time data and be switched to receiving historical data. These switches between real-time routing and historical replay are invisible to clients.\n\nZEPS makes some important assumptions:\n\n* *The fanout ratio is significant, at least ten times.* This means, publishers will never be a bottleneck, and a ZEPS broker can always accept new published messages, without confirmation. Publshing is thus a fully asynchronous operation with no possibilty of message loss.\n* *The ZEPS broker is utterly reliable, and will not crash.* In reality the hardware will fail at some stage. However the broker can be made simple, and reliable enough, to never fail. This allows it to act as a single point of failure, and take responsibility for message sequencing.\n\nWhen publishers are faced with network congestion, or a lost connection to the broker, they will either block, or execute other strategies. Messages will not be dropped unless the publisher explicitly does this.\n\n## Protocol Specification\n\nThe following ABNF grammar defines the ZEPS protocol:\n\n    ZEPS         = open-stream *use-stream [ close-stream ]\n\n    open-stream  = C:ATTACH ( S:ATTACH-OK / error )\n\n    use-stream   = C:SUBSCRIBE ( S:SUBSCRIBE-OK / error )\n                 / C:CREDIT\n                 / S:DELIVER\n                 / C:PUBLISH\n                 / C:PING S:PING-OK\n\n    close-stream = C:DETACH / S:DETACH\n\n    error        = S:INVALID\n\n    ;     Client opens stream\n    attach          = signature %d1 protocol version stream\n    signature       = %xAA %xA5             ; two octets\n    protocol        = string                ; Constant \"ZEPS\"\n    version         = number-2              ; Protocol version 1\n    stream          = string                ; Stream name\n\n    ;     Server grants the client access\n    attach_ok       = signature %d2\n\n    ;     Client subscribes to a set of topics\n    subscribe       = signature %d3 pattern latest\n    pattern         = string                ; Subscription pattern\n    latest          = number-8              ; Last known sequence\n\n    ;     Server confirms the subscription\n    subscribe_ok    = signature %d4\n\n    ;     Client sends credit to the server\n    credit          = signature %d5 credit\n    credit          = number-8              ; Credit, in bytes\n\n    ;     Client publishes a new message to the open stream\n    publish         = signature %d6 key body\n    key             = string                ; Message routing key\n    body            = chunk                 ; Message contents\n\n    ;     Server delivers a message to the client\n    deliver         = signature %d7 sequence key body\n    sequence        = number-8              ; Sequenced per stream\n    key             = string                ; Message routing key\n    body            = chunk                 ; Message contents\n\n    ;     Client or server sends a heartbeat\n    ping            = signature %d8\n\n    ;     Client or server answers a heartbeat\n    ping_ok         = signature %d9\n\n    ;     Client closes the peering\n    detach          = signature %d10\n\n    ;     Server handshake the close\n    detach_ok       = signature %d11\n\n    ;     Server tells client it sent an invalid message\n    invalid         = signature %d12 reason\n    reason          = string                ; Printable explanation\n\n    ; A chunk has 4-octet length + binary contents\n    chunk           = number-4 *OCTET\n\n    ; Strings are always length + text contents\n    string          = number-1 *VCHAR\n\n    ; Numbers are unsigned integers in network byte order\n    number-1        = 1OCTET\n    number-2        = 2OCTET\n    number-4        = 4OCTET\n    number-8        = 8OCTET\n\n## Detailed Design Notes\n\nThe broker uses \"streams\" to segregate and isolate traffic. Each stream corresponds to a single journal file (implementations may do further sharding to improve performance). Within a stream, each message has a \"routing key\" and a multipart content (currently implemented as a chunk). Messages are sequenced per stream.\n\nZEPS uses a credit-based flow control scheme to control buffer sizes on the broker. In this model, the client sends credit (in bytes) to the broker, which send messages to the client. These two flows happen asynchronously. When there is no credit, the broker stops sending to the client, and switches it to historical replay. This will happen, for instance, when a client is disconnected or restarted.\n\nWithin a stream, clients can subscribe to sub-streams of messages using pattern matching on a message \"routing key\". ZeroMQ PUB-SUB does prefix matching, using a trie structure. ZEPS does not yet specify a matching style. Some options are:\n\n* prefix matching using a trie\n* regular expression matching\n\nMore complex matching needs more complex data structures. This is an area we may make extensible, internally.\n\n## Multicast Output\n\nFor certain use cases, real-time data can be sent over multicast (using NORM or PGM). Historical data would never be multicast. If clients want both multicast real-time data, and historical data, they would create two connections and reconcile them.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzeromq%2Fzeps","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzeromq%2Fzeps","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzeromq%2Fzeps/lists"}