{"id":15554185,"url":"https://github.com/vyzo/gerbil-simsub","last_synced_at":"2025-07-04T02:34:24.194Z","repository":{"id":50367722,"uuid":"120441979","full_name":"vyzo/gerbil-simsub","owner":"vyzo","description":"A gossipsub protocol simulator","archived":false,"fork":false,"pushed_at":"2022-08-04T03:29:02.000Z","size":132,"stargazers_count":39,"open_issues_count":3,"forks_count":7,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-28T16:32:35.907Z","etag":null,"topics":["gerbil","ipfs","pubsub","scheme"],"latest_commit_sha":null,"homepage":"","language":"Scheme","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vyzo.png","metadata":{"files":{"readme":"README-literate-gossipsub.org","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-06T10:58:12.000Z","updated_at":"2025-04-13T10:50:42.000Z","dependencies_parsed_at":"2022-08-26T01:11:50.617Z","dependency_job_id":null,"html_url":"https://github.com/vyzo/gerbil-simsub","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vyzo/gerbil-simsub","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyzo%2Fgerbil-simsub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyzo%2Fgerbil-simsub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyzo%2Fgerbil-simsub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyzo%2Fgerbil-simsub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vyzo","download_url":"https://codeload.github.com/vyzo/gerbil-simsub/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyzo%2Fgerbil-simsub/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263434361,"owners_count":23465949,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gerbil","ipfs","pubsub","scheme"],"created_at":"2024-10-02T14:51:34.956Z","updated_at":"2025-07-04T02:34:24.165Z","avatar_url":"https://github.com/vyzo.png","language":"Scheme","funding_links":[],"categories":[],"sub_categories":[],"readme":"* A PubSub Protocol and its Simulator\n\nThis is a pubsub protocol, dubbed ~gossipsub~, aka ~meshsub/1.0.0~.\nThe basic idea is to start from a connected network with an arbitrary\ndegree and reduce it to a mesh with a specific degree. Messages are\nforwarded through the reduced mesh. The mesh is augmented by /gossip/\nabout forwarded messages, which is regularly forwarded to a random subset\nof known peers with a heartbeat.\n\n* Reference Implementation\n\nThe reference code is written in [[https://github.com/vyzo/gerbil][Gerbil]], and you can install it through the\nGerbil package manager:\n\n#+BEGIN_EXAMPLE\n$ gxpkg install github.com/vyzo/gerbil-simsub\n#+END_EXAMPLE\n\n* The gossipsub Protocol\n\nThe gossipsub router is implemented as an /actor/ in [[simsub/gossip.ss]].\n\n** Protocol\nThe protocol is defined in two layers: basic pubsub, and gossipsub:\n\n#+BEGIN_SRC gerbil\n(defproto pubsub\n  event:\n  (connect)\n  (publish id data))\n\n(defproto gossipsub\n  extend: pubsub\n  event:\n  (ihave ids)\n  (iwant ids)\n  (graft)\n  (prune))\n#+END_SRC\n\nThe baseline ~pubsub~ protocol specifies 2 messages:\n- ~CONNECT~ which establishes a symmetric connection to the peer.\n- ~PUBLISH~ which forwards a message to the peer.\n\nThe baseline protocol provides just the primitives to connect the\noverlay with ~CONNECT~ and forward messages to peers with ~PUBLISH~.\nA basic implementation of ~pubsub~, aka ~floodsub~, utilizes flooding:\nupon receiving a published message, the peer forwards it to all known\npeers except the origin.\n\nThe ~gossipsub~ protocol layers on top and augments the baseline protocol with 4\n control messages:\n- ~IHAVE~ is the gossip message, which specifies recent messages ids in the peer's\n  history.\n- ~IWANT~ is used to ask for specific messages by id.\n- ~GRAFT~ is used to notify a peer that a mesh link has been grafted.\n- ~PRUNE~ is used to notify a peer that a mesh link has been pruned or to reject\n  a ~GRAFT~.\n\nIn contrast to ~floodsub~, ~gossipsub~ reduces the publish\namplification by routing only through mesh peers. The ~gossipsub~\nrouter maintains the mesh by using ~GRAFT~ and ~PRUNE~ messages, which\neffect symmetric links. The mesh is augmented by gossip messages,\n~IHAVE~ and ~IWANT~, which allows the overlay to overcome connectivity\npathologies and jump hops opportunistically. More advanced ~gossipsub~\nrouters can utilize gossip propagation to optimize the overlay for certain\nconfigurations -- e.g. an epidemic broadcast tree router such as in\n[[https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/episub.md][Episub]]\ncan ~GRAFT~ on fresh gossip and ~PRUNE~ on late messages in order to\noptimize for single source transmission.\n\n** Overlay Parameters\n\n#+BEGIN_SRC gerbil\n(def N 6)                            ; target mesh degree\n(def N-low 4)                        ; low water mark for mesh degree\n(def N-high 12)                      ; high water mark for mesh degree\n\n(def history-gossip 3)               ; length of gossip history\n(def history-length 120)             ; length of total message history\n#+END_SRC\n\n** Actor State\n\n#+BEGIN_SRC gerbil\n  (def messages (make-hash-table-eqv))  ; messages seen: message-id -\u003e data\n  (def window [])                       ; messages in current window: [message-id ...]\n  (def history [])                      ; message history: [window ...]\n  (def peers [])                        ; connected peers\n  (def D [])                            ; direct peers in the mesh\n  (def heartbeat                        ; next heartbeat time\n    (make-timeout (1+ (random-real))))\n#+END_SRC\n\n** Reaction Loop\n\nThe reaction loop implements the main logic loop of the actor. The actor\nreceives new messages and reacts accordingly, and dispatches the\nheartbeat procedure when the timeout is reached.\n\n#+BEGIN_SRC gerbil\n  (def (loop)\n    (\u003c- ((!pubsub.connect)\n         (unless (memq @source peers)\n           (set! peers (cons @source peers))))\n\n        ((!pubsub.publish id msg)\n         (unless (hash-get messages id) ; seen?\n           (hash-put! messages id msg)\n           (set! window (cons id window))\n           ;; deliver\n           (receive id msg)\n           ;; and forward\n           (for (peer (remq @source D))\n             (send! (!!pubsub.publish peer id msg)))))\n\n        ((!gossipsub.ihave ids)\n         (let (iwant (filter (lambda (id) (not (hash-get messages id)))\n                             ids))\n           (unless (null? iwant)\n             (send! (!!gossipsub.iwant @source iwant)))))\n\n        ((!gossipsub.iwant ids)\n         (for (id ids)\n           (alet (msg (hash-get messages id))\n             (send! (!!pubsub.publish @source id msg)))))\n\n        ((!gossipsub.graft)\n         (unless (memq @source D)\n           (set! D (cons @source D))))\n\n        ((!gossipsub.prune)\n         (when (memq @source D)\n           (set! D (remq @source D))))\n\n        (! heartbeat (heartbeat!)))\n    (loop))\n#+END_SRC\n\n** Heartbeat\n\nThe heartbeat is responsible for actor state management and runs once a second:\n- when the mesh degree of the peer is less than the low water mark,\n  it selects some random known peers, adds them to the mesh\n  peer list, and emits ~GRAFT~ messages to notify them.\n- when the mesh degree of the peer is more than the high water mark,\n  it selects some random mesh peers, drops them from the mesh\n  peer list, and emits ~PRUNE~ messages to notify them.\n- the history of messages is rolled by 1, and if it exceeds\n  ~history-length~, the earliest seen messages are forgotten.\n- The message ids of messages seen in the last ~history-gossip~ windows\n  are forwarded to ~N~ random peers with an ~IHAVE~ gossip message.\n\n#+BEGIN_SRC gerbil\n  (def (heartbeat!)\n    (def d (length D))\n\n    ;; overlay management\n    (when (\u003c d N-low)\n      ;; we need some links, add some peers and send GRAFT\n      (let* ((i-need (- N d))\n             (candidates (filter (lambda (peer) (not (memq peer D)))\n                                 peers))\n             (candidates (shuffle candidates))\n             (new-peers (if (\u003e (length candidates) i-need)\n                          (take candidates i-need)\n                          candidates)))\n        (for (peer new-peers)\n          (send! (!!gossipsub.graft peer)))\n        (set! D (append D new-peers))))\n\n    (when (\u003e d N-high)\n      ;; we have too many links, drop some peers and send PRUNE\n      (let* ((to-drop (- d N))\n             (candidates (shuffle D))\n             (pruned-peers (take candidates to-drop)))\n        (for (peer pruned-peers)\n          (send! (!!gossipsub.prune peer)))\n        (set! D (filter (lambda (peer) (not (memq peer pruned-peers)))\n                        D))))\n\n    ;; message history management\n    (set! history (cons window history))\n    (set! window [])\n    (when (\u003e (length history) history-length)\n      (let (ids (last history))\n        (set! history\n          (drop-right history 1))\n        (for (id ids)\n          (hash-remove! messages id))))\n\n    ;; gossip about messages in our history (if any)\n    (let (ids (foldl (lambda (window r) (foldl cons r window))\n                     []\n                     (if (\u003e (length history) history-gossip)\n                       (take history history-gossip)\n                       history)))\n      (unless (null? ids)\n        (let* ((peers (shuffle peers))\n               (peers (if (\u003e (length peers) N)\n                        (take peers N)\n                        peers)))\n          (for (peer peers)\n            (unless (memq peer D)\n              (send! (!!gossipsub.ihave peer ids)))))))\n\n    (set! heartbeat (make-timeout 1)))\n#+END_SRC\n\n** Initialization\n\n#+BEGIN_SRC gerbil\n  (def (connect new-peers)\n    (let (new-peers (filter (lambda (peer) (not (memq peer peers)))\n                            new-peers))\n      (for (peer new-peers)\n        (send! (!!pubsub.connect peer)))\n      (set! peers\n        (foldl cons peers new-peers))))\n\n  (connect initial-peers)\n  (loop)\n#+END_SRC\n\n\n* Simulation\n\nThe [[simsub/simulator.ss][simulator]] constructs a network of ~N~ nodes, and randomly connects\nit with a connectivity degree ~N-connect~.\nThere is a random latency between any pair of nodes, selected uniformly\nin the ~[.01s, .15s]~ interval.\nThe simulation [[simsub/scripts.ss][script]] sends a number ~M~ of messages, by selecting ~fanout~ random\npeers and publishing to them. Each successive message is sent after some delay\n~M-delay~.\n\nHere are some example simulations with 100 and 1000 nodes:\n\n#+BEGIN_EXAMPLE\n$ gxi\n\u003e (import :vyzo/simsub/scripts)\n\u003e (simple-gossipsub-simulation trace: void) ; N = 100, N-connect = 10, M = 10, M-delay = 1\n=== simulation summary ===\nnodes: 100\nmessages: 10\nfanout: 5\npublish: 50\ndeliver: 1000\n!!gossipsub.graft: 380\n!!pubsub.connect: 1000\n!!gossipsub.prune: 7\n!!gossipsub.iwant: 31\n!!pubsub.publish: 6473\n!!gossipsub.ihave: 4402\n\n\u003e (simple-gossipsub-simulation trace: void messages: 100 message-delay: .1)\n=== simulation summary ===\nnodes: 100\nmessages: 100\nfanout: 5\npublish: 500\ndeliver: 10000\n!!gossipsub.graft: 374\n!!pubsub.connect: 1000\n!!gossipsub.prune: 8\n!!gossipsub.iwant: 163\n!!pubsub.publish: 63351\n!!gossipsub.ihave: 4844\n\n\u003e (simple-gossipsub-simulation trace: void messages: 1000 message-delay: .01)\n=== simulation summary ===\nnodes: 100\nmessages: 1000\nfanout: 5\npublish: 5000\ndeliver: 100000\n!!gossipsub.graft: 376\n!!pubsub.connect: 1000\n!!gossipsub.iwant: 1037\n!!pubsub.publish: 646973\n!!gossipsub.ihave: 8413\n\n\u003e (simple-gossipsub-simulation trace: void nodes: 1000)\n=== simulation summary ===\nnodes: 1000\nmessages: 10\nfanout: 5\npublish: 50\ndeliver: 10000\n!!gossipsub.graft: 3651\n!!pubsub.connect: 10000\n!!gossipsub.prune: 15\n!!gossipsub.iwant: 155\n!!pubsub.publish: 61957\n!!gossipsub.ihave: 45456\n\n\u003e (simple-gossipsub-simulation trace: void nodes: 1000 messages: 100 message-delay: .5)\n=== simulation summary ===\nnodes: 1000\nmessages: 100\nfanout: 5\npublish: 500\ndeliver: 100000\n!!gossipsub.graft: 3661\n!!pubsub.connect: 10000\n!!gossipsub.prune: 21\n!!gossipsub.iwant: 1146\n!!pubsub.publish: 621559\n!!gossipsub.ihave: 198372\n\n\u003e (simple-gossipsub-simulation trace: void nodes: 1000 messages: 100 message-delay: .1)\n=== simulation summary ===\nnodes: 1000\nmessages: 100\nfanout: 5\npublish: 500\ndeliver: 100000\n!!gossipsub.graft: 3740\n!!pubsub.connect: 10000\n!!gossipsub.prune: 53\n!!gossipsub.iwant: 20749\n!!pubsub.publish: 653634\n!!gossipsub.ihave: 84297\n\n#+END_EXAMPLE\n\nNote that as you run bigger simulations, you'll need a faster computer or\nthe simulator will lag. This can still be useful, as it analyzes the behaviour\nof the protocol in extreme lag conditions, where messages can take seconds to\npropagate some links.\n\nIf you want to see a trace of the developing simulation,\nthen omit the ~trace: void~ argument to the simulation invocation.\nThe default ~trace:~ will be ~displayln~, which will print out the simulation\nin the current output port.\n\nThe simulator also accepts a transcript procedure, which can save the simulation\ntrace to a file when it ends. For example, the following transcript function will\nsave the trace to ~/tmp/simsub.out~:\n\n#+BEGIN_EXAMPLE\n(def (transcript trace)\n  (let (trace (reverse trace))\n    (call-with-output-file \"/tmp/simsub.out\"\n      (lambda (port)\n        (parameterize ((current-output-port port))\n          (for-each displayln trace))))))\n\n\u003e (simple-gossipsub-simulation trace: void transcript: transcript)\n...\n#+END_EXAMPLE\n\n* License\n\nMIT; © 2018 vyzo\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvyzo%2Fgerbil-simsub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvyzo%2Fgerbil-simsub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvyzo%2Fgerbil-simsub/lists"}