{"id":36677910,"url":"https://github.com/bnclabs/gofast","last_synced_at":"2026-01-12T10:52:23.725Z","repository":{"id":24846839,"uuid":"28261797","full_name":"bnclabs/gofast","owner":"bnclabs","description":"High performance transport protocol for distributed applications.","archived":false,"fork":false,"pushed_at":"2021-02-10T11:14:02.000Z","size":1457,"stargazers_count":19,"open_issues_count":18,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-08-14T21:34:24.715Z","etag":null,"topics":["go","golang","multiplexer","network","peer-to-peer","pipeline","socketio","stream","transport"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bnclabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-12-20T08:42:35.000Z","updated_at":"2021-10-25T12:22:10.000Z","dependencies_parsed_at":"2022-08-26T22:30:33.797Z","dependency_job_id":null,"html_url":"https://github.com/bnclabs/gofast","commit_stats":null,"previous_names":["prataprc/gofast"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bnclabs/gofast","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bnclabs%2Fgofast","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bnclabs%2Fgofast/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bnclabs%2Fgofast/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bnclabs%2Fgofast/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bnclabs","download_url":"https://codeload.github.com/bnclabs/gofast/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bnclabs%2Fgofast/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28338847,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T10:40:25.642Z","status":"ssl_error","status_checked_at":"2026-01-12T10:39:27.820Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","multiplexer","network","peer-to-peer","pipeline","socketio","stream","transport"],"created_at":"2026-01-12T10:52:23.657Z","updated_at":"2026-01-12T10:52:23.711Z","avatar_url":"https://github.com/bnclabs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"Summary\n=======\n\n**High performance protocol for distributed applications.**\n\n[![Build Status](https://travis-ci.org/bnclabs/gofast.png)](https://travis-ci.org/bnclabs/gofast)\n[![Coverage Status](https://coveralls.io/repos/bnclabs/gofast/badge.png?branch=master\u0026service=github)](https://coveralls.io/github/bnclabs/gofast?branch=master)\n[![GoDoc](https://godoc.org/github.com/bnclabs/gofast?status.png)](https://godoc.org/github.com/bnclabs/gofast)\n[![Go Report Card](https://goreportcard.com/badge/github.com/bnclabs/gofast)](https://goreportcard.com/report/github.com/bnclabs/gofast)\n\n* [CBOR](http://cbor.io),(Concise Binary Object Representation) based\n  protocol, avoids yet another protocol frame. Well formed gofast packets\n  are fully [CBOR](http://cbor.io) compliant.\n* Symmetric protocol - communication starts by client initiating\n  the connection with server, but there after client and server can exchange\n  messages like peers, that is both ends can:\n  - `POST` messages to remote node.\n  - `REQUEST` a `RESPONSE` from remote node.\n  - Start one or more bi-direction `STREAM` with remote node.\n* Concurrent request on a single connection, improves throughput when\n  latency is high.\n* Configurable batching of packets scheduled for transmission.\n* Periodic flusher for batching response and streams.\n* Send periodic heartbeat to remote node.\n* Add transport level compression like `gzip`, `lzw` ...\n* Sub-μs protocol overhead.\n* Scales with number of connection and number of cores.\n* And most importantly - does not attempt to solve all the world's problem.\n\nQuick links\n-----------\n\n* [Frame-format](#frame-format).\n* [Settings][settings-link].\n* [Getting-started](docs/gettingstarted.md).\n* [Http-endpoints](docs/httpendpoints.md).\n* [Performance benchmark][perf-article].\n* [How to contribute](#how-to-contribute).\n\n**dev-notes**\n\n* `Transport{}` is safe for concurrent access.\n* `Stream{}` is not safe for concurrent access.\n\nFrame-format\n------------\n\nFrames encode packet in CBOR compliant form, identifying the exchange\nas one of the following:\n\n**Post-request**, client post a packet and expects no response:\n\n```text\n| 0xd9 0xd9f7 | 0xc6 | packet |\n```\n\n**Request-response**, client make a request and expects a single response:\n\n```text\n| 0xd9 0xd9f7 | 0x81 | packet |\n```\n\n**Bi-directional streaming**, where client and server will have to close\nthe stream by sending a 0xff:\n\n```text\n | 0xd9 0xd9f7         | 0x9f | packet1    |\n        | 0xd9 0xd9f7  | 0xc7 | packet2    |\n        ...\n        | 0xd9 0xd9f7  | 0xc8 | end-packet |\n```\n\n* `0xd9` says frame is a tag with 2-bye extension.\n* Following two bytes `0xd9f7` is tag-number `Tag-55799`.\n* As per the RFC - **0xd9 0xd9f7 appears not to be in use as a\n  distinguishing mark for frequently used file types**.\n* Maximum length of a packet can be 4GB.\n* 0xc6 is gofast reserved tag (tagvalue-6) to denote that the following\n  packet is a post.\n* 0x81 denotes a cbor array of single item, a special meaning for new\n  request that expects a single response from remote.\n* 0x9f denotes a cbor array of indefinite items, a special meaning\n  for a new request that starts a bi-directional stream.\n* 0xc7 is gofast reserved tag (tagvalue-7) to denote that the following\n  packet is part of a stream.\n* 0xc8 is gofast reserved tag (tagvalue-8) to denote that this packet\n  is an end-packet closing the bi-directional stream.\n* `Packet` shall always be encoded as CBOR byte-array.\n\nExcept for post-request, the exchange between client and server is always\nsymmetrical.\n\nPacket-format\n-------------\n\nA packet is CBOR byte-array that can carry tags and payloads, it has\nthe following format:\n\n```text\n  | len | tag1 |         payload1               |\n               | tag2 |      payload2           |\n                      | tag3 |   payload3       |\n                             | tag 4 | hdr-data |\n```\n\n* Entire packet is encoded as CBOR byte-array.\n* `len` is nothing but the byte-array length (Major-type-2).\n* Payload shall always be encoded as CBOR byte-array.\n* HDR-DATA shall always be encoded as CBOR map.\n* Tags are uint64 numbers that will either be prefixed to payload or hdr-data.\n* Tag1, will always be a opaque number falling within a reserved tag-space\n  called opaque-space.\n* **Opaque-space should not start before 256**.\n* Tag2, Tag3 can be one of the values predefined by this library.\n* Final embedded tag, in this case tag4, shall always be tagMsg (value 43).\n\n**hdr-data**\n\n* TagId, identifies message with unique id.\n* TagData, identified encoded message as byte array.\n\n**end-packet**\n\n```text\n    | len | tag1  | 0x40 | 0xff |\n```\n\n* End-packet does not carry any useful payload, it simply signifies a stream\n  close.\n* For that, tag1 opaque-value is required to identify the stream.\n* 0x40 is the single-byte payload for this packet, which says that the\n  payload-data is ZERO bytes.\n* The last 0xff is important since it will match with 0x9f that indicates\n  a stream-start as Indefinite array of items.\n\n**Reading a frame from socket**\n\nFraming of packets are done such that any gofast packet will at-least be 9\nbytes long. Here is how it happens:\n\n* The smallest `payload` should be at-least 1 byte length, because it is\nencoded as CBOR byte-array or as end-packet (0xff).\n* Every payload will be prefix with opaque-tag, which is always \u003e= 256 in\nvalue. That is 3 bytes.\n* Entire `packet` is encoded as CBOR byte-array, that is another 1 byte\noverhead.\n* And finally framing always takes up 4 bytes.\n\nThat is a total of: 1 + 3 + 1 + 4\n\nIncidentally these 9 bytes are enough to learn how many more bytes to read\nfrom the socket to complete the entire packet.\n\n**Regarding Opaque-value**\n\nBy design opaque value should be \u003e= 256. These are ephemeral tag values\nthat do not carry any meaning other than identifying the stream. Opaque\nvalues will continuously reused for the life-time of connection. Users\nare expected to give a range of these ephemeral tag-values, and gofast\nwill skip [reserved TAGS](https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml).\n\nReserved-tags\n-------------\n\nFollowing list of CBOR tags are reserved for gofast protocol. Some of these\ntags may be standardised by CBOR specification, but the choice of these\ntag values will speed-up frame encoding and decoding.\n\n`tag-6`\n    TagPost, following tagged CBOR byte-array carries a POST request.\n\n`tag-7`\n    TagStream, following tagged CBOR byte-array carries a STREAM message.\n\n`tag-8`\n    TagFinish, following is tagged CBOR breakstop (0xff) item.\n\n`tag-43`\n    TagMsg, following CBOR map carries message header and data.\n\n`tag-44`\n    TagId, used as key in CBOR header-data mapping to unique message ID.\n\n`tag-45`\n    TagData, used as key in CBOR header-data mapping to message, binary\n    encoded as CBOR byte-array.\n\n`tag-46`\n    TagGzip, following CBOR byte array is compressed using gzip encoding.\n\n`tag-47`\n    TagLzw, following CBOR byte array is compressed using gzip encoding.\n\nThese reserved tags are not part of CBOR specification or IANA registry,\nplease refer/follow issue [#1](https://github.com/bnclabs/gofast/issues/1).\n\nSizing\n------\n\nBased on the configuration following heap allocations can affect memory\nsizing.\n\n* Batch of packets copied into a single buffers before flushing into socket:\n  `writebuf := make([]byte, batchsize*buffersize)`\n  for configured range of opaque space between [opaque.start, opaque.end]\n* As many stream{} objects will be pre-created and pooled:\n  `((opaque.end-opaque.start)+1) * sizeof(stream{})`\n* Each stream will allocate 3 buffers for sending/receiving packets.\n  `buffersize * 3`\n* As many txproto{} objects will be pre-create and pooled:\n  `((opaque.end-opaque.start)+1) * sizeof(txproto{})`\n* As many tx protocol encode buffers will be pre-created and pooled:\n  `((opaque.end-opaque.start)+1) * buffersize`\n\nPanic and Recovery\n------------------\n\nPanics are to expected when APIs are misused. Programmers might choose\nto ignore the errors, but not panics. For example:\n\n* When trying to subscribe message to transport whose ID is already\n  reserved.\n* When transport is created with invalid opaque-range.\n* All transport instances are named, and if transports are created twice\n  with same name.\n* Trying to close an invalid transport.\n* When unforeseen panic happens in doTx() routine, it recovers\n  from panic, dumps the stack-trace, closes the transport and exits.\n* When unforeseen panic happens in doRx() routine, it recovers\n  from panic, dumps the stack-trace, closes the transport and exits.\n* When unforeseen panic happens in syncRx() routine, it recovers from\n  panic, dumps the stack trace, closes the transport, all live pending\n  streams and exits.\n\nHow to contribute\n-----------------\n\n[![Issue Stats](http://issuestats.com/github/bnclabs/gofast/badge/pr)](http://issuestats.com/github/bnclabs/gofast)\n[![Issue Stats](http://issuestats.com/github/bnclabs/gofast/badge/issue)](http://issuestats.com/github/bnclabs/gofast)\n\n* Pick an issue, or create an new issue. Provide adequate documentation for\n  the issue.\n* Assign the issue or get it assigned.\n* Work on the code, once finished, raise a pull request.\n* Gofast is written in [golang](https://golang.org/), hence expected to\n  follow the global guidelines for writing go programs.\n* If the changeset is more than few lines, please generate a\n  [report card][report-link].\n* As of now, branch ``master`` is the development branch.\n\n[perf-article]: https://prataprc.github.io/gofast-standalone-performance.html\n[settings-link]: https://godoc.org/github.com/bnclabs/gofast#DefaultSettings\n[report-link]: https://goreportcard.com/report/github.com/bnclabs/gofast\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbnclabs%2Fgofast","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbnclabs%2Fgofast","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbnclabs%2Fgofast/lists"}