{"id":13983719,"url":"https://github.com/lithdew/reliable","last_synced_at":"2025-04-23T22:03:59.078Z","repository":{"id":62828455,"uuid":"261434063","full_name":"lithdew/reliable","owner":"lithdew","description":"A reliability layer for UDP connections in Go.","archived":false,"fork":false,"pushed_at":"2020-05-31T01:30:23.000Z","size":147,"stargazers_count":138,"open_issues_count":1,"forks_count":16,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-23T22:03:52.537Z","etag":null,"topics":["arq","golang","network","udp"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lithdew.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-05T10:55:13.000Z","updated_at":"2024-11-30T03:18:49.000Z","dependencies_parsed_at":"2022-11-07T13:00:18.116Z","dependency_job_id":null,"html_url":"https://github.com/lithdew/reliable","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lithdew%2Freliable","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lithdew%2Freliable/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lithdew%2Freliable/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lithdew%2Freliable/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lithdew","download_url":"https://codeload.github.com/lithdew/reliable/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250522302,"owners_count":21444511,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arq","golang","network","udp"],"created_at":"2024-08-09T05:01:52.609Z","updated_at":"2025-04-23T22:03:58.744Z","avatar_url":"https://github.com/lithdew.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# reliable\n\n[![MIT License](https://img.shields.io/apm/l/atomic-design-ui.svg?)](LICENSE)\n[![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go\u0026logoColor=white\u0026style=flat-square)](https://pkg.go.dev/github.com/lithdew/reliable)\n[![Discord Chat](https://img.shields.io/discord/697002823123992617)](https://discord.gg/HZEbkeQ)\n\n**reliable** is a reliability layer for UDP connections in Go.\n\nWith only 9 bytes of packet overhead at most, what **reliable** does for your UDP-based application is:\n\n1. handle acknowledgement over the recipient of packets you sent,\n2. handle sending acknowledgements when too many are being buffered up,\n3. handle resending sent packets whose recipient hasn't been acknowledged after some timeout, and\n4. handle stopping/buffering up packets to be sent when the recipients read buffer is suspected to be full.\n\n** This project is still a WIP! Scrub through the source code, write some unit tests, help out with documentation, or open up a Github issue if you would like to help out or have any questions!\n\n## Protocol\n\n### Packet Header\n\n**reliable** uses the same packet header layout described in [`networkprotocol/reliable.io`](https://github.com/networkprotocol/reliable.io).\n\nAll packets start with a single byte (8 bits) representing 8 different flags. Packets are sequential, and are numbered using an unsigned 16-bit integer included in the packet header unless the packet is marked to be unreliable.\n\nPacket acknowledgements (ACKs) are redundantly included in every sent packet using a total of 5 bytes: two bytes representing an unsigned 16-bit packet sequence number (ack), and three bytes representing a 32-bit bitfield (ackBits).\n\nThe packet header layout, much like [`networkprotocol/reliable.io`](https://github.com/networkprotocol/reliable.io), is delta-encoded and RLE-encoded to reduce the size overhead per packet.\n\n### Packet ACKs\n\nGiven a packet we have just received from our peer, for each set bit (i) in the bitfield (ackBits), we mark a packet we have sent to be acknowledged if its sequence number is (ack - i).\n\nIn the case of peer A sending packets to B, with B not sending any packets at all to A, B will send an empty packet for every 32 packets received from A so that A will be aware that B has acknowledged its packets.\n\nMore explicitly, a counter (lui) is maintained representing the last consecutive packet sequence number that we have received whose acknowledgement we have told to our peer about.\n\nFor example, if (lui) is 0, and we have sent acknowledgements for packets whose sequence numbers are 2, 3, 4, and 6, and we have then acknowledged packet sequence number 1, then lui would be 4.\n\nUpon updating (lui), if the next 32 consecutive sequence numbers are sequence numbers of packets we have previously received, we will increment (lui) by 32 and send a single empty packet containing the following packet acknowledgements: (ack=lui+31, ackBits=[lui,lui+31]).\n\n### Packet Buffering\n\nTwo fixed-sized sequence buffers are maintained for packets that we have sent (wq), and packets that we have received (rq). The size fixed for these buffers must evenly divide into the max value of an unsigned 16-bit integer (65536). The data structure is described in [this blog post by Glenn Fiedler](https://gafferongames.com/post/reliable_ordered_messages/).\n\nWe keep track of a counter (oui), representing the last consecutive sequence number of a packet we have sent that was acknowledged by our peer. For example, if we have sent packets whose sequence numbers are in the range [0, 256], and we have received acknowledgements for packets (0, 1, 2, 3, 4, 8, 9, 10, 11, 12), then (oui) would be 4.\n\nLet cap(q) be the fixed size or capacity of sequence buffer q.\n\nWhile sending packets, we intermittently stop and buffer the sending of packets if we believe sending more packets would overflow the read buffer of our recipient. More explicitly, if the next packet we sent is assigned a packet number greater than (oui + cap(rq)), we stop all sends until (oui) has incremented through the recipient of a packet from our peer.\n\n### Retransmitting Lost Packets\n\nThe logic for retransmitting stale, unacknowledged sent packets and maintaining acknowledgements was taken with credit to [this blog post by Glenn Fiedler](https://gafferongames.com/post/reliable_ordered_messages/).\n\nPackets are suspected to be lost if they are not acknowledged by their recipient after 100ms. Once a packet is suspected to be lost, it is resent. As of right now, packets are resent for a maximum of 10 times.\n\nIt might be wise to not allow packets to be resent a capped number of times, and to leave it up to the developer. However, that is open for discussion which I am happy to have over on my Discord server or through a Github issue.\n\n## Rationale\n\nOn my quest for finding a feasible solution against TCP head-of-line blocking, I looked through a _lot_ of reliable UDP libraries in Go, with the majority primarily suited for either file transfer or gaming:\n\n1. A direct port of the reference C implementation of [networkprotocol/reliable.io](https://github.com/networkprotocol/reliable.io): [jakecoffman/rely](https://github.com/jakecoffman/rely)\n2. A realtime multiplayer network gaming protocol: [obsilp/rmnp](https://github.com/obsilp/rmnp)\n3. A reliable, production-grade ARQ protocol: [xtaci/kcp-go](https://github.com/xtaci/kcp-go)\n4. An encrypted session-based streaming protocol: [ooclab/es](https://github.com/ooclab/es/tree/master/proto/udp)\n5. A game networking protocol: [arl/udpnet](https://github.com/arl/udpnet)\n6. A direct port of uTP (Micro Transport Protocol): [warjiang/utp](https://github.com/warjiang/utp/tree/master/utp)\n7. A protocol for sending arbitrarily large, chunked amounts of data: [go-guoyk/sptp](https://github.com/go-guoyk/sptp)\n8. A small-scale fast transmission protocol: [spance/suft](https://github.com/spance/suft/)\n9. A direct port of QUIC: [lucas-clemente/quic-go](https://github.com/lucas-clemente/quic-go)\n\nGoing through all of them, I felt that they did just a little too much for me. For my work and side projects, I have been working heavily on decentralized p2p networking protocols. The nature of these protocols is that they suffer heavily from TCP head-of-line blocking operating in high-latency/high packet loss environments.\n\nIn many cases, a lot of the features provided by these libraries were either not needed, or honestly felt like they would best be handled and thought through by the developer using these libraries. For example:\n \n1. handshaking/session management\n2. packet fragmentation/reassembly\n3. packet encryption/decryption\n\nSo, I began working on a modular approach and decided to abstract away the reliability portion of protocols I have built into a separate library.\n\nI feel that this approach is best versus the popular alternatives like QUIC or SCTP that may, depending on your circumstances, do just a bit too much for you. After all, getting _just_ the reliability bits of a UDP-based protocol correct and well-tested is hard enough.\n\n## Todo\n\n1. Estimate the round-trip time (RTT) and adjust the system's packet re-transmission delay based off of it.\n2. Encapsulate away protocol logic and `net.PacketConn`-related bits for a finer abstraction.\n3. Keep a cache of the string representations of passed-in `net.UDPAddr`.\n4. Reduce locking in as many code hot paths as possible.\n5. Networking statistics (packet loss, RTT, etc.).\n6. More unit tests.\n\n## Usage\n\n**reliable** uses Go modules. To include it in your project, run the following command:\n\n```\n$ go get github.com/lithdew/reliable\n```\n\nShould you just be looking to quickly get a project or demo up and running, use `Endpoint`. If you require more flexibility, consider directly working with `Conn`.\n\nNote that some sort of keep-alive mechanism or heartbeat system needs to be bootstrapped on top, otherwise packets may indefinitely be resent as they will have failed to be acknowledged. \n\n## Options\n\n1. The read buffer size may be configured using `WithReadBufferSize`. The default read buffer size is 256.\n2. The write buffer size may be configured using `WithWriteBufferSize`. The default write buffer size is 256.\n3. The minimum period of time before we retransmit an packet that has yet to be acknowledged may be configured using `WithResendTimeout`. The default resend timeout is 100 milliseconds.\n4. A packet handler which is to be called back when a packet is received may be configured using `WithEndpointPacketHandler` or `WithProtocolPacketHandler`. By default, a nil handler is provided which ignores all incoming packets.\n5. An error handler which is called when errors occur on a connection that may be configured using `WithEndpointErrorHandler` or `WithProtocolErrorHandler`. By default, a nil handler is provided which ignores all errors.\n6. A byte buffer pool may be passed in using `WithBufferPool`. By default, a new byte buffer pool is instantiated.\n\n## Benchmarks\n\nA benchmark was done using [`cmd/benchmark`](examples/benchmark) from Japan to a DigitalOcean 2GB / 60 GB Disk / NYC3 server.\n\nThe benchmark task was to spam 1400 byte packets from Japan to New York. Given a ping latency of roughly 220 milliseconds, the throughput was roughly 1.2 MiB/sec.\n\nUnit test benchmarks have also been performed, as shown below.\n\n```\n$ cat /proc/cpuinfo | grep 'model name' | uniq\nmodel name : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz\n\n$ go test -bench=. -benchtime=10s\ngoos: linux\ngoarch: amd64\npkg: github.com/lithdew/reliable\nBenchmarkEndpointWriteReliablePacket-8           2053717              5941 ns/op             183 B/op          9 allocs/op\nBenchmarkEndpointWriteUnreliablePacket-8         2472392              4866 ns/op             176 B/op          8 allocs/op\nBenchmarkMarshalPacketHeader-8                  749060137               15.7 ns/op             0 B/op          0 allocs/op\nBenchmarkUnmarshalPacketHeader-8                835547473               14.6 ns/op             0 B/op          0 allocs/op\n```\n\n## Example\n\nYou may run the example below by executing the following command:\n\n```\n$ go run github.com/lithdew/reliable/examples/basic\n```\n\nThis example demonstrates:\n\n1. how to quickly construct two UDP endpoints listening on ports 44444 and 55555, and\n2. how to have the UDP endpoint at port 44444 spam 1400-byte packets to the UDP endpoint at port 55555 as fast as possible.\n\n```go\npackage main\n\nimport (\n\t\"bytes\"\n\t\"errors\"\n\t\"github.com/davecgh/go-spew/spew\"\n\t\"github.com/lithdew/reliable\"\n\t\"io\"\n\t\"log\"\n\t\"net\"\n\t\"os\"\n\t\"os/signal\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n)\n\nvar (\n\tPacketData = bytes.Repeat([]byte(\"x\"), 1400)\n\tNumPackets = uint64(0)\n)\n\nfunc check(err error) {\n\tif err != nil \u0026\u0026 !errors.Is(err, io.EOF) {\n\t\tlog.Panic(err)\n\t}\n}\n\nfunc listen(addr string) net.PacketConn {\n\tconn, err := net.ListenPacket(\"udp\", addr)\n\tcheck(err)\n\treturn conn\n}\n\nfunc handler(buf []byte, _ net.Addr) {\n\tif bytes.Equal(buf, PacketData) || len(buf) == 0 {\n\t\treturn\n\t}\n\tspew.Dump(buf)\n\tos.Exit(1)\n}\n\nfunc main() {\n\texit := make(chan struct{})\n\n\tvar wg sync.WaitGroup\n\twg.Add(2)\n\n\tca := listen(\"127.0.0.1:44444\")\n\tcb := listen(\"127.0.0.1:55555\")\n\n\ta := reliable.NewEndpoint(ca, reliable.WithEndpointPacketHandler(handler))\n\tb := reliable.NewEndpoint(cb, reliable.WithEndpointPacketHandler(handler))\n\n\tdefer func() {\n\t\tcheck(ca.SetDeadline(time.Now().Add(1 * time.Millisecond)))\n\t\tcheck(cb.SetDeadline(time.Now().Add(1 * time.Millisecond)))\n\n\t\tclose(exit)\n\n\t\tcheck(a.Close())\n\t\tcheck(b.Close())\n\n\t\tcheck(ca.Close())\n\t\tcheck(cb.Close())\n\n\t\twg.Wait()\n\t}()\n\n\tgo a.Listen()\n\tgo b.Listen()\n\n\t// The two goroutines below have endpoint A spam endpoint B, and print out how\n\t// many packets of data are being sent per second.\n\n\tgo func() {\n\t\tdefer wg.Done()\n\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase \u003c-exit:\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\n\t\t\tcheck(a.WriteReliablePacket(PacketData, b.Addr()))\n\t\t\tatomic.AddUint64(\u0026NumPackets, 1)\n\t\t}\n\t}()\n\n\tgo func() {\n\t\tdefer wg.Done()\n\n\t\tticker := time.NewTicker(1 * time.Second)\n\t\tdefer ticker.Stop()\n\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase \u003c-exit:\n\t\t\t\treturn\n\t\t\tcase \u003c-ticker.C:\n\t\t\t\tnumPackets := atomic.SwapUint64(\u0026NumPackets, 0)\n\t\t\t\tnumBytes := float64(numPackets) * 1400.0 / 1024.0 / 1024.0\n\n\t\t\t\tlog.Printf(\n\t\t\t\t\t\"Sent %d packet(s) comprised of %.2f MiB worth of data.\",\n\t\t\t\t\tnumPackets,\n\t\t\t\t\tnumBytes,\n\t\t\t\t)\n\t\t\t}\n\t\t}\n\t}()\n\n\tch := make(chan os.Signal, 1)\n\tsignal.Notify(ch, os.Interrupt)\n\t\u003c-ch\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flithdew%2Freliable","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flithdew%2Freliable","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flithdew%2Freliable/lists"}