{"id":13724146,"url":"https://github.com/acln0/zerocopy","last_synced_at":"2025-04-04T19:14:06.687Z","repository":{"id":43347951,"uuid":"178302899","full_name":"acln0/zerocopy","owner":"acln0","description":"Zero-copy I/O primitives and pipelines for Go. Linux-specific.","archived":false,"fork":false,"pushed_at":"2024-08-03T13:23:00.000Z","size":41,"stargazers_count":544,"open_issues_count":4,"forks_count":35,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-28T18:14:46.841Z","etag":null,"topics":["go","linux","pipelines","splice","tee","zero-copy"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/acln0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-29T00:28:22.000Z","updated_at":"2025-02-25T05:48:31.000Z","dependencies_parsed_at":"2025-01-18T15:16:19.444Z","dependency_job_id":null,"html_url":"https://github.com/acln0/zerocopy","commit_stats":{"total_commits":25,"total_committers":2,"mean_commits":12.5,"dds":"0.040000000000000036","last_synced_commit":"ac749309e8976edf3c02aa32e579b2c8f760fc51"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acln0%2Fzerocopy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acln0%2Fzerocopy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acln0%2Fzerocopy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acln0%2Fzerocopy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/acln0","download_url":"https://codeload.github.com/acln0/zerocopy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247234923,"owners_count":20905854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","linux","pipelines","splice","tee","zero-copy"],"created_at":"2024-08-03T01:01:51.111Z","updated_at":"2025-04-04T19:14:06.664Z","avatar_url":"https://github.com/acln0.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"zerocopy\n================\n\n`import \"acln.ro/zerocopy\"`\n\n[![GoDoc](https://godoc.org/acln.ro/zerocopy?status.svg)](https://godoc.org/acln.ro/zerocopy)\n\nPackage zerocopy facilitates the construction of accelerated I/O\npipelines. Under circumstances where I/O acceleration is not\npossible, the pipelines fall back to userspace data transfer\ntransparently. Currently, package zerocopy only offers accelerated\nI/O on Linux, and for specific types of file descriptors.\n\n## Status\n\nThis package is alpha quality software. It is under active development,\nit almost certainly has bugs, or exhibits deadlocks.\n\nBug reports and contributions are welcome.\n\n## Requirements\n\nPackage zerocopy requires at least Go 1.12. On the Go side of\nthings, package zerocopy uses type assertions on the `io.Reader`\nand `io.Writer` arguments to `ReadFrom`, `WriteTo`, or `Transfer`,\nin order to determine splice capabilities.\n\nThe first requirement is for the Go types to implement\nthe [syscall.Conn](https://golang.org/pkg/syscall/#Conn)\ninterface. Secondly, the `syscall.Conn` implementations must be\nbacked by real file descriptors, the file descriptors must be\nset to non-blocking mode, and must be registered with the runtime\nnetwork poller.\n\nMore concretely, `*net.TCPConn` and `*net.UnixConn` (on the `\"unix\"`\nnetwork, with `SOCK_STREAM` semantics) should work out of the\nbox. Non-exotic varieties of `*os.File` should also work.\n\nGenerally, file descriptors involved in such transfers must be\nstream-oriented. Stream-orientation is a necessary, but not sufficient\ncondition for `splice(2)` to work on a file descriptor. Consult the\nappropriate subsystem manual, or the kernel source code, if you need\nto be sure.\n\n## Usage\n\nConsider this diagram of a proxy server:\n\n```\n+----------+       +----------------+       +------------+\n|          +\u003c------+                +\u003c------+            |\n| upstream |   P   |  proxy server  |   Q   | downstream |\n|          +-------\u003e                +-------\u003e            |\n+----------+       +----------------+       +------------+\n```\n\n`P` and `Q` represent streaming communication protocols, for example\nTCP, or a streaming UNIX domain socket. Implementing this proxy server\nis straightforward:\n\n\n```go\nfunc proxy(upstream, downstream net.Conn) error {\n\tgo zerocopy.Transfer(upstream, downstream)\n\tgo zerocopy.Transfer(downstream, upstream)\n\n\t// ... wait, clean up, etc.\n}\n```\n\n---\n\nConsider now a slightly more complex data flow diagram.\n\n```\n                     +-----------+\n                     |           |\n                     | recording |\n                     |           |\n                     +-----+-----+\n                           ^\n                           |\n                           | file\n                           |\n+----------+       +-------+--------+       +----------+\n|          |  TCP  |                |  TCP  |          |\n|  camera  +------\u003e+     server     +------\u003e+  client  |\n|          |       |                |       |          |\n+----------+       +----------------+       +----------+\n```\n\nThis server component could be implemented as follows:\n\n```go\nfunc server(camera net.Conn, recording *os.File, client net.Conn) error {\n\t// Create a pipe between the camera and the client.\n\tcampipe, err := zerocopy.NewPipe()\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer campipe.Close()\n\n\t// Create a pipe to the recording.\n\trecpipe, err := zerocopy.NewPipe()\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer recpipe.Close()\n\n\t// Arrange for data on campipe to be duplicated to recpipe.\n\tcampipe.Tee(recpipe)\n\n\t// Run the world.\n\tgo campipe.ReadFrom(camera)\n\tgo recpipe.WriteTo(recording)\n\tgo campipe.WriteTo(client)\n\n\t// ... wait, clean up etc.\n}\n```\n\n## Additional reading\n\n* [man 2 splice](http://man7.org/linux/man-pages/man2/splice.2.html)\n* [man 2 tee](http://man7.org/linux/man-pages/man2/tee.2.html)\n* [Linus Torvalds on splice(2) and tee(2)](https://yarchive.net/comp/linux/splice.html)\n\n## Benchmarks\n\n### `Transfer` - in userspace vs. splice-accelerated\n\n```\nbenchmark                                    old ns/op     new ns/op     delta\nBenchmarkTransfer/tcp-to-tcp/1024-4          3972          3667          -7.68%\nBenchmarkTransfer/tcp-to-tcp/2048-4          3361          3193          -5.00%\nBenchmarkTransfer/tcp-to-tcp/4096-4          3522          3613          +2.58%\nBenchmarkTransfer/tcp-to-tcp/8192-4          4400          3840          -12.73%\nBenchmarkTransfer/tcp-to-tcp/16384-4         6293          4038          -35.83%\nBenchmarkTransfer/tcp-to-tcp/32768-4         13637         5465          -59.93%\nBenchmarkTransfer/tcp-to-tcp/65536-4         22652         10155         -55.17%\nBenchmarkTransfer/tcp-to-tcp/131072-4        44927         17892         -60.18%\nBenchmarkTransfer/tcp-to-tcp/262144-4        111338        37230         -66.56%\nBenchmarkTransfer/tcp-to-tcp/524288-4        227587        77118         -66.11%\nBenchmarkTransfer/tcp-to-tcp/1048576-4       482285        295034        -38.83%\nBenchmarkTransfer/unix-to-tcp/1024-4         1173          1342          +14.41%\nBenchmarkTransfer/unix-to-tcp/2048-4         1376          1478          +7.41%\nBenchmarkTransfer/unix-to-tcp/4096-4         2337          1814          -22.38%\nBenchmarkTransfer/unix-to-tcp/8192-4         2879          2167          -24.73%\nBenchmarkTransfer/unix-to-tcp/16384-4        5353          3422          -36.07%\nBenchmarkTransfer/unix-to-tcp/32768-4        9816          5651          -42.43%\nBenchmarkTransfer/unix-to-tcp/65536-4        20921         11871         -43.26%\nBenchmarkTransfer/unix-to-tcp/131072-4       37644         21996         -41.57%\nBenchmarkTransfer/unix-to-tcp/262144-4       76739         44324         -42.24%\nBenchmarkTransfer/unix-to-tcp/524288-4       148243        78678         -46.93%\nBenchmarkTransfer/unix-to-tcp/1048576-4      311495        151634        -51.32%\nBenchmarkTransfer/tcp-to-unix/1024-4         4258          3993          -6.22%\nBenchmarkTransfer/tcp-to-unix/2048-4         3268          3299          +0.95%\nBenchmarkTransfer/tcp-to-unix/4096-4         3572          3253          -8.93%\nBenchmarkTransfer/tcp-to-unix/8192-4         2776          3737          +34.62%\nBenchmarkTransfer/tcp-to-unix/16384-4        4556          4989          +9.50%\nBenchmarkTransfer/tcp-to-unix/32768-4        10038         6707          -33.18%\nBenchmarkTransfer/tcp-to-unix/65536-4        19597         10823         -44.77%\nBenchmarkTransfer/tcp-to-unix/131072-4       40673         15997         -60.67%\nBenchmarkTransfer/tcp-to-unix/262144-4       76247         30427         -60.09%\nBenchmarkTransfer/tcp-to-unix/524288-4       148763        60432         -59.38%\nBenchmarkTransfer/tcp-to-unix/1048576-4      307637        117157        -61.92%\nBenchmarkTransfer/unix-to-unix/1024-4        1090          1102          +1.10%\nBenchmarkTransfer/unix-to-unix/2048-4        1094          1111          +1.55%\nBenchmarkTransfer/unix-to-unix/4096-4        1558          1591          +2.12%\nBenchmarkTransfer/unix-to-unix/8192-4        2013          2597          +29.01%\nBenchmarkTransfer/unix-to-unix/16384-4       3373          2765          -18.03%\nBenchmarkTransfer/unix-to-unix/32768-4       6864          3327          -51.53%\nBenchmarkTransfer/unix-to-unix/65536-4       14177         9587          -32.38%\nBenchmarkTransfer/unix-to-unix/131072-4      26012         17204         -33.86%\nBenchmarkTransfer/unix-to-unix/262144-4      48500         31884         -34.26%\nBenchmarkTransfer/unix-to-unix/524288-4      102363        62569         -38.88%\nBenchmarkTransfer/unix-to-unix/1048576-4     204508        131515        -35.69%\n\nbenchmark                                    old MB/s     new MB/s     speedup\nBenchmarkTransfer/tcp-to-tcp/1024-4          257.77       279.21       1.08x\nBenchmarkTransfer/tcp-to-tcp/2048-4          609.32       641.23       1.05x\nBenchmarkTransfer/tcp-to-tcp/4096-4          1162.72      1133.52      0.97x\nBenchmarkTransfer/tcp-to-tcp/8192-4          1861.72      2132.81      1.15x\nBenchmarkTransfer/tcp-to-tcp/16384-4         2603.52      4057.02      1.56x\nBenchmarkTransfer/tcp-to-tcp/32768-4         2402.79      5995.63      2.50x\nBenchmarkTransfer/tcp-to-tcp/65536-4         2893.13      6453.09      2.23x\nBenchmarkTransfer/tcp-to-tcp/131072-4        2917.40      7325.41      2.51x\nBenchmarkTransfer/tcp-to-tcp/262144-4        2354.48      7041.18      2.99x\nBenchmarkTransfer/tcp-to-tcp/524288-4        2303.68      6798.46      2.95x\nBenchmarkTransfer/tcp-to-tcp/1048576-4       2174.18      3554.08      1.63x\nBenchmarkTransfer/unix-to-tcp/1024-4         872.62       762.84       0.87x\nBenchmarkTransfer/unix-to-tcp/2048-4         1487.48      1385.61      0.93x\nBenchmarkTransfer/unix-to-tcp/4096-4         1752.30      2257.64      1.29x\nBenchmarkTransfer/unix-to-tcp/8192-4         2844.91      3780.21      1.33x\nBenchmarkTransfer/unix-to-tcp/16384-4        3060.64      4787.20      1.56x\nBenchmarkTransfer/unix-to-tcp/32768-4        3337.99      5797.91      1.74x\nBenchmarkTransfer/unix-to-tcp/65536-4        3132.48      5520.36      1.76x\nBenchmarkTransfer/unix-to-tcp/131072-4       3481.81      5958.82      1.71x\nBenchmarkTransfer/unix-to-tcp/262144-4       3416.03      5914.26      1.73x\nBenchmarkTransfer/unix-to-tcp/524288-4       3536.66      6663.65      1.88x\nBenchmarkTransfer/unix-to-tcp/1048576-4      3366.26      6915.13      2.05x\nBenchmarkTransfer/tcp-to-unix/1024-4         240.46       256.40       1.07x\nBenchmarkTransfer/tcp-to-unix/2048-4         626.59       620.76       0.99x\nBenchmarkTransfer/tcp-to-unix/4096-4         1146.67      1258.78      1.10x\nBenchmarkTransfer/tcp-to-unix/8192-4         2950.44      2191.74      0.74x\nBenchmarkTransfer/tcp-to-unix/16384-4        3596.05      3283.65      0.91x\nBenchmarkTransfer/tcp-to-unix/32768-4        3264.14      4884.94      1.50x\nBenchmarkTransfer/tcp-to-unix/65536-4        3344.10      6055.24      1.81x\nBenchmarkTransfer/tcp-to-unix/131072-4       3222.57      8193.26      2.54x\nBenchmarkTransfer/tcp-to-unix/262144-4       3438.06      8615.32      2.51x\nBenchmarkTransfer/tcp-to-unix/524288-4       3524.30      8675.60      2.46x\nBenchmarkTransfer/tcp-to-unix/1048576-4      3408.48      8950.12      2.63x\nBenchmarkTransfer/unix-to-unix/1024-4        939.04       929.05       0.99x\nBenchmarkTransfer/unix-to-unix/2048-4        1871.53      1842.52      0.98x\nBenchmarkTransfer/unix-to-unix/4096-4        2627.44      2574.18      0.98x\nBenchmarkTransfer/unix-to-unix/8192-4        4068.58      3153.97      0.78x\nBenchmarkTransfer/unix-to-unix/16384-4       4856.35      5925.29      1.22x\nBenchmarkTransfer/unix-to-unix/32768-4       4773.31      9846.81      2.06x\nBenchmarkTransfer/unix-to-unix/65536-4       4622.41      6835.42      1.48x\nBenchmarkTransfer/unix-to-unix/131072-4      5038.74      7618.43      1.51x\nBenchmarkTransfer/unix-to-unix/262144-4      5405.02      8221.65      1.52x\nBenchmarkTransfer/unix-to-unix/524288-4      5121.83      8379.33      1.64x\nBenchmarkTransfer/unix-to-unix/1048576-4     5127.30      7972.99      1.56x\n```\n\n### `Tee`\n\nTODO(acln): add a benchmark\n\n## License\n\nPacakge zerocopy is distributed under a BSD-style license. Apart from\nthe work belonging to the author, package zerocopy adapts and copies\nGo standard library code and tests. See the LICENSE file, as well as\nthe individual copyright headers in source files.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facln0%2Fzerocopy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Facln0%2Fzerocopy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facln0%2Fzerocopy/lists"}