{"id":13413579,"url":"https://github.com/stabbycutyou/buffstreams","last_synced_at":"2026-01-15T00:34:27.030Z","repository":{"id":34344783,"uuid":"38265793","full_name":"StabbyCutyou/buffstreams","owner":"StabbyCutyou","description":"A library to simplify writing applications using TCP sockets to stream protobuff messages","archived":false,"fork":false,"pushed_at":"2020-08-14T20:02:54.000Z","size":97,"stargazers_count":253,"open_issues_count":5,"forks_count":34,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-08-03T19:09:59.441Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StabbyCutyou.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-06-29T19:07:31.000Z","updated_at":"2024-04-26T18:02:49.000Z","dependencies_parsed_at":"2022-08-26T13:23:10.179Z","dependency_job_id":null,"html_url":"https://github.com/StabbyCutyou/buffstreams","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StabbyCutyou%2Fbuffstreams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StabbyCutyou%2Fbuffstreams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StabbyCutyou%2Fbuffstreams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StabbyCutyou%2Fbuffstreams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StabbyCutyou","download_url":"https://codeload.github.com/StabbyCutyou/buffstreams/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221498771,"owners_count":16833059,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:01:43.662Z","updated_at":"2026-01-15T00:34:26.955Z","avatar_url":"https://github.com/StabbyCutyou.png","language":"Go","readme":"BuffStreams [![GoDoc](http://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)](http://godoc.org/github.com/StabbyCutyou/buffstreams) [![Build Status](https://travis-ci.org/StabbyCutyou/buffstreams.svg)](https://travis-ci.org/StabbyCutyou/buffstreams)\n====================\n\nStreaming Protocol Buffers messages over TCP in Golang\n\nWhat is BuffStreams?\n=====================\n\nBuffStreams is a set of abstraction over TCPConns for streaming  connections that write data in a format involving the length of the message + the message payload itself (like Protocol Buffers, hence the name).\n\nBuffStreams gives you a simple interface to start a (blocking or non) listener on a given port, which will stream arrays of raw bytes into a callback you provide it. In this way, BuffStreams is not so much a daemon, but a library to build networked services that can communicate over TCP using Protocol Buffer messages.\n\nWhy BuffStreams\n====================\n\nI was writing a few different projects for fun in Golang, and kept writing code something like what is in the library, but less organized. I decided to focus on the networking code, pulling it out and improving it so I knew it could be trusted to perform reliably across projects.\n\nEthos\n=====\n\nThere is nothing special or magical about Buffstreams, or the code in here. The idea isn't that it's a better, faster socket abstraction - it's to do as much of the boilerplate for you when handling streaming data like protobuff messages, with as little impact to performance as possible. Currently, Buffstreams is able to do over 1.1 million messsages per second, at 110 bytes per message on a single listening socket which saturates a 1gig nic.\n\nThe idea of Buffstreams is to do the boring parts and handle common errors, enabling you to write systems on top of it, while performing with as little overhead as possible.\n\nHow does it work?\n=================\n\nSince protobuff messages lack any kind of natural delimeter, BuffStreams uses the method of adding a fixed header of bytes (which is configurable) that describes the size of the actual payload. This is handled for you, by the call to write. You never need to pack on the size yourself.\n\nOn the server side, it will listen for these payloads, read the fixed header, and then the subsequent message. The server must have the same maximum size as the client for this to work. BuffStreams will then pass the byte array to a callback you provided for handling messages received on that port. Deserializing the messages and interpreting their value is up to you.\n\nOne important note is that internally, BuffStreams does not actually use or rely on the Protocol Buffers library itself in any way. All serialization / deserialization is handled by the client prior to / after interactions with BuffStreams. In this way, you could theoretically use this library to stream any data over TCP that uses the same strategy of a fixed header of bytes + a subsequent message body.\n\nCurrently, I have only used it for ProtocolBuffers messages.\n\nLogging\n=======================\n\nYou can optionally enable logging of errors, although this naturally comes with a performance penalty under extreme load.\n\nBenchmarks\n==========\n\nI've tried very hard to optimize BuffStreams as best as possible, striving to keep it's averages above 1M messages per second, with no errors during transit.\n\nSee [Bench](https://github.com/StabbyCutyou/buffstreams/blob/master/BENCH.md)\n\nHow do I use it?\n===================\n\nDownload the library\n\n```go\ngo get \"github.com/StabbyCutyou/buffstreams\"\n```\n\nImport the library\n\n```go\nimport \"github.com/StabbyCutyou/buffstreams\"\n```\n\nFor a quick example of a complete end to end client and server, check out the examples in the test/ directory, namely test/client/test_client.go and test/server/test_server.go. These two files are designed to work together to demonstrate an end to end integration of Buffstreams, in the simplest possible way.\n\nListening for connections\n=========================\n\nOne of the core objects in Buffstreams is the TCPListener. This struct allows you to open a socket on a local port, and begin waiting for clients to connect. Once a connection is made, each full message written by the client will be received by the Listener, and a callback you define will be invoked with the message contents (an array of bytes).\n\nTo begin listening, first create a TCPListenerConfig object to define how the listener should behave. A sample TCPListenerConfig might look like this:\n\n```go\ncfg := TCPListenerConfig {\n  EnableLogging: false, // true will have log messages printed to stdout/stderr, via log\n  MaxMessageSize: 4096,\n  Callback: func(byte[])error{return nil} // Any function type that adheres to this signature, you'll need to deserialize in here if need be\n  Address: FormatAddress(\"\", strconv.Itoa(5031)) // Any address with the pattern ip:port. The FormatAddress helper is here for convenience. For listening, you normally don't want to provide an ip unless you have a reason.\n}\n```\n\n```go\nbtl, err := buffstreams.ListenTCP(cfg)\n```\nOnce you've opened a listener this way, the socket is now in use, but the listener itself has not yet begun to accept connections.\n\nTo do so, you have two choices. By default, this operation will block the current thread. If you want to avoid that, and use a fire and forget approach, you can call\n\n```go\nerr := btl.StartListeningAsync()\n```\n\nIf there is an error while starting up, it will be returned by this method. Alternatively, if you want to handle running the call yourself, or don't care that it blocks, you can call\n\n```go\nerr := btl.StartListening()\n```\n\nThe ListenCallback\n==================\n\nThe way Buffstreams handles acting over the incoming messages is to let you provide a callback to operate on the bytes. ListenCallback takes in an array/slice of bytes, and returns an error.\n\n```go\ntype ListenCallback func([]byte) error\n```\n\nThe callback will receive the raw bytes for a given protobuff message. The header containing the size will have been removed. It is the callbacks responsibility to deserialize and act upon the message.\n\nThe Listener gets the message, your callback does the work.\n\nA sample callback might start like so:\n\n```go\n  func ListenCallbackExample ([]byte data) error {\n    msg := \u0026message.ImportantProtoBuffStreamingMessage{}\n    err := proto.Unmarshal(data, msg)\n    // Now you do some stuff with msg\n    ...\n  }\n```\n\nThe callback is currently run in it's own goroutine, which also handles reading from the connection until the reader disconnects, or there is an error. Any errors reading from a connection incoming will be up to the client to handle.\n\nWriting messages\n================\n\nTo begin writing messages to a new connection, you'll need to dial a using TCPConnConfig\n\n```go\ncfg := TCPConnConfig {\n  EnableLogging: false, // true will have log messages printed to stdout/stderr, via log\n  MaxMessageSize: 4096, // You want this to match the MaxMessageSize the server expects for messages on that socket\n  Address: FormatAddress(\"127.0.0.1\", strconv.Itoa(5031)) // Any address with the pattern ip:port. The FormatAddress helper is here for convenience.\n}\n```\n\nOnce you have a configuration object, you can Dial out.\n\n```go\nbtc, err := buffstreams.DialTCP(cfg)\n```\n\nThis will open a connection to the endpoint at the specified location. Additionally, the TCPConn that the TCPListener returns will also allow you to write data, using the same methods as below.\n\nFrom there, you can write your data\n\n```go\n  bytesWritten, err := btc.Write(msgBytes, true)\n```\n\nIf there is an error in writing, that connection will be closed and be reopened on the next write. There is no guarantee if any the bytesWritten value will be \u003e0 or not in the event of an error which results in a reconnect.\n\nManager\n===========\n\nThere is a third option, the provided Manager class. This class will give you a simple but effective Manager abstraction over dialing and listening over ports, managing the connections for you. You provide the normal configuration for dialing out or listening for incoming connections, and let the manager hold onto the references. The Manager is considered threadsafe, as it internally uses locks to ensure consistency and coordination between concurrent access to the connections being held.\n\nThe Manager is not really a \"Pool\", in that it doesn't open and hold X connections for you to re-use. However, it maintains many of the same behaviors as a pool, including caching and re-using connections, and as mentioned is threadsafe.\n\nCreating a Manager\n\n```go\nbm := buffstreams.NewManager()\n```\n\nListening on a port. Manager always makes this asynchronous and non blocking\n\n```go\n// Assuming you've got a configuration object cfg, see above\nerr := bm.StartListening(cfg)\n```\n\nDialing out to a remote endpoint\n\n```go\n// Assuming you've got a configuration object cfg, see above\nerr := bm.Dial(cfg)\n```\n\nHaving opened a connection, writing to that connection in a constant fashion\n\n```go\nbytesWritten, err := bm.Write(\"127.0.0.1:5031\", dataBytes)\n```\n\nThe Manager will keep listening and dialed out connections cached internally. Once you open one, it'll be kept open. The writer will match your incoming write destination, such that any time you write to that same address, the correct writer will be re-used. The listening connection will simply remain open, waiting to receive requests.\n\nYou can forcibly close these connections, by calling either\n\n```go\nerr := bm.CloseListener(\"127.0.0.1:5031\")\n```\n\nor\n\n```go\nerr := bm.CloseWriter(\"127.0.0.1:5031\")\n```\n\nThanks\n=======\nSpecial thanks to those who have reported bugs or helped me improve Buffstreams\n* Judson White\n\nRoadmap\n=======\n* Release proper set of benchmarks, including more real-world cases\n* Configurable retry for the client, configurable errored-message queue for user to define failover process to handle.\n* Optional channel based streaming approach instead of callbacks\n* Further library optimizations via tools such as pprof\n\nLICENSE\n=========\nApache v2 - See LICENSE\n","funding_links":[],"categories":["Networking","网络","\u003cspan id=\"网络-networking\"\u003e网络 Networking\u003c/span\u003e","網絡","网络相关库"],"sub_categories":["Transliteration","Advanced Console UIs","音译","\u003cspan id=\"高级控制台用户界面-advanced-console-uis\"\u003e高级控制台用户界面 Advanced Console UIs\u003c/span\u003e","Uncategorized","交流","高級控制台界面","Strings","暂未分类这些库被放在这里是因为其他类别似乎都不适合。","暂未分类","高级控制台界面"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstabbycutyou%2Fbuffstreams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstabbycutyou%2Fbuffstreams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstabbycutyou%2Fbuffstreams/lists"}