{"id":26667119,"url":"https://github.com/joeycumines/go-bigbuff","last_synced_at":"2025-07-06T17:33:03.719Z","repository":{"id":48883847,"uuid":"129746376","full_name":"joeycumines/go-bigbuff","owner":"joeycumines","description":"Package bigbuff implements a one-many ordered queue producer-consumer pattern + utilities.","archived":false,"fork":false,"pushed_at":"2024-12-29T13:50:08.000Z","size":287,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-11T23:51:29.846Z","etag":null,"topics":["async","concurrency","consumer","context","debounce","golang","message-queue","ordered","producer","producer-consumer","queue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joeycumines.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-04-16T13:17:39.000Z","updated_at":"2024-12-29T13:50:11.000Z","dependencies_parsed_at":"2024-01-17T22:37:36.927Z","dependency_job_id":"5a358ea0-5fd6-442c-a45c-de6ec9a22ec8","html_url":"https://github.com/joeycumines/go-bigbuff","commit_stats":null,"previous_names":["joeycumines/bigbuff.go"],"tags_count":26,"template":false,"template_full_name":null,"purl":"pkg:github/joeycumines/go-bigbuff","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeycumines%2Fgo-bigbuff","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeycumines%2Fgo-bigbuff/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeycumines%2Fgo-bigbuff/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeycumines%2Fgo-bigbuff/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joeycumines","download_url":"https://codeload.github.com/joeycumines/go-bigbuff/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeycumines%2Fgo-bigbuff/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263942567,"owners_count":23533419,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","concurrency","consumer","context","debounce","golang","message-queue","ordered","producer","producer-consumer","queue"],"created_at":"2025-03-25T19:36:18.976Z","updated_at":"2025-07-06T17:33:03.676Z","avatar_url":"https://github.com/joeycumines.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# go-bigbuff\n\n\u003e test coverage: 97.1%\n\nPackage bigbuff implements many useful concurrency primitives and utilities. It was originally created around\n`bigbuff.Buffer`, a one-to-many unbounded FIFO queue with a built in retention policy mechanism, and two interfaces\n`bigbuff.Consumer`, and `bigbuff.Producer`, which generalise the pattern in a way that supports complex use cases such\nas at-least-once semantics. Over time this package has collected a range of different, often highly specialised\nimplementations, that deal with different aspects of the complicated challenge of concurrent programming.\n\nGodoc here: [github.com/joeycumines/go-bigbuff](https://godoc.org/github.com/joeycumines/go-bigbuff)\n\nSpecialised tools should be carefully considered against possible use cases, and those provided by this package are no\ndifferent. A good rule of thumb is to keep them peripheral and replaceable, where possible. Choose wisely ;).\n\n## Highlights\n\n- `bigbuff.ChanCaster` a high-performance, low-level, single-channel broadcast mechanism\n- `bigbuff.ChanPubSub` a high-performance, scalable, flexible, in-memory pub-sub, with notably low memory footprint\n- `bigbuff.Buffer` is the most mature implementation in this package, and is battle tested one-many producer / consumer\n   implementation. It operates as an unbounded FIFO queue by default, reaping messages after they are read by all (and\n   at least one) consumer. Custom behavior may be implemented using the `bigbuff.Cleaner` type. The buffer's behavior\n   may be modified to enforce (soft) bounding of the size of the queue, via the `bigbuff.FixedBufferCleaner` function.\n   Benchmarks... someday.\n   **PLEASE NOTE: Please carefully consider if you _actually_ need an unbounded buffer.** I remain confident that my use\n   cases are valid, but have since concluded that flow control patterns should be strongly considered. Even if those\n   patterns are in addition to the use of an unbounded buffer.\n- Any readable channel may be used as a `bigbuff.Consumer`, using `bigbuff.NewChannel`\n- `bigbuff.Notifier` uses `reflect.Select` to provide synchronous fan-out pub-sub (keyed, sending to any supported\n  channel that is subscribed at time of publish). This implementation is far more compact than `bigbuff.Buffer`, but\n  at this time it is not actively being used in a production environment. It is also far slower, and suitable for a\n  much smaller number of concurrent consumers (per key / buffer instance). That said it is much easier to use, and\n  solves my original problem case of dynamically wiring up existing message processing that makes heavy use of channels\n  very well. The learning curve for this implementation should be trivial, as it's behavior is identical to direct\n  interaction with multiple channels + context, using select.\n- `bigbuff.Exclusive` is another battle-tested implementation, providing debouncing of _keyed operations_, either as\n  part of a background process, or in the foreground, potentially blocking for a return value (which may be shared\n  between multiple, debounced callers)\n- Useful for limiting concurrent executions, `bigbuff.Workers` is compatible with `bigbuff.Exclusive`. It's a simple\n  on-demand background worker orchestrator, which deliberately uses a compatible function signature with relevant\n  methods from `bigbuff.Exclusive`. `bigbuff.MinDuration` is provided in the same vein.\n\n## Caveats\n\n- All implementations were learning experiences, and in some, particularly older cases, the API has been left unchanged\n  solely for backwards compatibility\n- Unbounded queues tend to require additional management, especially if there is any chance that the consumer side may\n  fall behind the producer side (cascading failures == bad)\n- The use cases of most of these tools are complicated enough to deserve extensive examples and discussion\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoeycumines%2Fgo-bigbuff","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoeycumines%2Fgo-bigbuff","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoeycumines%2Fgo-bigbuff/lists"}