{"id":37104299,"url":"https://github.com/thisisdevelopment/flashflood","last_synced_at":"2026-01-14T12:36:32.423Z","repository":{"id":113560343,"uuid":"219473616","full_name":"thisisdevelopment/flashflood","owner":"thisisdevelopment","description":"flashflood is a ringbuffer on steroids for golang","archived":false,"fork":false,"pushed_at":"2025-09-25T21:02:51.000Z","size":57,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-09-25T22:45:11.886Z","etag":null,"topics":["buffer","channels","go","golang","leaky-bucket","ring-buffer","ringbuffer"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thisisdevelopment.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-11-04T10:20:59.000Z","updated_at":"2025-09-25T18:22:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"97807d96-4f2c-4878-b50c-0715b5952f8c","html_url":"https://github.com/thisisdevelopment/flashflood","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/thisisdevelopment/flashflood","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thisisdevelopment%2Fflashflood","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thisisdevelopment%2Fflashflood/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thisisdevelopment%2Fflashflood/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thisisdevelopment%2Fflashflood/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thisisdevelopment","download_url":"https://codeload.github.com/thisisdevelopment/flashflood/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thisisdevelopment%2Fflashflood/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28420814,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T10:47:48.104Z","status":"ssl_error","status_checked_at":"2026-01-14T10:46:19.031Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["buffer","channels","go","golang","leaky-bucket","ring-buffer","ringbuffer"],"created_at":"2026-01-14T12:36:31.700Z","updated_at":"2026-01-14T12:36:32.415Z","avatar_url":"https://github.com/thisisdevelopment.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"- [Why FlashFlood?](#why-flashflood)\n- [Key Features](#key-features)\n- [Real-World Use Cases](#real-world-use-cases)\n- [Quick Start](#quick-start)\n- [Gate Mechanism - Predictable Batching](#gate-mechanism---predictable-batching)\n- [Advanced Usage](#advanced-usage)\n- [Performance](#performance)\n- [API Reference](#api-reference)\n- [About Us Th\\[is\\]](#about-us-this)\n- [Contributing](#contributing)\n- [License](#license)\n\n# FlashFlood v2: High-Performance Generic Ring Buffer\n\n[![go report card](https://goreportcard.com/badge/github.com/thisisdevelopment/flashflood \"go report card\")](https://goreportcard.com/report/github.com/thisisdevelopment/flashflood)\n[![codecov](https://codecov.io/gh/thisisdevelopment/flashflood/branch/master/graph/badge.svg)](https://codecov.io/gh/thisisdevelopment/flashflood)\n[![CircleCI](https://circleci.com/gh/thisisdevelopment/flashflood.svg?style=svg)](https://circleci.com/gh/thisisdevelopment/flashflood)\n[![GoDoc](https://godoc.org/github.com/thisisdevelopment/flashflood?status.svg)](https://godoc.org/github.com/thisisdevelopment/flashflood)\n\nFlashFlood v2 is a **high-performance generic ring buffer** with advanced batching capabilities that goes far beyond what standard Go channels offer. Built with full Go generics support, it provides compile-time type safety while delivering exceptional performance. FlashFlood excels when you need **predictable batch sizes**, **automatic timeouts**, and **element transformations** without interface{} casting.\n\n## Why FlashFlood?\n\n**Standard Go channels** force you to choose:\n- Process elements one-by-one (inefficient for bulk operations)\n- Build complex batching logic yourself (error-prone and verbose)\n- Handle timeouts manually (more boilerplate)\n\n**FlashFlood v2 solves this** by providing:\n- ✅ **Type-safe generics** - no more interface{} casting or runtime type assertions\n- ✅ **Guaranteed batch sizes** via the gate mechanism\n- ✅ **Automatic timeout handling** for incomplete batches\n- ✅ **Generic transformations** with FuncStack[T] callbacks\n- ✅ **Thread-safe operations** with minimal overhead\n- ✅ **Back-pressure control** through configurable buffer sizes\n- ✅ **50%+ performance improvement** over v1 with generics optimization\n\n## Key Features\n\n### 🎯 **Gate-Based Batching**\nSet `GateAmount: 10` and receive exactly 10 elements at a time - perfect for database bulk inserts, API batch calls, or file writing operations.\n\n### ⏱️ **Smart Timeout Handling**\nBuffer doesn't fill to gate size? No problem - automatic timeout ensures data still flows even during low-traffic periods.\n\n### 🔄 **Element Transformations**\nApply transformations to batches before output using FuncStack callbacks - merge byte arrays, aggregate data, or format for APIs.\n\n### 🚀 **High Performance**\nBenchmarks show consistent performance with minimal allocations, even under heavy load.\n\n### 🔒 **Thread-Safe**\nConcurrent producers and consumers work seamlessly with internal mutex protection.\n\n## Real-World Use Cases\n\n### 📊 **Database Batch Inserts**\n```go\n// Collect exactly 100 records, then bulk insert\nff := flashflood.New[DatabaseRecord](\u0026flashflood.Opts{\n    BufferAmount: 1000,\n    GateAmount:   100,    // Always insert 100 records at once\n    Timeout:      5*time.Second,  // Flush incomplete batches after 5s\n})\n```\n\n### 🌐 **API Rate Limiting \u0026 Batching**\n```go\n// Group API calls into batches of 25 to stay under rate limits\nff := flashflood.New[APIRequest](\u0026flashflood.Opts{\n    GateAmount:   25,     // Batch 25 API calls together\n    Timeout:      2*time.Second,  // Don't wait longer than 2s\n})\n```\n\n### 📝 **Log Aggregation**\n```go\n// Collect log entries and write to disk efficiently\nff := flashflood.New[LogEntry](\u0026flashflood.Opts{\n    GateAmount:   50,     // Write 50 log entries at once\n    Timeout:      1*time.Second,  // Flush every second for real-time monitoring\n})\n```\n\n### 📦 **Message Queue Publishing**\n```go\n// Batch messages for better throughput\nff := flashflood.New[Message](\u0026flashflood.Opts{\n    GateAmount:   20,     // Publish 20 messages per batch\n    Timeout:      500*time.Millisecond,\n})\n```\n\n## Quick Start\n\n### Basic Usage\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"time\"\n    \"github.com/thisisdevelopment/flashflood/v2\"\n)\n\nfunc main() {\n    // Create buffer that flushes when 3+ elements or after 250ms\n    ff := flashflood.New[string](\u0026flashflood.Opts{\n        BufferAmount: 10,  // Internal buffer size\n        Timeout: 250 * time.Millisecond,\n    })\n\n    // Get the overflow channel\n    ch, _ := ff.GetChan()\n\n    // Add elements to buffer\n    ff.Push(\"item1\", \"item2\", \"item3\", \"item4\")\n\n    // Receive flushed elements\n    for i := 0; i \u003c 4; i++ {\n        select {\n        case item := \u003c-ch:\n            fmt.Printf(\"Received: %v\\n\", item)\n        }\n    }\n}\n```\n\n## Gate Mechanism - Predictable Batching\n\n**The gate mechanism is FlashFlood's killer feature** - it guarantees consistent batch sizes that your receivers can count on.\n\n### How Gates Work\n```go\nff := flashflood.New[string](\u0026flashflood.Opts{\n    BufferAmount: 100,    // Internal buffer holds 100 items\n    GateAmount:   10,     // Release exactly 10 items at a time\n    Timeout:      1*time.Second,  // Fallback: flush after 1 second\n})\n```\n\n**Behavior:**\n- Buffer collects elements until it has **exactly `GateAmount`** items\n- Then releases **all `GateAmount` items at once**\n- If timeout occurs before gate fills, flushes whatever is available\n- Receiver always knows: \"I'll get exactly 10 items, or it's a timeout flush\"\n\n### Gate Examples\n\n#### Database Batch Inserts\n```go\n// Always insert exactly 50 records at once\nff := flashflood.New[[]DatabaseRecord](\u0026flashflood.Opts{\n    GateAmount: 50,\n    Timeout:    5*time.Second,\n})\n\n// Add function to keep batched elements grouped together\nff.AddFunc(flashflood.FuncMergeChunkedElements[DatabaseRecord]())\n\n// Your receiver gets batches: either 50 records (normal) or \u003c50 (timeout)\nfor {\n    select {\n    case records := \u003c-ch:\n        // records is []DatabaseRecord - no casting needed!\n        if len(records) == 50 {\n            // Normal batch - optimal performance\n            db.BulkInsert(records)\n        } else {\n            // Timeout batch - still insert but log it\n            log.Printf(\"Timeout flush: %d records\", len(records))\n            db.BulkInsert(records)\n        }\n    }\n}\n```\n\n#### Byte Stream Processing\n```go\n// Process data in 1KB chunks\nff := flashflood.New[[]byte](\u0026flashflood.Opts{\n    GateAmount: 1024,  // Exactly 1KB chunks\n    Timeout:    100*time.Millisecond,\n})\n\nff.AddFunc(flashflood.FuncMergeBytes())  // Merge individual bytes into single chunk\n\n// Receiver gets exactly 1KB chunks for optimal processing\nfor {\n    select {\n    case chunk := \u003c-ch:\n        // chunk is always []byte of exactly 1024 bytes (or timeout)\n        processChunk(chunk)  // No casting needed!\n    }\n}\n```\n\n## Advanced Usage\n\n### Element Transformations with FuncStack\nApply functions to batches before they're sent to the channel:\n\n```go\nff := flashflood.New[string](\u0026flashflood.Opts{\n    GateAmount: 5,\n})\n\n// Add custom transformation\nff.AddFunc(func(items []string, ff *flashflood.FlashFlood[string]) []string {\n    // Transform each item (e.g., add timestamp)\n    result := make([]string, len(items))\n    for i, item := range items {\n        result[i] = fmt.Sprintf(\"%s_processed_at_%d\", item, time.Now().Unix())\n    }\n    return result\n})\n```\n\n### Built-in Transformation Functions\n```go\n// For FlashFlood[[]byte] - merge byte slices into single byte array\nffBytes := flashflood.New[[]byte](\u0026flashflood.Opts{GateAmount: 10})\nffBytes.AddFunc(flashflood.FuncMergeBytes())\n\n// For FlashFlood[byte] - return individual bytes from byte slices\nffByte := flashflood.New[byte](\u0026flashflood.Opts{GateAmount: 10})\nffByte.AddFunc(flashflood.FuncReturnIndividualBytes())\n\n// For FlashFlood[[]T] - keep elements grouped in chunks\nffChunked := flashflood.New[[]string](\u0026flashflood.Opts{GateAmount: 3})\nffChunked.AddFunc(flashflood.FuncMergeChunkedElements[string]())\n```\n\n### Multiple Transformations\n```go\n// Chain multiple transformations - they execute in order\nff.AddFunc(transformFunc1)  // Executes first\nff.AddFunc(transformFunc2)  // Then this\nff.AddFunc(transformFunc3)  // Finally this\n```\n\n### Manual Control\n```go\n// Force flush current buffer to channel\nff.Drain(true, false)  // (toChannel=true, respectGate=false)\n\n// Get elements directly without using channel\nitems, _ := ff.Get(10)  // Get up to 10 items\n\n// Check buffer status\ncount := ff.Count()  // How many items in buffer\n\n// Clear buffer\nff.Purge()\n\n// Update activity (resets timeout)\nff.Ping()\n```\n\n## Performance\n\nFlashFlood v2 with generics delivers exceptional performance across different scenarios:\n\n```\nOperation                                    Ops/sec    ns/op    Allocs\n─────────────────────────────────────────────────────────────────────\nBenchmarkPushChan-12                        7.9M       136.8    2\nBenchmarkPushNoChan-12                       9.6M       119.6    2\nBenchmarkPushChanGate-12                     8.8M       132.7    2\nBenchmarkPushChanBigBuffer-12                8.9M       136.8    2\nBenchmarkWithGet-12                          6.1M       201.6    5\n\nWith Callback Functions (Power of 2 scaling):\nBenchmarkPushChanBigBufferPowCBFunc/pow/1-12    2.8M       466.0    2\nBenchmarkPushChanBigBufferPowCBFunc/pow/16-12   1.9M       627.5    2\nBenchmarkPushChanBigBufferPowCBFunc/pow/1024-12 1.7M       691.2    2\n```\n\n**Key Performance Benefits:**\n- **High throughput**: 7-9+ million operations per second\n- **Low latency**: Sub-200ns operations with generics optimization\n- **Minimal allocations**: Typically just 2 allocations per operation\n- **Consistent performance**: Stable across different gate sizes and buffer configurations\n- **Type safety**: Zero-cost generics provide compile-time type checking\n\n## API Reference\n\n### Core Methods\n\n```go\n// Create new FlashFlood instance with type parameter\nff := flashflood.New[string](\u0026flashflood.Opts{\n    BufferAmount:  100,               // Internal buffer size\n    GateAmount:    10,                // Batch size for releases\n    Timeout:       1*time.Second,     // Auto-flush timeout\n    TickerTime:    10*time.Millisecond, // Timeout check frequency\n    ChannelBuffer: 1000,              // Output channel buffer size\n    Debug:         false,             // Enable debug output\n})\n\n// Get output channel (returns \u003c-chan string)\nch, err := ff.GetChan()\n\n// Add elements (type-safe)\nff.Push(\"item1\", \"item2\", \"item3\")\nff.Unshift(\"priority_item\")  // Add to front of buffer\n\n// Manual operations\nff.Drain(true, false)      // Force flush to channel (toChannel, respectGate)\nitems, _ := ff.Get(5)      // Get up to 5 items directly (returns []string)\nff.GetOnChan(5)           // Get 5 items and send to channel\nff.Purge()                // Clear buffer (returns error)\ncount := ff.Count()       // Buffer size (returns uint64)\nff.Ping()                 // Reset timeout (no return value)\nff.Close()                // Cleanup resources\n\n// Add transformations (type-safe)\nff.AddFunc(func(items []string, ff *flashflood.FlashFlood[string]) []string {\n    // Your transformation logic here\n    return items\n})\n```\n\n### Configuration Options\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `BufferAmount` | 256 | Internal buffer size before overflow |\n| `GateAmount` | 1 | Number of elements to release at once |\n| `Timeout` | 100ms | Time before auto-flushing incomplete batches |\n| `TickerTime` | 10ms | How often to check for timeouts |\n| `ChannelBuffer` | 4096 | Output channel buffer size |\n| `FlushTimeout` | 0 | Alternative timeout for different flush behavior |\n| `FlushEnabled` | false | Enable separate flush timeout logic |\n| `Debug` | false | Print debug information |\n| `DisableRingUntilChanActive` | false | Prevent overflow until channel is retrieved |\n\n**Full documentation and more examples:** https://godoc.org/github.com/thisisdevelopment/flashflood/v2\n\n## About Us Th[is]\n\n[This.nl](https://this.nl) is a digital agency based in Utrecht, the Netherlands, specializing in crafting high-performance, resilient, and scalable digital solutions, api's, microservices, and more. FlashFlood represents our commitment to building robust, efficient tooling that solves real-world performance challenges in Go applications.\n\n# Contributing\nYou can help to deliver a better flashflood buffer, check out how you can do things [CONTRIBUTING.md](CONTRIBUTING.md)\n\n# License \n© [This is Development BV](https://www.thisisdevelopment.nl), 2019~time.Now()\nReleased under the [MIT License](https://github.com/thisisdevelopment/flashflood/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthisisdevelopment%2Fflashflood","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthisisdevelopment%2Fflashflood","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthisisdevelopment%2Fflashflood/lists"}