{"id":34837053,"url":"https://github.com/modelingevolution/zerobuffer","last_synced_at":"2025-12-25T16:08:33.906Z","repository":{"id":308118273,"uuid":"1031436011","full_name":"modelingevolution/zerobuffer","owner":"modelingevolution","description":null,"archived":false,"fork":false,"pushed_at":"2025-10-15T11:05:03.000Z","size":2492,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-11-29T07:35:01.506Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/modelingevolution.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-03T18:25:40.000Z","updated_at":"2025-11-21T10:53:56.000Z","dependencies_parsed_at":"2025-08-20T09:22:31.181Z","dependency_job_id":"543e2b41-13ab-408c-ae6a-3d1456426484","html_url":"https://github.com/modelingevolution/zerobuffer","commit_stats":null,"previous_names":["modelingevolution/zerobuffer"],"tags_count":42,"template":false,"template_full_name":null,"purl":"pkg:github/modelingevolution/zerobuffer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelingevolution%2Fzerobuffer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelingevolution%2Fzerobuffer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelingevolution%2Fzerobuffer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelingevolution%2Fzerobuffer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/modelingevolution","download_url":"https://codeload.github.com/modelingevolution/zerobuffer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelingevolution%2Fzerobuffer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28032372,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-25T02:00:05.988Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-25T16:06:56.277Z","updated_at":"2025-12-25T16:08:33.901Z","avatar_url":"https://github.com/modelingevolution.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ZeroBuffer\n\n[![NuGet](https://img.shields.io/nuget/v/ZeroBuffer.svg)](https://www.nuget.org/packages/ZeroBuffer/)\n[![PyPI](https://img.shields.io/pypi/v/zerobuffer-ipc.svg)](https://pypi.org/project/zerobuffer-ipc/)\n[![vcpkg](https://img.shields.io/badge/vcpkg-zerobuffer-blue)](https://github.com/modelingevolution/zerobuffer-vcpkg-registry)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\nA high-performance, zero-copy inter-process communication library with implementations in C++, C#, and Python. This ring-buffer implementation is designed for low-latency data transfer between processes, particularly for video streaming applications.\n\n**Supported platforms**: Linux, Windows, macOS (Python/C#), with platform-specific optimizations.\n\nThe protocol is based on structure called zero-buffer(name):\n- shared-memory (SHMB) defined by ${name}\n- named-semaphore called sem-w-${name} (write semaphore): signaled by writer when new data is available for reading\n- named-semaphore called sem-r-${name} (read semaphore): signaled by reader when data has been consumed and buffer space is available for writing\n\nThe SHMB:\n- Fixed size Operation info exchange block (64-byte aligned)\n- Fixed size Metadata block (64-byte aligned)\n- Fixed size Payload block (64-byte aligned)\n\n- Each item in the Payload's buffer is defined by the structure:\n8B: Payload size\n8B: Payload sequence number (unsigned long long)\ndynamic: Payload data\n\n## Operation info exchange block (OIEB)\n\nOperation exchange block is used to inform reader and writer about how much data has been written or read.\n8B: Operation size\n\n8B: Metadata size\n8B: Metadata free bytes\n8B: Metadata written bytes\n\n8B: Payload size\n8B: Payload free bytes\n8B: Payload written bytes\n8B: Payload written count\n8B: Payload read count\n\n8B: Writer PID (process ID of writer, 0 if no writer attached)\n8B: Reader PID (process ID of reader, 0 if no reader attached)\n\n## Metadata block\nMetadata is a structure that allows the writer to specify detailed information about the schema it will write. The reader might not know exactly what data to expect. The metadata block has the following structure:\n- 8B: Metadata size (excluding this size field)\n- Dynamic: Metadata content (e.g., protobuf serialized schema)\n\n## Use case:\n\nA Python docker container (reader) uses the zerobuffer to exchange data with streamer (writer) that would run in a different container on the same host machine. The python docker container would create zero-buffer and writes zero-buffer connection-string in json-rpc on std-out, that is started with '\u003e' char:\n\n\u003e { \"jsonrpc\": \"2.0\", \"method\": \"start-stream\", \"params\": [ \"e39640a7-50e4-4261-ac92-d9e09e480074\", 1024, 2048] }\n\nthe rocket-welder2 reads this from log and thus can run gstreamer with proper parameters passed to the pipeline.\n\ngst-launch-1.0 testvidoersrc ! zero-sink name=\"e39640a7-50e4-4261-ac92-d9e09e480074\"\n\nthe zero-sink plugin, would then write the metadata and than frames to the zero-buffer based on the passed name: \"e39640a7-50e4-4261-ac92-d9e09e480074\". \n\n## Reader preparation:\n\n- creates a file lock: `/tmp/zerobuffer/{name}.lock` (Linux) or appropriate temp directory (Windows)\n- checks if resources should be cleaned up - tries to delete any stale *.lock files, if successful, deletes correlated resources (shared memory, semaphores)\n- creates shared memory segment with identifier `${name}` (using POSIX shm_open on Linux, CreateFileMapping on Windows)\n- zeros memory and initializes OIEB structure\n- creates 2 named semaphores: `sem-w-${name}` and `sem-r-${name}` (using POSIX sem_open on Linux, CreateSemaphore on Windows)\n\n## The write flow:\n\n- The write client queries the size of metadata size and data size.\n\n- The client verifies there is enough free bytes in metadata block\n- The client writes metadata to shared memory. The metadata is prefixed with its size (protocol-level), can be then serialized via protobuf (application-level, not protocol level).\n\nGiven the client wants to write a frame (certain amount of bytes):\n- The client verifies how much continuous free space is available in the buffer by examining the OIEB\n- If there is not enough continuous space, it waits on the sem-r-${name} semaphore (waiting for reader to free space)\n- When semaphore is signaled, the check is performed again - until we have enough continuous free memory\n- The client writes the frame with its size prefix and sequence number\n- After writing is complete, the client updates OIEB counters (written bytes, write count)\n- The client signals sem-w-${name} to notify reader that new data is available \n\n## The read flow:\n\n- The reader waits on the sem-w-${name} semaphore (waiting for writer to signal new data)\n- When signaled, reader examines OIEB to determine data location and size\n- Reader reads the frame header (size + sequence number) and validates sequence continuity\n- Reader constructs a frame object that points to the shared buffer data (zero-copy)\n- When reader finishes processing the frame (which may be at a later time):\n  - Updates OIEB read count\n  - Updates OIEB free bytes by adding the consumed frame size\n  - Signals sem-r-${name} to notify writer that buffer space is available\n\n## Ring Buffer Wrap-Around Handling\n\nWhen writing a message of size N bytes:\n1. If (free_bytes \u003e= N \u0026\u0026 continuous_free_bytes \u003e= N):\n   - Write message at current write position\n2. If (free_bytes \u003e= N \u0026\u0026 continuous_free_bytes \u003c N):\n   - Write a wrap marker (FrameHeader with payload_size=0) at current position\n   - Update write position to beginning of buffer\n   - Write the actual message at the beginning\n   - Writer MUST NOT split messages across buffer boundary\n3. If (free_bytes \u003c N):\n   - Wait for reader to free more space\n\nWhere:\n- free_bytes = total free space in buffer (may be fragmented)\n- continuous_free_bytes = free space from write_pos to min(read_pos, buffer_end)\n\n### Wrap Marker Details\n\nA wrap marker is a special FrameHeader with:\n- `payload_size = 0` (indicates this is a wrap marker, not a real frame)\n- `sequence_number = 0` (wrap markers don't have sequence numbers)\n\nWhen the reader encounters a wrap marker:\n- It updates the read position to the beginning of the buffer\n- It reclaims the wasted space at the end of the buffer\n- It continues reading the actual frame from the beginning\n- It does NOT increment the expected sequence number\n\n### Semaphore Signaling with Wrap Markers\n\n**Important**: Wrap markers are implementation details, not logical frames.\n\n- **Writer**: Signals the write semaphore ONCE after writing a frame, even if it includes a wrap marker\n- **Reader**: Signals the read semaphore ONCE after reading a frame, even if it processed a wrap marker\n\nThis ensures symmetric semaphore behavior: one signal per logical frame on both sides.\n\n## Memory Ordering and Synchronization\n\nTo ensure correct visibility of data across processes:\n\n1. **Memory Barriers**: \n   - Writer uses memory_order_release when updating OIEB counters after writing data\n   - Reader uses memory_order_acquire when reading OIEB counters before reading data\n   - This ensures data written before counter update is visible after counter read\n\n2. **OIEB Update Order**:\n   - Writer: Update data → memory barrier → update written_bytes/count → signal semaphore\n   - Reader: Wait semaphore → read written_bytes/count → memory barrier → read data\n\n3. **Sequence Number Validation**:\n   - Reader verifies sequence numbers are consecutive\n   - Detection of corrupted/out-of-order data indicates memory corruption\n\n## Capacity Planning\n\n### Buffer Size Calculation\n\n**Basic Formula:**\n```\npayload_size = max_frame_size * 3 + overhead\noverhead = frames_in_flight * sizeof(FrameHeader)\n```\n\nWhere:\n- `max_frame_size`: Largest frame you'll write\n- `frames_in_flight`: Expected frames between write and read\n- `sizeof(FrameHeader)`: 16 bytes (8B size + 8B sequence)\n\n### Examples:\n\n**1080p @ 30 FPS Video (6MB frames):**\n```\nmax_frame_size = 6MB\nframes_in_flight = 3 (100ms buffer @ 30fps)\npayload_size = 6MB * 3 + 3 * 16 = 18MB\nRecommended: 20MB (with padding)\n```\n\n**4K @ 60 FPS Video (25MB frames):**\n```\nmax_frame_size = 25MB\nframes_in_flight = 6 (100ms buffer @ 60fps)\npayload_size = 25MB * 3 + 6 * 16 = 75MB\nRecommended: 80MB (with padding)\n```\n\n**Telemetry Data (1KB @ 1000Hz):**\n```\nmax_frame_size = 1KB\nframes_in_flight = 100 (100ms buffer @ 1000Hz)\npayload_size = 1KB * 3 + 100 * 16 = 4.6KB\nRecommended: 8KB (with padding)\n```\n\n### Rules of Thumb:\n- Use 3x your maximum frame size as baseline\n- Add 16 bytes per expected concurrent frame\n- Round up to power of 2 for alignment efficiency\n- Monitor buffer utilization; resize if consistently \u003e80%\n\n## Health Monitoring\n\nThe OIEB provides real-time buffer health information:\n\n```\nutilization = (payload_size - payload_free_bytes) / payload_size * 100%\n```\n\n- **Healthy**: \u003c 80% utilization\n- **Degraded**: 80-95% utilization (log warnings)\n- **Critical**: \u003e 95% utilization (imminent frame loss)\n\nApplications should monitor `payload_free_bytes` and log warnings when entering degraded state.\n\n## State Diagrams\n\n### Reader States\n```\n[Created] -\u003e [Waiting] \u003c-\u003e [Reading] -\u003e [Processing] -\u003e [Waiting]\n                 |                           |\n                 v                           v\n            [Error/Dead]                [Shutdown]\n```\n\n### Writer States  \n```\n[Connecting] -\u003e [Connected] -\u003e [Writing] \u003c-\u003e [Blocked] -\u003e [Writing]\n      |             |             |             |\n      v             v             v             v\n   [Error]    [No Reader]   [Reader Dead]  [Shutdown]\n```\n\n### Error Conditions\n- **Reader Dead**: Writer detects via PID check after 5s timeout\n- **Writer Dead**: Reader detects via PID check after 5s timeout\n- **No Reader**: Writer cannot find active reader on connect\n- **Buffer Full**: Writer blocks on sem-r until space available\n\n## Security Model\n\n**Note**: ZeroBuffer is designed for trusted internal use only. It provides:\n- No encryption of data\n- No authentication between processes\n- Security via filesystem permissions only\n- Same-user access by default\n\nFor secure IPC, consider encrypted alternatives. ZeroBuffer optimizes for lowest latency in trusted environments.\n\n## Documentation\n\n📚 **[See DOCUMENTATION_INDEX.md](DOCUMENTATION_INDEX.md) for complete documentation structure**\n\n### Quick Links\n- [Protocol Specification](PROTOCOL.md) - Technical protocol details\n- [API Reference](API.md) - Complete API for all languages  \n- [Test Scenarios](TEST_SCENARIOS.md) - Comprehensive test coverage\n- [Changelog](CHANGELOG.md) - Version history\n\n## Installation\n\n### C++ with vcpkg\n\nConfigure the custom registry in your `vcpkg-configuration.json`:\n```json\n{\n  \"registries\": [\n    {\n      \"kind\": \"git\",\n      \"repository\": \"https://github.com/modelingevolution/zerobuffer-vcpkg-registry\",\n      \"baseline\": \"YOUR_BASELINE_HERE\",\n      \"packages\": [\"zerobuffer\"]\n    }\n  ]\n}\n```\n\nThen install:\n```bash\nvcpkg install zerobuffer\n```\n\nOr use CMake with vcpkg toolchain:\n```cmake\nfind_package(zerobuffer CONFIG REQUIRED)\ntarget_link_libraries(your_target PRIVATE zerobuffer::zerobuffer)\n```\n\n### C# with NuGet\n\n[![NuGet](https://img.shields.io/nuget/v/ZeroBuffer.svg)](https://www.nuget.org/packages/ZeroBuffer/)\n\n```bash\n# Package Manager Console\nInstall-Package ZeroBuffer\n\n# .NET CLI\ndotnet add package ZeroBuffer\n\n# PackageReference in .csproj\n\u003cPackageReference Include=\"ZeroBuffer\" Version=\"1.0.*\" /\u003e\n```\n\n### Python with pip\n\n[![PyPI](https://img.shields.io/pypi/v/zerobuffer-ipc.svg)](https://pypi.org/project/zerobuffer-ipc/)\n\n```bash\n# Install from PyPI\npip install zerobuffer-ipc\n\n# Or install with specific version\npip install zerobuffer-ipc==1.0.0\n```\n\n## Quick Start\n\n### C++ Example\n```cpp\n#include \u003czerobuffer/writer.h\u003e\n#include \u003czerobuffer/reader.h\u003e\n\n// Writer\nzerobuffer::Writer writer(\"my-buffer\", 1024*1024, 64*1024);\nwriter.connect();\nwriter.write(data, size);\n\n// Reader\nzerobuffer::Reader reader(\"my-buffer\", 1024*1024, 64*1024);\nauto frame = reader.read();\n```\n\n### C# Example\n```csharp\nusing ZeroBuffer;\n\n// Writer\nusing var writer = new Writer(\"my-buffer\", 1024*1024, 64*1024);\nwriter.Connect();\nwriter.Write(data);\n\n// Reader\nusing var reader = new Reader(\"my-buffer\", 1024*1024, 64*1024);\nvar frame = reader.Read();\n```\n\n### Python Example\n```python\nimport zerobuffer\n\n# Writer\nwriter = zerobuffer.Writer(\"my-buffer\", 1024*1024, 64*1024)\nwriter.connect()\nwriter.write(data)\n\n# Reader\nreader = zerobuffer.Reader(\"my-buffer\", 1024*1024, 64*1024)\nframe = reader.read()\n```\n\n## Documentation\n\n- [API Documentation](API.md) - Detailed API reference for C++, C#, and Python\n- [Cross-Platform Tests](CROSS_PLATFORM_TESTS.md) - Testing strategy and standardized interfaces\n- [Benchmarking Guide](BENCHMARKING_GUIDE.md) - How to run performance benchmarks\n- [Test Coverage \u0026 Inconsistencies](TEST_COVERAGE_AND_INCONSISTENCIES.md) - Unit test analysis and known issues\n- [Improvements TODO](IMPROVEMENTS_TODO.md) - Planned enhancements and known issues\n- [C++ vcpkg Usage](cpp/VCPKG.md) - Using ZeroBuffer with vcpkg\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodelingevolution%2Fzerobuffer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmodelingevolution%2Fzerobuffer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodelingevolution%2Fzerobuffer/lists"}