{"id":41163009,"url":"https://github.com/dallison/subspace","last_synced_at":"2026-04-21T02:13:38.998Z","repository":{"id":169515711,"uuid":"637517481","full_name":"dallison/subspace","owner":"dallison","description":"Subspace IPC","archived":false,"fork":false,"pushed_at":"2026-03-18T02:36:04.000Z","size":2082,"stargazers_count":99,"open_issues_count":5,"forks_count":8,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-03-18T12:40:00.731Z","etag":null,"topics":["abseil-cpp","cplusplus","cplusplus-17","ipc","linux","macos","maxosx","protobuf","pubsub","robotics","robotics-operating-system","tcp","udp"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dallison.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":"COPYRIGHT","agents":null,"dco":null,"cla":null}},"created_at":"2023-05-07T19:47:00.000Z","updated_at":"2026-03-17T23:30:25.000Z","dependencies_parsed_at":"2023-07-04T11:01:17.726Z","dependency_job_id":"10945d76-46f6-4057-a0d4-61c06318d5b3","html_url":"https://github.com/dallison/subspace","commit_stats":null,"previous_names":["dallison/subspace"],"tags_count":47,"template":false,"template_full_name":null,"purl":"pkg:github/dallison/subspace","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dallison%2Fsubspace","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dallison%2Fsubspace/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dallison%2Fsubspace/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dallison%2Fsubspace/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dallison","download_url":"https://codeload.github.com/dallison/subspace/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dallison%2Fsubspace/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31290709,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["abseil-cpp","cplusplus","cplusplus-17","ipc","linux","macos","maxosx","protobuf","pubsub","robotics","robotics-operating-system","tcp","udp"],"created_at":"2026-01-22T19:26:24.740Z","updated_at":"2026-04-21T02:13:38.989Z","avatar_url":"https://github.com/dallison.png","language":"C++","readme":"# Subspace IPC\nNext Generation, sub-microsecond latency shared memory IPC.\n\nThis is a shared-memory based pub/sub Interprocess Communication system that can be used\nin robotics and other applications.  Why *subspace*?  If your messages are transported\nbetween processes on the same computer, they travel through extremely low latency\nand high bandwidth shared memory buffers, kind of like they are going\nfaster than light (not really, of course).  If they go between computers, they are\ntransported over the network at sub-light speed.\n\n## Acknowledgments\n\nSome of the code in this project was contributed by Cruise LLC.\n\n## Features\n\nIt has the following features:\n\n1.\tSingle threaded coroutine based server process written in C++17\n1.\tCoroutine-aware client library, in C++17.\n1.\tNative Rust client library with the same shared-memory performance as the C++ client.\n1.\tC client wrapper for easy integration into other language bindings.\n1.\tPublish/subscribe methodology with multiple publisher and multiple subscribers per channel.\n1.\tNo communication with server for message transfer.\n1.\tMessage type agnostic transmission – bring your own serialization.\n2.  Channel types, meaningful to user, not system.\n1.\tSingle lock POSIX shared memory channels\n1.\tBoth unreliable and reliable communications between publishers and subscribers.\n1.\tAbility to read the next or newest message in a channel.\n1.\tFile-descriptor-based event triggers.\n1.\tAutomatic UDP discovery and TCP bridging of channels between servers.\n1.\tShadow process for crash recovery -- the server can restart and resume without losing shared memory state.\n1.\tShared and weak pointers for message references.\n1.\tPorts to MacOS and Linux, ARM64 and x86_64.\n1.\tBuilds using Bazel and uses Abseil and Protocol Buffers from Google.\n1.\tUses my C++ coroutine library (https://github.com/dallison/co)\n\nSee the file docs/subspace.pdf for full documentation.  Additional documentation:\n- [Checksums and User Metadata](docs/checksums-and-metadata.md)\n- [Client Architecture](docs/client-architecture.md)\n- [Server Architecture](docs/server-architecture.md)\n- [Rust Client](docs/rust-client.md)\n- [Shadow Process (Crash Recovery)](docs/shadow-process.md)\n\n# Building\n\nSubspace can be built using either Bazel or CMake. Both build systems will automatically download and build all required dependencies.\n\n## Building with Bazel\n\nThis uses Google's Bazel to build.  You will need to download Bazel to build it.\nThe build also needs some external libraries, but Bazel takes care of downloading them.\nThe *.bazelrc* file contains some configuration options.\n\n### To build on Mac\n```\nbazel build --config=macos_arm64 ...    # Apple Silicon\nbazel build --config=macos_x86_64 ...   # Intel\n```\n\n### To build on Linux\nSubspace really wants to be built using *clang* but modern *GCC* versions work well too.  Depending on how your OS is configured, you\nmight need to tell bazel what compiler to use.\n\n```\nCC=clang bazel build ...\n```\n\n\n### Example: Ubuntu 20.04\nBuild a minimal set of binaries:\n\n```\nCC=clang bazel build //server:subspace_server //manual_tests:{pub,sub}\n```\n\nThen run each in a separate terminal:\n\n * `./bazel-bin/server/subspace_server`\n * `./bazel-bin/manual_tests/sub`\n * `./bazel-bin/manual_tests/pub`\n\n### Running Tests with Bazel\n\nYou can run tests directly using `bazel run` or `bazel test`. The `bazel run` command will build and execute the test in one step, while `bazel test` runs tests in test mode (useful for CI/CD).\n\n#### macOS: `xcrun` / `DEVELOPER_DIR` errors\n\nIf C++ compile actions fail with `xcrun: error: invalid DEVELOPER_DIR path (/Library/Developer/CommandLineTools), missing xcrun`, the active developer directory does not contain a usable toolchain (often an incomplete Command Line Tools install, or `DEVELOPER_DIR` set incorrectly in your shell, IDE, or CI).\n\n1. Prefer full **Xcode** and point the active developer dir at it:\n\n   ```bash\n   sudo xcode-select -s /Applications/Xcode.app/Contents/Developer\n   ```\n\n2. Or install/repair **Command Line Tools** so `xcrun` exists under that path:\n\n   ```bash\n   xcode-select --install\n   ```\n\n3. If you export `DEVELOPER_DIR` yourself (e.g. in `~/.zshrc`), remove it or set it to match `xcode-select -p`.\n\n4. Optional: after (1), you can force Bazel actions to use Xcode with a **user** `.bazelrc` line:\n\n   ```bash\n   build --action_env=DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer\n   ```\n\n   Adjust the path if Xcode is installed elsewhere.\n\n**Note:** All tests automatically start a subspace server in a separate thread, so you don't need to run the server separately. The tests handle server lifecycle management internally.\n\n#### client_test\n\nThe `client_test` is a comprehensive test suite that validates the core client functionality including publishers, subscribers, reliable/unreliable channels, message reading modes, and more.\n\n```bash\n# Run the test\nbazel run //client:client_test\n\n# Or run as a test (better for CI)\nbazel test //client:client_test\n```\n\n#### latency_test\n\nThe `latency_test` measures message transmission latency between publishers and subscribers. This is useful for benchmarking performance.\n\n```bash\n# Run the latency test\nbazel run //client:latency_test\n\n# Run with custom options (if supported)\nbazel run //client:latency_test -- --help\n```\n\n#### stress_test\n\nThe `stress_test` performs stress testing with high message rates and multiple publishers/subscribers to verify system stability under load.\n\n```bash\n# Run the stress test (may take a while)\nbazel run //client:stress_test\n\n# Or run as a test\nbazel test //client:stress_test\n```\n\n#### Rust Client Tests\n\nThe Rust client tests exercise the full Rust client API against a real C++ server\n(started in-process via FFI), including cross-language interoperability tests\nthat verify C++ publishers can talk to Rust subscribers and vice versa with\nchecksums and metadata.\n\n```bash\nbazel test //rust_client:client_test\n```\n\n#### Running All Tests\n\nTo run all tests at once:\n\n```bash\n# Run all tests\nbazel test //...\n\n# Run all tests in a specific directory\nbazel test //client/...\nbazel test //common/...\nbazel test //rust_client/...\n```\n\n## Building with CMake\n\nSubspace also supports building with CMake (version 3.15 or later). CMake uses FetchContent to automatically download and build all dependencies including Abseil, Protobuf, Googletest, cpp_toolbelt, and co.\n\n### Prerequisites\n\n- CMake 3.15 or later\n- C++17 compatible compiler (clang or g++)\n- Git (for fetching dependencies)\n\n### Basic Build\n\n```bash\nmkdir build\ncd build\ncmake ..\nmake\n```\n\n### Build Options\n\nYou can customize the build with CMake options:\n\n```bash\ncmake -DCMAKE_BUILD_TYPE=Release ..\nmake -j$(nproc)\n```\n\n### Running Tests\n\nAfter building, you can run the tests:\n\n```bash\ncd build\nctest\n```\n\nOr run individual tests:\n\n```bash\n./client/client_test\n./common/common_test\n```\n\n### Example: Building and Running\n\n```bash\n# Configure and build\nmkdir build \u0026\u0026 cd build\ncmake -DCMAKE_BUILD_TYPE=Release ..\nmake -j$(nproc)\n\n# Run the server in one terminal\n./server/subspace_server\n\n# Run publisher/subscriber examples in other terminals\n./client/latency_test\n./client/stress_test\n```\n\n### CMake Integration in Your Project\n\nTo use Subspace in your CMake project, you can add it as a subdirectory:\n\n```cmake\n# In your CMakeLists.txt\nadd_subdirectory(subspace)\ntarget_link_libraries(your_target\n    subspace_client\n    subspace_common\n    subspace_proto\n)\n```\n\nOr use FetchContent:\n\n```cmake\ninclude(FetchContent)\nFetchContent_Declare(\n    subspace\n    GIT_REPOSITORY https://github.com/dallison/subspace.git\n    GIT_TAG main  # or specific tag/commit\n)\nFetchContent_MakeAvailable(subspace)\n\ntarget_link_libraries(your_target\n    subspace_client\n    subspace_common\n    subspace_proto\n)\n```\n\n### CMake Build Targets\n\nThe CMake build provides the following targets:\n\n- `subspace_client` - Client library\n- `subspace_common` - Common utilities library\n- `subspace_proto` - Protocol buffer definitions library\n- `libserver` - Server library (includes shadow replicator)\n- `subspace_server` - Server executable\n- `shadow_lib` - Shadow process library\n- `subspace_shadow` - Shadow process executable\n- `subspace_client_rust` - Rust client library (built via Cargo; requires `cargo`)\n- `client_test`, `latency_test`, `stress_test` - C++ client test executables\n- `common_test` - Common library tests\n- `c_client_test` - C client tests\n- `shadow_test` - Shadow process tests\n- `rust_client_test` - Rust client tests (via `cargo test`; requires the server binary)\n\n# Bazel WORKSPACE\nAdd this to your Bazel WORKSPACE file to get access to this library without downloading it manually.\n\n```\nhttp_archive(\n  name = \"subspace\",\n  urls = [\"https://github.com/dallison/subspace/archive/refs/tags/A.B.C.tar.gz\"],\n  strip_prefix = \"subspace-A.B.C\",\n)\n\n```\n\nYou can also add a sha256 field to ensure a canonical build if you like.  Bazel\nwill tell you what to put in for the hash when you first build it.\n\n# Using Subspace\n\n## Overview\n\nSubspace provides a high-performance, shared-memory based publish/subscribe IPC system. Messages are transmitted through POSIX shared memory with sub-microsecond latency. The system supports both reliable and unreliable message delivery, allowing you to choose the appropriate semantics for your use case.\n\n## Client API\n\n### Creating a Client\n\nThe `Client` class is the main entry point for using Subspace. You can create a client in two ways:\n\n**Method 1: Using `Create()` (recommended)**\n```cpp\n#include \"client/client.h\"\n\nauto client_or = subspace::Client::Create(\"/tmp/subspace\", \"my_client\");\nif (!client_or.ok()) {\n    // Handle error\n    return;\n}\nauto client = client_or.value();\n```\n\n**Method 2: Constructor + Init()**\n```cpp\nsubspace::Client client;\nauto status = client.Init(\"/tmp/subspace\", \"my_client\");\nif (!status.ok()) {\n    // Handle error\n    return;\n}\n```\n\n**Parameters:**\n- `server_socket` (default: `\"/tmp/subspace\"`): Path to the Unix domain socket where the Subspace server is listening\n- `client_name` (default: `\"\"`): Optional name for this client instance\n- `c` (optional): Pointer to a coroutine if using coroutine-aware mode\n\n### Client Methods\n\n```cpp\nclass Client {\npublic:\n    // Initialize the client by connecting to the server\n    absl::Status Init(const std::string \u0026server_socket = \"/tmp/subspace\",\n                      const std::string \u0026client_name = \"\");\n\n    // Create a publisher for a channel\n    absl::StatusOr\u003cPublisher\u003e\n    CreatePublisher(const std::string \u0026channel_name, \n                    int slot_size, \n                    int num_slots,\n                    const PublisherOptions \u0026opts = PublisherOptions());\n\n    // Create a publisher with options specifying slot size and count\n    absl::StatusOr\u003cPublisher\u003e\n    CreatePublisher(const std::string \u0026channel_name,\n                    const PublisherOptions \u0026opts = PublisherOptions());\n\n    // Create a subscriber for a channel\n    absl::StatusOr\u003cSubscriber\u003e\n    CreateSubscriber(const std::string \u0026channel_name,\n                     const SubscriberOptions \u0026opts = SubscriberOptions());\n\n    // Get information about channels\n    absl::StatusOr\u003cconst ChannelInfo\u003e GetChannelInfo(const std::string \u0026channelName);\n    absl::StatusOr\u003cconst std::vector\u003cChannelInfo\u003e\u003e GetChannelInfo();\n    \n    absl::StatusOr\u003cconst ChannelStats\u003e GetChannelStats(const std::string \u0026channelName);\n    absl::StatusOr\u003cbool\u003e ChannelExists(const std::string \u0026channelName);\n\n    // Enable/disable debug output\n    void SetDebug(bool v);\n    \n    // Enable/disable thread-safe mode\n    void SetThreadSafe(bool v);\n};\n```\n\n### Convenience Free Functions\n\nFor simple use cases where you don't need to reuse a client, Subspace provides\nfree functions that create a temporary client, perform the operation, and return\na standalone `Publisher` or `Subscriber`. The returned object keeps the\nunderlying client alive for its lifetime.\n\n```cpp\n#include \"client/client.h\"\n\n// Create a publisher without managing a Client object.\nauto pub_or = subspace::CreatePublisher(\"my_channel\",\n    subspace::PublisherOptions{.slot_size = 1024, .num_slots = 10});\nif (!pub_or.ok()) {\n    // Handle error\n    return;\n}\nauto pub = std::move(*pub_or);\n\n// Create a subscriber without managing a Client object.\nauto sub_or = subspace::CreateSubscriber(\"my_channel\");\nif (!sub_or.ok()) {\n    // Handle error\n    return;\n}\nauto sub = std::move(*sub_or);\n```\n\n**Parameters:**\n- `channel_name`: Name of the channel\n- `opts`: `PublisherOptions` or `SubscriberOptions` (defaults apply)\n- `server_socket` (default: `\"/tmp/subspace\"`): Path to the server socket\n- `client_name` (default: `\"\"`): Optional client name\n- `c` (optional): Coroutine pointer for coroutine-aware mode\n\n## Publisher API\n\n### Creating a Publisher\n\nPublishers send messages to channels. You can create a publisher in two ways:\n\n**Method 1: Explicit slot size and count**\n```cpp\nauto pub_or = client-\u003eCreatePublisher(\"my_channel\", 1024, 10);\nif (!pub_or.ok()) {\n    // Handle error\n    return;\n}\nauto pub = pub_or.value();\n```\n\n**Method 2: Using PublisherOptions**\n```cpp\nauto pub_or = client-\u003eCreatePublisher(\"my_channel\", \n    subspace::PublisherOptions()\n        .SetSlotSize(1024)\n        .SetNumSlots(10)\n        .SetReliable(true));\n```\n\n### Publishing Messages\n\n```cpp\n// Get a message buffer\nauto buffer_or = pub.GetMessageBuffer(1024);\nif (!buffer_or.ok()) {\n    // Handle error (e.g., no free slots for reliable publisher)\n    return;\n}\nvoid* buffer = buffer_or.value();\n\n// Fill in your message data\nMyMessageType* msg = reinterpret_cast\u003cMyMessageType*\u003e(buffer);\nmsg-\u003efield1 = 42;\nmsg-\u003efield2 = \"hello\";\n\n// Publish the message\nauto msg_info_or = pub.PublishMessage(sizeof(MyMessageType));\nif (!msg_info_or.ok()) {\n    // Handle error\n    return;\n}\nauto msg_info = msg_info_or.value();\n// msg_info.ordinal contains the message sequence number\n// msg_info.timestamp contains the publish timestamp\n```\n\n**Using GetMessageBufferSpan (C++17 style):**\n```cpp\nauto span_or = pub.GetMessageBufferSpan(1024);\nif (!span_or.ok() || span_or.value().empty()) {\n    // Handle error\n    return;\n}\nauto span = span_or.value();\n// span is an absl::Span\u003cstd::byte\u003e\nMyMessageType* msg = reinterpret_cast\u003cMyMessageType*\u003e(span.data());\n// ... fill message ...\npub.PublishMessage(sizeof(MyMessageType));\n```\n\n### Publisher Methods\n\n```cpp\nclass Publisher {\npublic:\n    // Get a message buffer for writing\n    absl::StatusOr\u003cvoid*\u003e GetMessageBuffer(int32_t max_size = -1, bool lock = true);\n    absl::StatusOr\u003cabsl::Span\u003cstd::byte\u003e\u003e GetMessageBufferSpan(int32_t max_size = -1, bool lock = true);\n    \n    // Publish a message\n    absl::StatusOr\u003cconst Message\u003e PublishMessage(int64_t message_size);\n    \n    // Cancel a publish (releases lock in thread-safe mode)\n    void CancelPublish();\n    \n    // Wait for a reliable publisher to have a free slot\n    absl::Status Wait(const co::Coroutine *c = nullptr);\n    absl::Status Wait(std::chrono::nanoseconds timeout, const co::Coroutine *c = nullptr);\n    absl::StatusOr\u003cint\u003e Wait(const toolbelt::FileDescriptor \u0026fd, const co::Coroutine *c = nullptr);\n    \n    // Get file descriptor for polling\n    struct pollfd GetPollFd() const;\n    toolbelt::FileDescriptor GetFileDescriptor() const;\n    const toolbelt::FileDescriptor\u0026 GetRetirementFd() const;\n    \n    // Channel information\n    std::string Name() const;\n    std::string Type() const;\n    bool IsReliable() const;\n    bool IsLocal() const;\n    bool IsFixedSize() const;\n    int32_t SlotSize() const;\n    int32_t NumSlots() const;\n    \n    // Statistics\n    void GetStatsCounters(uint64_t \u0026total_bytes, uint64_t \u0026total_messages,\n                          uint32_t \u0026max_message_size, uint32_t \u0026total_drops);\n    \n    // Resize callback registration\n    absl::Status RegisterResizeCallback(\n        std::function\u003cabsl::Status(Publisher*, int, int)\u003e callback);\n\n    // Prefix area and checksum/metadata sizes\n    int32_t PrefixSize() const;     // Total prefix bytes (multiple of 64)\n    int32_t ChecksumSize() const;   // Bytes reserved for checksum\n    int32_t MetadataSize() const;   // Bytes of user metadata\n\n    // Writable span over the user metadata area in the current slot's prefix.\n    // Call between GetMessageBuffer() and PublishMessage().\n    absl::Span\u003cstd::byte\u003e GetMetadata();\n\n    // Custom checksum support\n    void SetChecksumCallback(ChecksumCallback cb);\n    void ResetChecksumCallback();\n};\n```\n\n### Reliable Publisher Example\n\n```cpp\n// Create a reliable publisher\nauto pub_or = client-\u003eCreatePublisher(\"reliable_channel\", 256, 5,\n    subspace::PublisherOptions().SetReliable(true));\n\nauto pub = pub_or.value();\n\nwhile (true) {\n    // Wait for a free slot (blocks until available)\n    auto status = pub.Wait();\n    if (!status.ok()) {\n        // Handle error\n        break;\n    }\n    \n    // Get message buffer\n    auto buffer_or = pub.GetMessageBuffer(256);\n    if (!buffer_or.ok()) {\n        continue; // Should not happen after Wait()\n    }\n    \n    // Fill and publish\n    MyMessage* msg = reinterpret_cast\u003cMyMessage*\u003e(buffer_or.value());\n    msg-\u003edata = compute_data();\n    pub.PublishMessage(sizeof(MyMessage));\n}\n```\n\n## Subscriber API\n\n### Creating a Subscriber\n\n```cpp\nauto sub_or = client-\u003eCreateSubscriber(\"my_channel\");\nif (!sub_or.ok()) {\n    // Handle error\n    return;\n}\nauto sub = sub_or.value();\n```\n\n### Reading Messages\n\n**Method 1: Read next message**\n```cpp\nauto msg_or = sub.ReadMessage(subspace::ReadMode::kReadNext);\nif (!msg_or.ok()) {\n    // Handle error\n    return;\n}\nauto msg = msg_or.value();\nif (msg.length == 0) {\n    // No message available\n    return;\n}\n// msg.buffer points to the message data\n// msg.length is the message size in bytes\n// msg.ordinal is the sequence number\n// msg.timestamp is the publish timestamp\nconst MyMessageType* data = reinterpret_cast\u003cconst MyMessageType*\u003e(msg.buffer);\n```\n\n**Method 2: Read newest message**\n```cpp\nauto msg_or = sub.ReadMessage(subspace::ReadMode::kReadNewest);\n// This skips to the most recent message, discarding older ones\n```\n\n**Method 3: Typed read (returns shared_ptr)**\n```cpp\nauto msg_ptr_or = sub.ReadMessage\u003cMyMessageType\u003e();\nif (!msg_ptr_or.ok() || !msg_ptr_or.value()) {\n    // No message or error\n    return;\n}\nauto msg_ptr = msg_ptr_or.value();\n// msg_ptr is a subspace::shared_ptr\u003cMyMessageType\u003e\n// Access data: msg_ptr-\u003efield1, (*msg_ptr).field2\n// Message is automatically released when msg_ptr goes out of scope\n```\n\n### Waiting for Messages\n\n```cpp\n// Wait indefinitely\nauto status = sub.Wait();\nif (!status.ok()) {\n    // Handle error\n    return;\n}\n\n// Wait with timeout\nauto status = sub.Wait(std::chrono::milliseconds(100));\nif (status.code() == absl::StatusCode::kDeadlineExceeded) {\n    // Timeout\n}\n\n// Wait with file descriptor (for integration with event loops)\ntoolbelt::FileDescriptor fd = /* your fd */;\nauto fd_or = sub.Wait(fd);\nif (fd_or.ok()) {\n    int triggered_fd = fd_or.value();\n    // Process message\n}\n```\n\n### Subscriber Methods\n\n```cpp\nclass Subscriber {\npublic:\n    // Read messages\n    absl::StatusOr\u003cMessage\u003e ReadMessage(ReadMode mode = ReadMode::kReadNext);\n    template \u003ctypename T\u003e\n    absl::StatusOr\u003cshared_ptr\u003cT\u003e\u003e ReadMessage(ReadMode mode = ReadMode::kReadNext);\n    \n    // Find message by timestamp\n    absl::StatusOr\u003cMessage\u003e FindMessage(uint64_t timestamp);\n    template \u003ctypename T\u003e\n    absl::StatusOr\u003cshared_ptr\u003cT\u003e\u003e FindMessage(uint64_t timestamp);\n    \n    // Wait for messages\n    absl::Status Wait(const co::Coroutine *c = nullptr);\n    absl::Status Wait(std::chrono::nanoseconds timeout, const co::Coroutine *c = nullptr);\n    absl::StatusOr\u003cint\u003e Wait(const toolbelt::FileDescriptor \u0026fd, const co::Coroutine *c = nullptr);\n    \n    // Get file descriptor for polling\n    struct pollfd GetPollFd() const;\n    toolbelt::FileDescriptor GetFileDescriptor() const;\n    \n    // Channel information\n    std::string Name() const;\n    std::string Type() const;\n    bool IsReliable() const;\n    int32_t SlotSize() const;\n    int32_t NumSlots() const;\n    int64_t GetCurrentOrdinal() const;\n    \n    // Callbacks\n    absl::Status RegisterDroppedMessageCallback(\n        std::function\u003cvoid(Subscriber*, int64_t)\u003e callback);\n    absl::Status RegisterMessageCallback(\n        std::function\u003cvoid(Subscriber*, Message)\u003e callback);\n    absl::Status ProcessAllMessages(ReadMode mode = ReadMode::kReadNext);\n    \n    // Statistics\n    const ChannelCounters\u0026 GetChannelCounters();\n    int NumActiveMessages() const;\n\n    // Prefix area and checksum/metadata sizes\n    int32_t PrefixSize() const;     // Total prefix bytes (multiple of 64)\n    int32_t ChecksumSize() const;   // Bytes reserved for checksum\n    int32_t MetadataSize() const;   // Bytes of user metadata\n\n    // Read-only span over the user metadata area in the most recently\n    // read message's prefix.  Valid while the message is active.\n    absl::Span\u003cconst std::byte\u003e GetMetadata();\n\n    // Custom checksum support\n    void SetChecksumCallback(ChecksumCallback cb);\n    void ResetChecksumCallback();\n};\n```\n\n### Subscriber Example with Callbacks\n\n```cpp\nauto sub_or = client-\u003eCreateSubscriber(\"my_channel\",\n    subspace::SubscriberOptions().SetReliable(true));\n\nauto sub = sub_or.value();\n\n// Register callback for dropped messages\nsub.RegisterDroppedMessageCallback([](subspace::Subscriber* sub, int64_t count) {\n    std::cerr \u003c\u003c \"Dropped \" \u003c\u003c count \u003c\u003c \" messages on \" \u003c\u003c sub-\u003eName() \u003c\u003c std::endl;\n});\n\n// Register callback for received messages\nsub.RegisterMessageCallback([](subspace::Subscriber* sub, subspace::Message msg) {\n    if (msg.length \u003e 0) {\n        process_message(msg);\n    }\n});\n\n// In your event loop\nwhile (true) {\n    // Process all available messages\n    sub.ProcessAllMessages();\n    \n    // Or wait and read manually\n    sub.Wait();\n    auto msg = sub.ReadMessage();\n    if (msg.ok() \u0026\u0026 msg-\u003elength \u003e 0) {\n        process_message(*msg);\n    }\n}\n```\n\n## Reliable vs Unreliable Channels\n\n### Reliable Channels\n\nReliable channels guarantee that **reliable subscribers** will never miss a message from **reliable publishers**. This is achieved through reference counting: a reliable publisher cannot reuse a slot until all reliable subscribers have released it.\n\n**Characteristics:**\n- Messages are never dropped for reliable subscribers\n- Publishers may block if all slots are in use\n- Higher memory usage (slots held until all subscribers release)\n- Use `Wait()` to block until a slot is available\n\n**When to use:**\n- Critical data that must not be lost\n- Control messages\n- State synchronization\n- Any scenario where message loss is unacceptable\n\n**Example:**\n```cpp\n// Reliable publisher\nauto pub = client-\u003eCreatePublisher(\"control\", 128, 10,\n    subspace::PublisherOptions().SetReliable(true)).value();\n\n// Reliable subscriber\nauto sub = client-\u003eCreateSubscriber(\"control\",\n    subspace::SubscriberOptions().SetReliable(true)).value();\n```\n\n### Unreliable Channels\n\nUnreliable channels provide best-effort delivery with no guarantees. If a subscriber cannot keep up, messages may be dropped. This provides the lowest latency and highest throughput.\n\n**Characteristics:**\n- Messages may be dropped if subscriber is slow\n- Publishers never block (always get a slot immediately)\n- Lower memory usage\n- Highest performance\n\n**When to use:**\n- High-frequency sensor data where occasional loss is acceptable\n- Video/audio streaming\n- Telemetry data\n- Any scenario where latency is more important than reliability\n\n**Example:**\n```cpp\n// Unreliable publisher (default)\nauto pub = client-\u003eCreatePublisher(\"sensor_data\", 64, 100).value();\n\n// Unreliable subscriber (default)\nauto sub = client-\u003eCreateSubscriber(\"sensor_data\").value();\n```\n\n### Mixed Reliability\n\nYou can mix reliable and unreliable publishers/subscribers on the same channel:\n- **Reliable subscriber + Reliable publisher**: Guaranteed delivery\n- **Reliable subscriber + Unreliable publisher**: Best effort (may drop)\n- **Unreliable subscriber + Reliable publisher**: May drop if slow\n- **Unreliable subscriber + Unreliable publisher**: Best effort, may drop\n\n## PublisherOptions\n\nThe `PublisherOptions` struct configures publisher behavior. You can use it in two ways:\n\n### Method 1: Chained Setters (Fluent API)\n\n```cpp\nauto opts = subspace::PublisherOptions()\n    .SetSlotSize(1024)\n    .SetNumSlots(10)\n    .SetReliable(true)\n    .SetLocal(false)\n    .SetType(\"MyMessageType\")\n    .SetFixedSize(false)\n    .SetChecksum(true);\n\nauto pub = client-\u003eCreatePublisher(\"channel\", opts).value();\n```\n\n### Method 2: Designated Initializer (C++20)\n\n```cpp\nauto pub = client-\u003eCreatePublisher(\"channel\", \n    subspace::PublisherOptions{\n        .slot_size = 1024,\n        .num_slots = 10,\n        .reliable = true,\n        .local = false,\n        .type = \"MyMessageType\",\n        .fixed_size = false,\n        .checksum = true,\n        .checksum_size = 4,   // default CRC32\n        .metadata_size = 0,   // no user metadata\n    }).value();\n```\n\n### PublisherOptions Fields and Methods\n\n| Field/Method | Type | Default | Description |\n|--------------|------|---------|-------------|\n| `slot_size` / `SetSlotSize()` | `int32_t` | `0` | Size of each message slot in bytes. Must be set if using options-only CreatePublisher. |\n| `num_slots` / `SetNumSlots()` | `int32_t` | `0` | Number of slots in the channel. Must be set if using options-only CreatePublisher. |\n| `reliable` / `SetReliable()` | `bool` | `false` | If true, reliable delivery (see Reliable Channels section). |\n| `local` / `SetLocal()` | `bool` | `false` | If true, messages are only visible on the local machine (not bridged). |\n| `type` / `SetType()` | `std::string` | `\"\"` | User-defined message type identifier. All publishers/subscribers must use the same type. |\n| `fixed_size` / `SetFixedSize()` | `bool` | `false` | If true, prevents automatic resizing of slots. |\n| `bridge` / `SetBridge()` | `bool` | `false` | Internal: marks this as a bridge publisher. |\n| `mux` / `SetMux()` | `std::string` | `\"\"` | Multiplexer name for virtual channels. |\n| `vchan_id` / `SetVchanId()` | `int` | `-1` | Virtual channel ID (-1 for server-assigned). |\n| `activate` / `SetActivate()` | `bool` | `false` | If true, channel is activated even if unreliable. |\n| `notify_retirement` / `SetNotifyRetirement()` | `bool` | `false` | If true, notify when slots are retired. |\n| `checksum` / `SetChecksum()` | `bool` | `false` | If true, calculate checksums for all messages. |\n| `checksum_size` / `SetChecksumSize()` | `int32_t` | `4` | Number of bytes reserved for the checksum (starting at the `checksum` field of `MessagePrefix`). Default 4 for CRC32. Increase for larger checksums (e.g. 20 for SHA-1). |\n| `metadata_size` / `SetMetadataSize()` | `int32_t` | `0` | Number of bytes of user metadata stored immediately after the checksum area. Accessible via `Publisher::GetMetadata()` / `Subscriber::GetMetadata()`. |\n\n**Getter Methods:**\n- `int32_t SlotSize() const`\n- `int32_t NumSlots() const`\n- `bool IsReliable() const`\n- `bool IsLocal() const`\n- `bool IsFixedSize() const`\n- `const std::string\u0026 Type() const`\n- `bool IsBridge() const`\n- `const std::string\u0026 Mux() const`\n- `int VchanId() const`\n- `bool Activate() const`\n- `bool NotifyRetirement() const`\n- `bool Checksum() const`\n- `int32_t ChecksumSize() const`\n- `int32_t MetadataSize() const`\n\n**Example: Creating a reliable publisher with checksums**\n```cpp\nauto pub = client-\u003eCreatePublisher(\"secure_channel\", 512, 20,\n    subspace::PublisherOptions()\n        .SetReliable(true)\n        .SetChecksum(true)\n        .SetType(\"SecureMessage\")).value();\n```\n\n### Checksums and the Prefix Area\n\nEach message slot has a prefix area preceding the message buffer.  The prefix\ncontains the `MessagePrefix` struct (ordinal, timestamp, size, flags, etc.) whose\n`checksum` field marks the start of the checksum storage.  The total prefix\narea is always aligned to a 64-byte boundary and its size is determined by:\n\n```\nprefix_size = align_up(offsetof(MessagePrefix, checksum) + checksum_size + metadata_size, 64)\n```\n\nWith the defaults (`checksum_size = 4`, `metadata_size = 0`) the prefix area\nis 64 bytes — the `MessagePrefix` itself.  Increasing either value causes the\nprefix to grow in 64-byte increments.\n\nWhen checksums are enabled (`SetChecksum(true)`), the built-in CRC32 writes a\n4-byte checksum at the start of the checksum area.  If you need a larger\nchecksum (e.g. 20 bytes for SHA-1), set `checksum_size` accordingly:\n\n```cpp\nauto pub = client-\u003eCreatePublisher(\"channel\", \n    subspace::PublisherOptions()\n        .SetSlotSize(1024)\n        .SetNumSlots(10)\n        .SetChecksum(true)\n        .SetChecksumSize(20)).value();\n```\n\n### User Metadata\n\nThe metadata area sits immediately after the checksum area in the prefix.\nSet `metadata_size` on the publisher to reserve space:\n\n```cpp\nauto pub = client-\u003eCreatePublisher(\"channel\",\n    subspace::PublisherOptions()\n        .SetSlotSize(1024)\n        .SetNumSlots(10)\n        .SetMetadataSize(16)).value();\n\n// Write metadata between GetMessageBuffer() and PublishMessage():\nauto buffer = pub.GetMessageBuffer(128).value();\nauto meta = pub.GetMetadata();  // absl::Span\u003cstd::byte\u003e, 16 bytes\nmemcpy(meta.data(), my_metadata, 16);\npub.PublishMessage(128);\n```\n\nSubscribers read metadata from the most recently read message:\n\n```cpp\nauto msg = sub.ReadMessage().value();\nauto meta = sub.GetMetadata();  // absl::Span\u003cconst std::byte\u003e, 16 bytes\n```\n\n### Checksum Callbacks\n\nIf you need a custom checksum algorithm, you can provide a callback that\nreplaces the built-in CRC32.  The callback receives the data to checksum\n(as three spans covering the prefix header, the prefix extension, and the\nmessage body) plus a writable\n`absl::Span\u003cstd::byte\u003e` of `ChecksumSize()` bytes where it should write\nthe result:\n\n```cpp\nusing ChecksumCallback =\n    std::function\u003cvoid(const std::array\u003cabsl::Span\u003cconst uint8_t\u003e, 3\u003e \u0026data,\n                       absl::Span\u003cstd::byte\u003e checksum)\u003e;\n```\n\n**Example: Simple additive checksum (4 bytes)**\n```cpp\nauto fake_crc = [](const std::array\u003cabsl::Span\u003cconst uint8_t\u003e, 3\u003e \u0026data,\n                   absl::Span\u003cstd::byte\u003e checksum) {\n    uint32_t sum = 0;\n    for (const auto \u0026span : data) {\n        for (uint8_t byte : span) {\n            sum += byte;\n        }\n    }\n    *reinterpret_cast\u003cuint32_t *\u003e(checksum.data()) = sum;\n};\n\npub.SetChecksumCallback(fake_crc);\nsub.SetChecksumCallback(fake_crc);\n```\n\n**Example: 20-byte custom checksum (requires `checksum_size = 20`)**\n```cpp\nauto sha1_like = [](const std::array\u003cabsl::Span\u003cconst uint8_t\u003e, 3\u003e \u0026data,\n                    absl::Span\u003cstd::byte\u003e checksum) {\n    // checksum.size() == 20\n    // Write your 20-byte digest into checksum.data()\n    my_sha1(data, checksum.data(), checksum.size());\n};\n\npub.SetChecksumCallback(sha1_like);\nsub.SetChecksumCallback(sha1_like);\n```\n\nCall `ResetChecksumCallback()` to revert to the built-in CRC32.\n\n## SubscriberOptions\n\nThe `SubscriberOptions` struct configures subscriber behavior. Like `PublisherOptions`, it supports both chained setters and designated initializers.\n\n### Method 1: Chained Setters\n\n```cpp\nauto opts = subspace::SubscriberOptions()\n    .SetReliable(true)\n    .SetType(\"MyMessageType\")\n    .SetMaxActiveMessages(10)\n    .SetChecksum(true)\n    .SetPassChecksumErrors(false);\n\nauto sub = client-\u003eCreateSubscriber(\"channel\", opts).value();\n```\n\n### Method 2: Designated Initializer\n\n```cpp\nauto sub = client-\u003eCreateSubscriber(\"channel\",\n    subspace::SubscriberOptions{\n        .reliable = true,\n        .type = \"MyMessageType\",\n        .max_active_messages = 10,\n        .checksum = true,\n        .pass_checksum_errors = false\n    }).value();\n```\n\n### SubscriberOptions Fields and Methods\n\n| Field/Method | Type | Default | Description |\n|--------------|------|---------|-------------|\n| `reliable` / `SetReliable()` | `bool` | `false` | If true, reliable delivery (see Reliable Channels section). |\n| `type` / `SetType()` | `std::string` | `\"\"` | User-defined message type identifier. Must match publisher type. |\n| `max_active_messages` / `SetMaxActiveMessages()` | `int` | `1` | Maximum number of active messages (shared_ptrs) that can be held simultaneously. |\n| `max_active_messages` / `SetMaxSharedPtrs()` | `int` | `0` | Alias: sets max_active_messages to n+1. |\n| `log_dropped_messages` / `SetLogDroppedMessages()` | `bool` | `true` | If true, log when messages are dropped. |\n| `bridge` / `SetBridge()` | `bool` | `false` | Internal: marks this as a bridge subscriber. |\n| `mux` / `SetMux()` | `std::string` | `\"\"` | Multiplexer name for virtual channels. |\n| `vchan_id` / `SetVchanId()` | `int` | `-1` | Virtual channel ID (-1 for server-assigned). |\n| `pass_activation` / `SetPassActivation()` | `bool` | `false` | If true, activation messages are passed to the user. |\n| `read_write` / `SetReadWrite()` | `bool` | `false` | If true, map buffers as read-write instead of read-only. |\n| `checksum` / `SetChecksum()` | `bool` | `false` | If true, verify checksums on received messages. |\n| `pass_checksum_errors` / `SetPassChecksumErrors()` | `bool` | `false` | If true, pass messages with checksum errors (with flag set). If false, return error. |\n\n**Getter Methods:**\n- `bool IsReliable() const`\n- `const std::string\u0026 Type() const`\n- `int MaxActiveMessages() const`\n- `int MaxSharedPtrs() const`\n- `bool LogDroppedMessages() const`\n- `bool IsBridge() const`\n- `const std::string\u0026 Mux() const`\n- `int VchanId() const`\n- `bool PassActivation() const`\n- `bool ReadWrite() const`\n- `bool Checksum() const`\n- `bool PassChecksumErrors() const`\n\n**Example: Creating a reliable subscriber with checksum verification**\n```cpp\nauto sub = client-\u003eCreateSubscriber(\"secure_channel\",\n    subspace::SubscriberOptions()\n        .SetReliable(true)\n        .SetChecksum(true)\n        .SetPassChecksumErrors(false)  // Return error on checksum failure\n        .SetType(\"SecureMessage\")\n        .SetMaxActiveMessages(5)).value();\n```\n\n## Complete Example\n\nHere's a complete example showing publisher and subscriber:\n\n```cpp\n#include \"client/client.h\"\n#include \u003ciostream\u003e\n\nstruct SensorData {\n    double temperature;\n    double pressure;\n    uint64_t timestamp;\n};\n\nint main() {\n    // Create client\n    auto client_or = subspace::Client::Create(\"/tmp/subspace\", \"sensor_app\");\n    if (!client_or.ok()) {\n        std::cerr \u003c\u003c \"Failed to create client: \" \u003c\u003c client_or.status() \u003c\u003c std::endl;\n        return 1;\n    }\n    auto client = client_or.value();\n\n    // Create reliable publisher\n    auto pub_or = client-\u003eCreatePublisher(\"sensors\", sizeof(SensorData), 10,\n        subspace::PublisherOptions()\n            .SetReliable(true)\n            .SetType(\"SensorData\"));\n    if (!pub_or.ok()) {\n        std::cerr \u003c\u003c \"Failed to create publisher: \" \u003c\u003c pub_or.status() \u003c\u003c std::endl;\n        return 1;\n    }\n    auto pub = pub_or.value();\n\n    // Create reliable subscriber\n    auto sub_or = client-\u003eCreateSubscriber(\"sensors\",\n        subspace::SubscriberOptions()\n            .SetReliable(true)\n            .SetType(\"SensorData\"));\n    if (!sub_or.ok()) {\n        std::cerr \u003c\u003c \"Failed to create subscriber: \" \u003c\u003c sub_or.status() \u003c\u003c std::endl;\n        return 1;\n    }\n    auto sub = sub_or.value();\n\n    // Publisher loop\n    for (int i = 0; i \u003c 100; ++i) {\n        // Wait for free slot\n        pub.Wait();\n        \n        // Get buffer\n        auto buffer_or = pub.GetMessageBuffer(sizeof(SensorData));\n        if (!buffer_or.ok()) continue;\n        \n        // Fill message\n        SensorData* data = reinterpret_cast\u003cSensorData*\u003e(buffer_or.value());\n        data-\u003etemperature = 20.0 + i * 0.1;\n        data-\u003epressure = 1013.25;\n        data-\u003etimestamp = std::chrono::steady_clock::now().time_since_epoch().count();\n        \n        // Publish\n        auto msg_or = pub.PublishMessage(sizeof(SensorData));\n        if (msg_or.ok()) {\n            std::cout \u003c\u003c \"Published message \" \u003c\u003c msg_or-\u003eordinal \u003c\u003c std::endl;\n        }\n    }\n\n    // Subscriber loop\n    for (int i = 0; i \u003c 100; ++i) {\n        // Wait for message\n        sub.Wait();\n        \n        // Read message\n        auto msg_or = sub.ReadMessage\u003cSensorData\u003e();\n        if (!msg_or.ok() || !msg_or.value()) {\n            continue;\n        }\n        \n        auto msg = msg_or.value();\n        std::cout \u003c\u003c \"Received: temp=\" \u003c\u003c msg-\u003etemperature \n                  \u003c\u003c \", pressure=\" \u003c\u003c msg-\u003epressure \n                  \u003c\u003c \", ordinal=\" \u003c\u003c msg.GetMessage().ordinal \u003c\u003c std::endl;\n    }\n\n    return 0;\n}\n```\n\n## C Client Interface\n\nSubspace provides a C API (`c_client/subspace.h`) for applications that need to use Subspace from C code or integrate it into other language bindings. The C API is simpler and has fewer dependencies than the C++ API, making it easier to integrate into projects that don't use C++.\n\n### Error Handling\n\nThe C API uses a thread-local error mechanism similar to `errno`. Most functions return a boolean indicating success (`true`) or failure (`false`). When a function fails, you can check for errors and retrieve the error message:\n\n```c\n#include \"c_client/subspace.h\"\n\n// Check if there was an error\nif (subspace_has_error()) {\n    // Get the error message\n    char* error = subspace_get_last_error();\n    fprintf(stderr, \"Error: %s\\n\", error);\n}\n```\n\nThe error message is a static string owned by the library and is thread-local (one error message per thread).\n\n### Creating a Client\n\n```c\n// Create client with default socket (\"/tmp/subspace\") and no name\nSubspaceClient client = subspace_create_client();\n\n// Create client with custom socket\nSubspaceClient client = subspace_create_client_with_socket(\"/tmp/my_subspace\");\n\n// Create client with socket and name\nSubspaceClient client = subspace_create_client_with_socket_and_name(\n    \"/tmp/subspace\", \"my_client_name\");\n\n// Check if client was created successfully\nif (client.client == NULL) {\n    fprintf(stderr, \"Failed to create client: %s\\n\", subspace_get_last_error());\n    return 1;\n}\n\n// Clean up when done\nsubspace_remove_client(\u0026client);\n```\n\n### Creating Publishers and Subscribers\n\n**Publisher Options:**\n\n```c\n// Create default publisher options\nSubspacePublisherOptions pub_opts = subspace_publisher_options_default(1024, 10);\n// pub_opts.slot_size = 1024\n// pub_opts.num_slots = 10\n// pub_opts.reliable = false\n// pub_opts.fixed_size = false\n// pub_opts.activate = false\n// pub_opts.checksum_size = 4   (CRC32)\n// pub_opts.metadata_size = 0   (no user metadata)\n\n// Customize options\npub_opts.reliable = true;\npub_opts.fixed_size = false;\npub_opts.type.type = \"MyMessageType\";\npub_opts.type.type_length = strlen(pub_opts.type.type);\npub_opts.checksum_size = 20;   // e.g. 20-byte digest\npub_opts.metadata_size = 32;   // 32 bytes of user metadata\n\n// Create publisher\nSubspacePublisher pub = subspace_create_publisher(client, \"my_channel\", pub_opts);\nif (pub.publisher == NULL) {\n    fprintf(stderr, \"Failed to create publisher: %s\\n\", subspace_get_last_error());\n    return 1;\n}\n```\n\n**Subscriber Options:**\n\n```c\n// Create default subscriber options\nSubspaceSubscriberOptions sub_opts = subspace_subscriber_options_default();\n// sub_opts.reliable = false\n// sub_opts.max_active_messages = 1\n// sub_opts.pass_activation = false\n// sub_opts.log_dropped_messages = false\n\n// Customize options\nsub_opts.reliable = true;\nsub_opts.max_active_messages = 10;\nsub_opts.type.type = \"MyMessageType\";\nsub_opts.type.type_length = strlen(sub_opts.type.type);\n\n// Create subscriber\nSubspaceSubscriber sub = subspace_create_subscriber(client, \"my_channel\", sub_opts);\nif (sub.subscriber == NULL) {\n    fprintf(stderr, \"Failed to create subscriber: %s\\n\", subspace_get_last_error());\n    return 1;\n}\n```\n\n### Publishing Messages\n\n```c\n// Get a message buffer\nSubspaceMessageBuffer buffer = subspace_get_message_buffer(pub, 1024);\nif (buffer.buffer == NULL) {\n    // For reliable publishers, you may need to wait\n    if (pub_opts.reliable) {\n        subspace_wait_for_publisher(pub);\n        buffer = subspace_get_message_buffer(pub, 1024);\n    } else {\n        fprintf(stderr, \"Failed to get buffer: %s\\n\", subspace_get_last_error());\n        return 1;\n    }\n}\n\n// Fill in your message data\nMyMessageType* msg = (MyMessageType*)buffer.buffer;\nmsg-\u003efield1 = 42;\nmsg-\u003efield2 = 3.14;\n\n// Publish the message\nconst SubspaceMessage pub_status = subspace_publish_message(pub, sizeof(MyMessageType));\nif (pub_status.length == 0) {\n    fprintf(stderr, \"Failed to publish: %s\\n\", subspace_get_last_error());\n    return 1;\n}\n// pub_status.ordinal contains the message sequence number\n// pub_status.timestamp contains the publish timestamp\n```\n\n### Reading Messages\n\n```c\n// Read next message\nSubspaceMessage msg = subspace_read_message(sub);\nif (msg.length == 0) {\n    // No message available\n    // For reliable subscribers, you may want to wait\n    if (sub_opts.reliable) {\n        subspace_wait_for_subscriber(sub);\n        msg = subspace_read_message(sub);\n    }\n}\n\nif (msg.length \u003e 0) {\n    // Process the message\n    const MyMessageType* data = (const MyMessageType*)msg.buffer;\n    printf(\"Received message ordinal: %lu\\n\", msg.ordinal);\n    printf(\"Message timestamp: %lu\\n\", msg.timestamp);\n    \n    // IMPORTANT: Free the message when done\n    subspace_free_message(\u0026msg);\n}\n\n// Read newest message (skips to most recent)\nSubspaceMessage newest = subspace_read_message_with_mode(sub, kSubspaceReadNewest);\nif (newest.length \u003e 0) {\n    // Process message\n    subspace_free_message(\u0026newest);\n}\n```\n\n**Important:** You must call `subspace_free_message()` when done with a message. The `max_active_messages` option determines how many messages you can hold simultaneously. If you don't free messages, the subscriber will run out of slots and be unable to read more messages.\n\n### Waiting for Messages\n\n```c\n// Wait indefinitely for a message\nif (!subspace_wait_for_subscriber(sub)) {\n    fprintf(stderr, \"Wait failed: %s\\n\", subspace_get_last_error());\n    return 1;\n}\n\n// Wait with file descriptor (for integration with event loops)\nint fd = /* your file descriptor */;\nint triggered_fd = subspace_wait_for_subscriber_with_fd(sub, fd);\nif (triggered_fd \u003c 0) {\n    fprintf(stderr, \"Wait failed: %s\\n\", subspace_get_last_error());\n    return 1;\n}\n```\n\n### Using Poll/Epoll\n\nThe C API provides file descriptors that can be used with `poll()`, `epoll()`, or other event notification mechanisms:\n\n```c\n// Get pollfd structure for subscriber\nstruct pollfd pfd = subspace_get_subscriber_poll_fd(sub);\n// pfd.fd is the file descriptor\n// pfd.events should be set to POLLIN\n\n// Use in poll() call\nint ret = poll(\u0026pfd, 1, timeout_ms);\nif (ret \u003e 0 \u0026\u0026 (pfd.revents \u0026 POLLIN)) {\n    // Message available, read it\n    SubspaceMessage msg = subspace_read_message(sub);\n    // ... process message ...\n    subspace_free_message(\u0026msg);\n}\n\n// Or get the raw file descriptor\nint fd = subspace_get_subscriber_fd(sub);\n// Use fd with epoll, select, etc.\n```\n\n### Callbacks\n\nThe C API supports callbacks for message reception and dropped messages:\n\n```c\n// Message callback\nvoid message_callback(SubspaceSubscriber sub, SubspaceMessage msg) {\n    if (msg.length \u003e 0) {\n        printf(\"Received message of size %zu\\n\", msg.length);\n        // Process message\n        // IMPORTANT: Free the message when done\n        subspace_free_message(\u0026msg);\n    }\n}\n\n// Register callback\nif (!subspace_register_subscriber_callback(sub, message_callback)) {\n    fprintf(stderr, \"Failed to register callback: %s\\n\", subspace_get_last_error());\n    return 1;\n}\n\n// Process all available messages (calls the callback for each)\nsubspace_process_all_messages(sub);\n\n// Unregister callback\nsubspace_remove_subscriber_callback(sub);\n\n// Dropped message callback\nvoid dropped_callback(SubspaceSubscriber sub, int64_t count) {\n    fprintf(stderr, \"Dropped %ld messages\\n\", count);\n}\n\nsubspace_register_dropped_message_callback(sub, dropped_callback);\n```\n\n### Complete C Example\n\n```c\n#include \"c_client/subspace.h\"\n#include \u003cstdio.h\u003e\n#include \u003cstring.h\u003e\n\nstruct SensorData {\n    double temperature;\n    double pressure;\n    uint64_t timestamp;\n};\n\nint main() {\n    // Create client\n    SubspaceClient client = subspace_create_client();\n    if (client.client == NULL) {\n        fprintf(stderr, \"Failed to create client: %s\\n\", subspace_get_last_error());\n        return 1;\n    }\n\n    // Create reliable publisher\n    SubspacePublisherOptions pub_opts = subspace_publisher_options_default(\n        sizeof(SensorData), 10);\n    pub_opts.reliable = true;\n    pub_opts.type.type = \"SensorData\";\n    pub_opts.type.type_length = strlen(pub_opts.type.type);\n    \n    SubspacePublisher pub = subspace_create_publisher(client, \"sensors\", pub_opts);\n    if (pub.publisher == NULL) {\n        fprintf(stderr, \"Failed to create publisher: %s\\n\", subspace_get_last_error());\n        return 1;\n    }\n\n    // Create reliable subscriber\n    SubspaceSubscriberOptions sub_opts = subspace_subscriber_options_default();\n    sub_opts.reliable = true;\n    sub_opts.type.type = \"SensorData\";\n    sub_opts.type.type_length = strlen(sub_opts.type.type);\n    \n    SubspaceSubscriber sub = subspace_create_subscriber(client, \"sensors\", sub_opts);\n    if (sub.subscriber == NULL) {\n        fprintf(stderr, \"Failed to create subscriber: %s\\n\", subspace_get_last_error());\n        return 1;\n    }\n\n    // Publisher loop\n    for (int i = 0; i \u003c 100; ++i) {\n        // Wait for free slot (reliable publisher)\n        subspace_wait_for_publisher(pub);\n        \n        // Get buffer\n        SubspaceMessageBuffer buffer = subspace_get_message_buffer(pub, sizeof(SensorData));\n        if (buffer.buffer == NULL) {\n            continue;\n        }\n        \n        // Fill message\n        struct SensorData* data = (struct SensorData*)buffer.buffer;\n        data-\u003etemperature = 20.0 + i * 0.1;\n        data-\u003epressure = 1013.25;\n        data-\u003etimestamp = /* get current time */;\n        \n        // Publish\n        const SubspaceMessage pub_status = subspace_publish_message(pub, sizeof(SensorData));\n        if (pub_status.length \u003e 0) {\n            printf(\"Published message %lu\\n\", pub_status.ordinal);\n        }\n    }\n\n    // Subscriber loop\n    for (int i = 0; i \u003c 100; ++i) {\n        // Wait for message\n        subspace_wait_for_subscriber(sub);\n        \n        // Read message\n        SubspaceMessage msg = subspace_read_message(sub);\n        if (msg.length \u003e 0) {\n            const struct SensorData* data = (const struct SensorData*)msg.buffer;\n            printf(\"Received: temp=%.2f, pressure=%.2f, ordinal=%lu\\n\",\n                   data-\u003etemperature, data-\u003epressure, msg.ordinal);\n            subspace_free_message(\u0026msg);\n        }\n    }\n\n    // Cleanup\n    subspace_remove_subscriber(\u0026sub);\n    subspace_remove_publisher(\u0026pub);\n    subspace_remove_client(\u0026client);\n    \n    return 0;\n}\n```\n\n### C API Reference\n\n**Client Functions:**\n- `SubspaceClient subspace_create_client(void)`\n- `SubspaceClient subspace_create_client_with_socket(const char *socket_name)`\n- `SubspaceClient subspace_create_client_with_socket_and_name(const char *socket_name, const char *client_name)`\n- `bool subspace_remove_client(SubspaceClient *client)`\n\n**Publisher Functions:**\n- `SubspacePublisherOptions subspace_publisher_options_default(int32_t slot_size, int num_slots)`\n- `SubspacePublisher subspace_create_publisher(SubspaceClient client, const char *channel_name, SubspacePublisherOptions options)`\n- `SubspaceMessageBuffer subspace_get_message_buffer(SubspacePublisher publisher, size_t max_size)`\n- `const SubspaceMessage subspace_publish_message(SubspacePublisher publisher, size_t messageSize)`\n- `bool subspace_wait_for_publisher(SubspacePublisher publisher)`\n- `int subspace_wait_for_publisher_with_fd(SubspacePublisher publisher, int fd)`\n- `struct pollfd subspace_get_publisher_poll_fd(SubspacePublisher publisher)`\n- `int subspace_get_publisher_fd(SubspacePublisher publisher)`\n- `bool subspace_register_resize_callback(SubspacePublisher publisher, bool (*callback)(SubspacePublisher, int32_t, int32_t))`\n- `bool subspace_unregister_resize_callback(SubspacePublisher publisher)`\n- `bool subspace_remove_publisher(SubspacePublisher *publisher)`\n\n**Subscriber Functions:**\n- `SubspaceSubscriberOptions subspace_subscriber_options_default(void)`\n- `SubspaceSubscriber subspace_create_subscriber(SubspaceClient client, const char *channel_name, SubspaceSubscriberOptions options)`\n- `SubspaceMessage subspace_read_message(SubspaceSubscriber subscriber)`\n- `SubspaceMessage subspace_read_message_with_mode(SubspaceSubscriber subscriber, SubspaceReadMode mode)`\n- `bool subspace_free_message(SubspaceMessage *message)`\n- `bool subspace_wait_for_subscriber(SubspaceSubscriber subscriber)`\n- `int subspace_wait_for_subscriber_with_fd(SubspaceSubscriber subscriber, int fd)`\n- `struct pollfd subspace_get_subscriber_poll_fd(SubspaceSubscriber subscriber)`\n- `int subspace_get_subscriber_fd(SubspaceSubscriber subscriber)`\n- `int32_t subspace_get_subscriber_slot_size(SubspaceSubscriber subscriber)`\n- `int subspace_get_subscriber_num_slots(SubspaceSubscriber subscriber)`\n- `SubspaceTypeInfo subspace_get_subscriber_type(SubspaceSubscriber subscriber)`\n- `bool subspace_register_subscriber_callback(SubspaceSubscriber subscriber, void (*callback)(SubspaceSubscriber, SubspaceMessage))`\n- `bool subspace_remove_subscriber_callback(SubspaceSubscriber subscriber)`\n- `bool subspace_register_dropped_message_callback(SubspaceSubscriber subscriber, void (*callback)(SubspaceSubscriber, int64_t))`\n- `bool subspace_remove_dropped_message_callback(SubspaceSubscriber subscriber)`\n- `bool subspace_process_all_messages(SubspaceSubscriber subscriber)`\n- `bool subspace_remove_subscriber(SubspaceSubscriber *subscriber)`\n\n**Error Functions:**\n- `char* subspace_get_last_error(void)`\n- `bool subspace_has_error(void)`\n\n## Rust Client\n\nSubspace includes a native Rust client library (`rust_client/`) that communicates\nwith the same C++ server and shares the same shared-memory layout as the C++\nclient.  Rust publishers and C++ subscribers (and vice versa) can exchange\nmessages on the same channels, including checksums and user metadata --\ncross-language interoperability is covered by automated tests.\n\n### Quick Start\n\n```rust\nuse subspace_client::{Client, ReadMode};\nuse subspace_client::options::{PublisherOptions, SubscriberOptions};\n\n// Connect to the server.\nlet client = Client::new(\"/tmp/subspace\", \"my_app\")?;\n\n// Create a publisher.\nlet pub_opts = PublisherOptions::new()\n    .set_slot_size(1024)\n    .set_num_slots(10);\nlet publisher = client.create_publisher(\"sensor_data\", \u0026pub_opts)?;\n\n// Publish a message.\nlet (buf, _cap) = publisher.get_message_buffer(64)?.unwrap();\nunsafe { std::ptr::copy_nonoverlapping(b\"hello\".as_ptr(), buf, 5); }\npublisher.publish_message(5)?;\n\n// Create a subscriber (in another client or the same one).\nlet sub_opts = SubscriberOptions::new();\nlet subscriber = client.create_subscriber(\"sensor_data\", \u0026sub_opts)?;\n\n// Read a message.\nlet msg = subscriber.read_message(ReadMode::ReadNext)?;\nassert_eq!(msg.length, 5);\n```\n\n### Features\n\n- Full pub/sub support: unreliable and reliable channels, read-next and\n  read-newest modes, activation messages, virtual channels.\n- Checksums (built-in CRC32 or custom callbacks) and per-message user metadata.\n- File-descriptor-based `wait()` for integration with event loops and `poll()`.\n- Slot retirement notification for reliable publishers.\n- Runs on Linux and macOS (ARM64 and x86_64).\n\n### Building\n\n```bash\n# Build the library\nbazel build //rust_client:subspace_client_rust\n\n# Run the tests (starts an in-process C++ server via FFI)\nbazel test //rust_client:client_test\n```\n\n### Bazel Dependency\n\n```starlark\nrust_binary(\n    name = \"my_app\",\n    deps = [\"@subspace//rust_client:subspace_client_rust\"],\n)\n```\n\nSee [docs/rust-client.md](docs/rust-client.md) for the full API reference and\nusage guide.\n\n## Message Types and Serialization\n\nSubspace is message-type agnostic. You can send any data structure as long as it fits in the slot size. Common approaches:\n\n1. **Plain C structs** (as shown above) - fastest, no serialization overhead\n2. **Protocol Buffers** - cross-language, versioned\n3. **Zero-copy facilities** like [Phaser](https://github.com/dallison/phaser) or [Neutron](https://github.com/dallison/neutron) - zero-copy, schema evolution\n4. **JSON** - human-readable, flexible\n5. **Custom binary formats**\n\nThe `type` field in `PublisherOptions` and `SubscriberOptions` is purely for application-level type checking - Subspace doesn't validate or enforce it.\n\n## Thread Safety\n\nBy default, the `Client` class is **not thread-safe**. To enable thread-safe mode:\n\n```cpp\nclient-\u003eSetThreadSafe(true);\n```\n\nIn thread-safe mode:\n- `GetMessageBuffer()` acquires a lock that is held until `PublishMessage()` or `CancelPublish()` is called\n- You must call `PublishMessage()` or `CancelPublish()` after `GetMessageBuffer()`\n- Multiple threads can safely use the same client instance\n\n\n## Coroutine Support\n\nSubspace is coroutine-aware. If you pass a coroutine pointer when creating the client, blocking operations will yield to other coroutines:\n\n```cpp\nco::CoroutineScheduler scheduler;\nco::Coroutine* co = scheduler.CreateCoroutine([]() {\n    auto client = subspace::Client::Create(\"/tmp/subspace\", \"co_client\", \n                                           co::Coroutine::Current()).value();\n    // ... use client ...\n});\nscheduler.Run();\n```\n\nWhen using coroutines, `Wait()` operations will yield instead of blocking the thread.\n\n## Shadow Server (Crash Recovery)\n\nSubspace supports a **shadow process** that mirrors the server's channel,\npublisher, and subscriber state.  If the server crashes and restarts it can\nreconnect to the shadow, reload the full state, and resume operation without\nlosing shared-memory buffers.  Existing clients can then reclaim their\npublishers and subscribers.\n\n### How It Works\n\nThe shadow is a lightweight, coroutine-based daemon that maintains a copy of\nthe server's channel database.  It communicates with the server over a Unix\ndomain socket using protobuf-encoded `ShadowEvent` messages.  Shared-memory\nfile descriptors (SCB, CCB, BCB, trigger, and retirement FDs) are passed once\nusing `SCM_RIGHTS` so the shadow holds them open even if the server dies.\n\nOn startup the server connects to the shadow and receives a state dump.  If\nthe state dump contains channels (i.e. the shadow has state from a previous\nserver instance), the server re-maps the existing shared memory and recreates\nits internal channel, publisher, and subscriber structures.  It then\nre-replicates all recovered state back to the shadow(s) so they stay in sync.\n\n### Dual Shadow Support\n\nThe server supports two shadows -- a primary and a secondary -- for\nadditional redundancy.  On recovery it tries the primary first; if the primary\nhas no state (or is unavailable) it falls back to the secondary.  After\nrecovery, both shadows are brought up to date with the full state.\n\n### Running\n\nStart one or two shadow processes:\n\n```bash\n# Primary shadow\n./bazel-bin/shadow/subspace_shadow --socket=/tmp/subspace_shadow\n\n# Optional secondary shadow\n./bazel-bin/shadow/subspace_shadow --socket=/tmp/subspace_shadow2\n```\n\nThen start the server with shadow sockets:\n\n```bash\n./bazel-bin/server/subspace_server \\\n    --shadow_socket=/tmp/subspace_shadow \\\n    --secondary_shadow_socket=/tmp/subspace_shadow2\n```\n\nIf the server is killed and restarted with the same flags, it will recover\nits full state from whichever shadow is available.\n\n### What Survives a Restart\n\n- Channel definitions (name, slot size, number of slots, type, flags).\n- Shared memory mappings -- buffers remain intact in `/dev/shm` (Linux) or\n  POSIX shared memory (macOS).\n- Publisher and subscriber metadata (IDs, trigger FDs, reliability settings,\n  tunnel flags).\n- The session ID, so clients can detect a server restart and reclaim their\n  connections.\n\n### What Does Not Survive\n\n- Active client TCP connections -- clients must reconnect and re-register\n  their publishers and subscribers.\n- In-flight messages that had not yet been consumed are still in shared memory,\n  but subscribers need to re-attach to resume reading.\n\n### Server Flags\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--shadow_socket` | `\"\"` (disabled) | Unix socket path for the primary shadow process |\n| `--secondary_shadow_socket` | `\"\"` (disabled) | Unix socket path for the secondary shadow process |\n\n### Shadow Flags\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--socket` | `/tmp/subspace_shadow` | Unix socket path to listen on |\n| `--log_level` | `info` | Log level (`debug`, `info`, `warning`, `error`) |\n\nSee [docs/shadow-process.md](docs/shadow-process.md) for the full design\ndocument including the protocol, FD lifecycle, and recovery sequence.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdallison%2Fsubspace","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdallison%2Fsubspace","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdallison%2Fsubspace/lists"}