https://github.com/netdata/plugin-ipc

Last synced: 13 days ago
JSON representation

Host: GitHub
URL: https://github.com/netdata/plugin-ipc
Owner: netdata
Created: 2026-03-08T15:45:04.000Z (4 months ago)
Default Branch: main
Last Pushed: 2026-06-29T13:06:43.000Z (15 days ago)
Last Synced: 2026-06-29T15:09:04.923Z (15 days ago)
Language: Go
Size: 15.2 MB
Stars: 1
Watchers: 0
Forks: 1
Open Issues: 1
Metadata Files:
- Readme: README.md
- Agents: AGENTS.md

Awesome Lists containing this project

README

# plugin-ipc

Cross-language IPC library for Netdata plugins and helper services.

This repository contains the C library, Rust crate, and Go package for the
same wire contracts and typed APIs. The goal is simple:

- one specification
- one interoperable protocol stack
- one typed service model
- one local snapshot/cache helper layer
- three implementations: C, Rust, Go

This README is a summary of the current verified state of the repository.
The authoritative specifications live under [docs/](docs/README.md).

## Non-Negotiable Contract

- These rules apply to every implementation language and every service unless
a future spec explicitly changes the global contract.
- C, Rust, and Go must implement the same wire contract and typed behavior.
- NetIPC does not do backward-compatible, forward-compatible, or best-effort
decoding across method, layout, status, echoed-key, or generation drift.
- Netdata plugins and NetIPC providers/clients must match the documented
contract exactly. Any mismatch is rejected.
- Mixed-generation stitched lookup responses are not supported.
- Level 2 lookup callers pass typed semantic keys; they do not split requests
around transport payload budgets and do not stitch responses manually.
- Lookup `PAYLOAD_EXCEEDED` is an internal Level 2 retry signal. The client
retries only the affected suffix and returns one logical response.
- Lookup `OVERSIZED_ITEM` is a final per-item outcome. One oversized valid item
must not invalidate the rest of the logical lookup batch.
- Payload budgets and logical lookup ceilings are initialization policy.
Zero-valued fields use documented defaults; consumers can override them for
small IoT systems or large-memory HPC deployments.
- Named defaults are defaults, not protocol hard limits. Explicit client/server
initialization config is the deployment authority.

## Service Model

The public contract is service-oriented, not plugin-oriented:

- clients connect to a specific service kind
- clients do not care which plugin/process serves that service
- one service endpoint serves exactly one request kind
- sessions are long-lived and reused for many requests

Examples of service kinds:

- `cgroups-snapshot`
- `cgroups-lookup`
- `apps-lookup`
- `ip-to-asn`
- `pid-traffic`

Operational implications:

- startup order is not guaranteed
- a client may start before its provider exists
- services may disappear and reappear during runtime
- enrichments are optional by design
- clients refresh / reconnect from their normal loop and must tolerate service absence

## What This Repository Implements

- **Level 1 transport**
- connection lifecycle
- handshake and profile negotiation
- framing, chunking, batching, pipelining
- baseline and shared-memory transports
- **Codec**
- wire encode/decode
- typed views
- response builders
- validation rules
- **Level 2 typed API**
- typed client calls
- managed typed servers
- retry / reconnect behavior
- internal reusable buffers
- **Level 3 snapshot API**
- refresh
- local cache construction
- fast hash lookup
- cache preservation on failure

The design is layered:

- **Level 1** and **Codec** are parallel building blocks
- **Level 2** composes Level 1 + Codec
- **Level 3** builds on Level 2

See:

- [docs/README.md](docs/README.md)
- [docs/level2-typed-api.md](docs/level2-typed-api.md)
- [docs/level3-snapshot-api.md](docs/level3-snapshot-api.md)
- [docs/getting-started.md](docs/getting-started.md)
- [docs/netipc-integrator-skill.md](docs/netipc-integrator-skill.md)

## Platforms And Transports

| Platform | Baseline transport | Negotiated fast path | Languages |
|---|---|---|---|
| POSIX / Linux | Unix domain `SOCK_SEQPACKET` | POSIX shared memory | C, Rust, Go |
| Windows | Named Pipes | Windows shared memory | C, Rust, Go |

Important facts:

- the same wire contracts are implemented in all three languages
- cross-language interoperability is mandatory
- Go stays pure Go, without `cgo`
- Level 2 and Level 3 are transport-agnostic from the caller perspective

## API Levels

### Level 1: Transport

Level 1 works with framed byte messages.

It owns:

- send / receive
- message IDs
- batch directories
- chunk continuation
- profile negotiation
- transport-specific session details

Relevant specs:

- [docs/level1-transport.md](docs/level1-transport.md)
- [docs/level1-wire-envelope.md](docs/level1-wire-envelope.md)
- [docs/level1-posix-uds.md](docs/level1-posix-uds.md)
- [docs/level1-posix-shm.md](docs/level1-posix-shm.md)
- [docs/level1-windows-np.md](docs/level1-windows-np.md)
- [docs/level1-windows-shm.md](docs/level1-windows-shm.md)

### Codec

Codec is pure wire-format logic.

It owns:

- encode / decode
- typed views over payload bytes
- response builders
- validation of field layout and bounds

It does **not** own:

- sockets
- pipes
- shared memory mappings
- retries
- cache policy

Relevant specs:

- [docs/codec.md](docs/codec.md)
- [docs/codec-cgroups-snapshot.md](docs/codec-cgroups-snapshot.md)
- [docs/codec-cgroups-lookup.md](docs/codec-cgroups-lookup.md)
- [docs/codec-apps-lookup.md](docs/codec-apps-lookup.md)

### Level 2: Typed API

Level 2 is the public convenience layer.

The public contract is:

- clients issue typed calls
- servers export one typed service kind
- one server endpoint serves one request kind only
- callers do not manage transport scratch buffers
- callers do not manipulate raw payload bytes

Relevant specs:

- [docs/level2-typed-api.md](docs/level2-typed-api.md)
- [docs/getting-started.md](docs/getting-started.md)

### Level 3: Snapshot / Cache

Level 3 provides:

- typed snapshot refresh
- local materialization
- O(1)-style hash lookup on the hot path
- cache retention across refresh failures

Relevant spec:

- [docs/level3-snapshot-api.md](docs/level3-snapshot-api.md)

## Interoperability

This repository is intentionally built around interoperability, not
single-language wrappers.

What is covered:

- C client -> C / Rust / Go server
- Rust client -> C / Rust / Go server
- Go client -> C / Rust / Go server
- baseline transport matrices on POSIX and Windows
- shared-memory matrices on POSIX and Windows
- typed Level 2 services
- snapshot refresh and local lookup flows
- benchmark matrices across all directed pairs

The interop results are validated by the test suite and by the checked-in
benchmark reports:

- [benchmarks-posix.md](benchmarks-posix.md)
- [benchmarks-windows.md](benchmarks-windows.md)

## Performance Snapshot

The repository includes checked-in benchmark reports with complete,
fail-closed matrices:

- POSIX report: [benchmarks-posix.md](benchmarks-posix.md)
- generated `2026-05-25`
- machine: local benchmark host
- complete matrix rows: `297`
- Windows report: [benchmarks-windows.md](benchmarks-windows.md)
- generated `2026-05-25`
- machine: local Windows benchmark host
- complete matrix rows: `201`

Headline numbers from the current checked-in reports:

- **POSIX baseline UDS ping-pong**
- `142.1k` to `169.5k` req/s across the 3x3 language matrix
- **POSIX SHM ping-pong**
- `2.32M` to `3.22M` req/s
- **POSIX UDS batch ping-pong**
- `20.18M` to `30.13M` req/s
- **POSIX SHM batch ping-pong**
- `28.71M` to `48.95M` req/s
- **POSIX UDS pipeline (depth=16)**
- `440.4k` to `620.6k` req/s
- **POSIX UDS pipeline+batch (depth=16)**
- `41.61M` to `76.45M` req/s
- **POSIX snapshot refresh**
- baseline: `131.2k` to `156.3k` req/s
- SHM: `1.10M` to `1.52M` req/s
- **POSIX lookup methods**
- `cgroups-lookup`: `31.8k` to `1.10M` req/s across max-rate scenarios
- `apps-lookup`: `35.7k` to `1.34M` req/s across max-rate scenarios
- **POSIX local cache lookup**
- C: `173.47M` req/s
- Go: `138.48M` req/s
- Rust: `147.68M` req/s

- **Windows Named Pipe ping-pong**
- `18.2k` to `21.0k` req/s
- **Windows SHM ping-pong**
- `2.10M` to `2.72M` req/s
- **Windows Named Pipe batch ping-pong**
- `7.01M` to `8.55M` req/s
- **Windows SHM batch ping-pong**
- `36.49M` to `58.77M` req/s
- **Windows Named Pipe pipeline (depth=16)**
- `245.4k` to `270.5k` req/s
- **Windows Named Pipe pipeline+batch (depth=16)**
- `28.98M` to `41.27M` req/s
- **Windows snapshot refresh**
- Named Pipe: `16.1k` to `21.0k` req/s
- SHM: `857.8k` to `1.26M` req/s
- **Windows local cache lookup**
- C: `130.65M` req/s
- Go: `107.47M` req/s
- Rust: `164.31M` req/s

The full reports include:

- per-pair throughput
- latency percentiles
- client and server CPU
- complete scenario validation summaries
- performance floor checks

## Reliability, Testing, And Coverage

The repo is not asking the reader to trust the design on words alone.
The current validation story includes:

- CMake-based build and `ctest` workflows
- unit tests
- cross-language interop tests
- typed service tests
- transport tests
- shared-memory tests
- coverage scripts for C, Go, and Rust
- benchmark generators that reject incomplete matrices

### Current verified state

Linux / POSIX:

- build: passing
- `ctest`: `46/46` passing
- C coverage: `94.1%`
- Go coverage: `95.8%`
- Rust coverage: `98.57%`
- measured with `cargo-llvm-cov`
- Linux run now excludes Windows-tagged Rust files from the Linux total
- Unix Rust service tests now live in a separate `cgroups_unix_tests.rs` file
- Unix Rust transport tests now live in separate `posix_tests.rs` and `shm_tests.rs` files
- the small Rust protocol test modules stay inline for now because externalizing them lowered the headline total without enough runtime-signal benefit

Windows (`win11`):

- build: passing
- `ctest`: `28/28` passing
- C coverage: `93.2%`
- Go coverage: `95.4%`
- Rust coverage: `92.08%`

Important honesty point:

- core build, transport, service, interop, and benchmark validation is strong
on both Linux and Windows
- Linux still has broader chaos / hardening / stress breadth than Windows
- so the platforms are in **good functional parity**, but not yet in
**full validation parity**

Coverage details:

- [WINDOWS-COVERAGE.md](WINDOWS-COVERAGE.md)
- [COVERAGE-EXCLUSIONS.md](COVERAGE-EXCLUSIONS.md)

## Specifications And Trust Model

The specs are authoritative.

Rule:

- when code and spec disagree, the spec wins unless explicitly revised

Start here:

- [docs/README.md](docs/README.md)

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/netdata/plugin-ipc

Awesome Lists containing this project

README