{"id":46941131,"url":"https://github.com/hyperi-io/hyperi-rustlib","last_synced_at":"2026-06-14T02:09:51.398Z","repository":{"id":343327157,"uuid":"1122107756","full_name":"hyperi-io/hyperi-rustlib","owner":"hyperi-io","description":"Opinionated, drop-in Rust toolkit for production services at scale. The patterns from blog posts as actual code: 8-layer config cascade, structured logging with PII masking, Prometheus + OpenTelemetry, tiered disk-spillover sinks, Kafka/gRPC transports, adaptive worker pools, graceful shutdown.","archived":false,"fork":false,"pushed_at":"2026-05-29T00:09:12.000Z","size":1613,"stargazers_count":0,"open_issues_count":9,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-29T00:20:48.498Z","etag":null,"topics":["async","backpressure","circuit-breaker","config","data-pipeline","grpc","hot-path","kafka","kubernetes","metrics","observability","opentelemetry","prometheus","rust","rust-library","secrets-management","simd","structured-logging","tokio","tracing"],"latest_commit_sha":null,"homepage":"https://hyperi.io","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hyperi-io.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-24T05:45:24.000Z","updated_at":"2026-05-29T00:08:37.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hyperi-io/hyperi-rustlib","commit_stats":null,"previous_names":["hyperi-io/hyperi-rustlib"],"tags_count":140,"template":false,"template_full_name":null,"purl":"pkg:github/hyperi-io/hyperi-rustlib","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperi-io%2Fhyperi-rustlib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperi-io%2Fhyperi-rustlib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperi-io%2Fhyperi-rustlib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperi-io%2Fhyperi-rustlib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hyperi-io","download_url":"https://codeload.github.com/hyperi-io/hyperi-rustlib/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperi-io%2Fhyperi-rustlib/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33632271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","backpressure","circuit-breaker","config","data-pipeline","grpc","hot-path","kafka","kubernetes","metrics","observability","opentelemetry","prometheus","rust","rust-library","secrets-management","simd","structured-logging","tokio","tracing"],"created_at":"2026-03-11T07:05:04.279Z","updated_at":"2026-06-14T02:09:51.393Z","avatar_url":"https://github.com/hyperi-io.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# hyperi-rustlib\n\n\u003c!-- BADGES:START --\u003e\n[![Build Status](https://github.com/hyperi-io/hyperi-rustlib/actions/workflows/ci.yml/badge.svg)](https://github.com/hyperi-io/hyperi-rustlib/actions)\n[![Crates.io](https://img.shields.io/crates/v/hyperi-rustlib?logo=rust)](https://crates.io/crates/hyperi-rustlib)\n[![docs.rs](https://img.shields.io/docsrs/hyperi-rustlib?logo=rust)](https://docs.rs/hyperi-rustlib)\n[![License](https://img.shields.io/badge/license-BUSL--1.1-blue)](LICENSE)\n\u003c!-- BADGES:END --\u003e\n\nThere's plenty of sage advice out there about how to run Rust services in production at scale — config cascades, structured logging, masking secrets, multi-backend secrets management, Prometheus, OpenTelemetry, Kafka transports, tiered disk-spillover sinks, adaptive worker pools, graceful shutdown — but almost none of it as code you can just install and use.\n\nThis is that code.\n\nOpinionated, drop-in, working out of the box. The patterns from blog posts, watercooler chats and beers with your Google mates as actual library — not a framework you assemble from twenty crates and 8 weeks of munging.\n\nBuilt as the foundation for HyperI's PB/hr data services. Generic enough\nthat you don't need to be at HyperI to use it.\n\nThis module exists because of this — \u003chttps://www.youtube.com/watch?v=xE9W9Ghe4Jk\u003e — but for the backend. And of course, no microservices.\n\n## Quick Start\n\n```toml\n[dependencies]\nhyperi-rustlib = \"2\"\n```\n\nDefault features: `config`, `logger`. Add the others you want explicitly.\n\n```rust\nuse hyperi_rustlib::{config, logger, env};\n\nfn main() -\u003e anyhow::Result\u003c()\u003e {\n    let environment = env::Environment::detect();\n    logger::setup_default()?;\n    config::setup(config::ConfigOptions {\n        env_prefix: \"MYAPP\".into(),\n        ..Default::default()\n    })?;\n\n    tracing::info!(\"Running in {environment:?}\");\n    Ok(())\n}\n```\n\n## Features\n\nPick the slice you need; pay only for what you use.\n\n| Feature | Description |\n|---------|-------------|\n| `env` | Environment detection (K8s, Docker, Container, BareMetal) |\n| `runtime` | Runtime path resolution (XDG/container-aware) |\n| `config` | 8-layer config cascade (figment-based) |\n| `config-reload` | `SharedConfig\u003cT\u003e` + `ConfigReloader` hot-reload |\n| `config-postgres` | PostgreSQL config source |\n| `logger` | Structured logging, JSON/text auto-detect, sensitive-field masking |\n| `metrics` | Prometheus metrics + process/container metrics |\n| `otel-metrics` | OpenTelemetry metrics export (OTLP) |\n| `otel-tracing` | OpenTelemetry distributed tracing |\n| `http` | HTTP client with retry middleware (reqwest) |\n| `http-server` | Axum HTTP server with health probe trinity (`/healthz/{startup,live,ready}`) |\n| `transport-kafka` | Kafka transport (rdkafka, dynamic-linking) |\n| `transport-grpc` | gRPC transport (tonic/prost) |\n| `transport-memory` | In-memory transport (testing/dev) |\n| `transport-grpc-vector-compat` | Vector wire-protocol compatibility |\n| `spool` | Disk-backed async FIFO queue (yaque + zstd) |\n| `tiered-sink` | Resilient delivery: hot buffer + circuit breaker + disk spillover |\n| `secrets` | Secrets management core (file backend) |\n| `secrets-vault` | OpenBao / HashiCorp Vault provider |\n| `secrets-aws` | AWS Secrets Manager provider |\n| `directory-config` | YAML directory-backed config store |\n| `directory-config-git` | Git integration for directory-config (git2) |\n| `scaling` | Back-pressure / scaling-pressure primitives |\n| `cli` | Standard CLI framework (clap) |\n| `top` | TUI metrics dashboard (ratatui) |\n| `io` | File rotation, NDJSON writer |\n| `dlq` | Dead-letter queue (file backend) |\n| `dlq-kafka` | DLQ Kafka backend |\n| `output-file` | File output sink |\n| `expression` | CEL expression evaluation |\n| `deployment` | Deployment-contract validation |\n| `version-check` | Optional startup version check |\n| `resilience` | Circuit breaker, retry, bulkhead (tower-resilience) |\n| `full` | Everything |\n\n## Native System Dependencies\n\nThis crate dynamically links against system C libraries for several features.\n**Both build hosts and deployment targets need the appropriate packages.**\n\n### Build Host (CI / Development)\n\n| Feature | Crate | Build Package | Notes |\n|---------|-------|--------------|-------|\n| `transport-kafka` | `rdkafka-sys` | `librdkafka-dev` (\u003e= 2.12.1) | Requires [Confluent APT repo](https://packages.confluent.io/clients/deb) — Ubuntu's default is too old |\n| `directory-config-git` | `libgit2-sys` | `libgit2-dev`, `libssh2-1-dev` | System lib avoids vendored C build |\n| `spool`, `tiered-sink` | `zstd-sys` | `libzstd-dev` | System lib avoids vendored C build |\n| (transitive) | `libz-sys` | `zlib1g-dev` | Used by multiple deps |\n| (transitive) | `openssl-sys` | `libssl-dev` | Dynamic linking via pkg-config |\n| `secrets-aws` | `aws-lc-sys` | — | C/C++ compiled from source (no system lib available); ~20–30s first build, cached by sccache |\n\nFor `librdkafka-dev` \u003e= 2.12.1, add the Confluent APT repo:\n\n```bash\ncurl -fsSL https://packages.confluent.io/clients/deb/archive.key \\\n  | sudo gpg --dearmor -o /usr/share/keyrings/confluent-clients.gpg\necho \"deb [signed-by=/usr/share/keyrings/confluent-clients.gpg] \\\n  https://packages.confluent.io/clients/deb noble main\" \\\n  | sudo tee /etc/apt/sources.list.d/confluent-clients.list\nsudo apt-get update\nsudo apt-get install -y librdkafka-dev libssl-dev libsasl2-dev pkg-config\n```\n\n### Deployment Host (Runtime)\n\nThe compiled binary links against `.so` files at runtime. Install the\n**runtime** packages (not `-dev`) on deployment hosts or in Docker images.\n\n| Feature | Runtime Package | Shared Object |\n|---------|----------------|---------------|\n| `transport-kafka` | `librdkafka1` (from Confluent repo) | `librdkafka.so.1` |\n| `directory-config-git` | `libgit2-1.7` (or matching version) | `libgit2.so` |\n| `spool`, `tiered-sink` | `libzstd1` | `libzstd.so.1` |\n| (transitive) | `zlib1g` | `libz.so.1` |\n| (transitive) | `libssl3` | `libssl.so.3` |\n\nOnly install what you use. Check the features your binary enables to\ndetermine which runtime packages are needed.\n\n### Docker Example\n\n```dockerfile\n# Build stage\nFROM rust:1 AS builder\nRUN apt-get update \u0026\u0026 apt-get install -y \\\n    pkg-config libssl-dev librdkafka-dev libgit2-dev libzstd-dev\nCOPY . .\nRUN cargo build --release\n\n# Runtime stage\nFROM ubuntu:24.04\nRUN apt-get update \u0026\u0026 apt-get install -y --no-install-recommends \\\n    librdkafka1 libssl3 libgit2-1.7 libzstd1 ca-certificates \\\n    \u0026\u0026 rm -rf /var/lib/apt/lists/*\nCOPY --from=builder /app/target/release/myapp /usr/local/bin/\n```\n\nFor `librdkafka1`, add the Confluent APT repo to both build and runtime stages.\n\n## Health Check Endpoints — The Probe Trinity\n\nFor services deployed to Kubernetes, the `http-server` feature provides\nthe three K8s probe types:\n\n| Probe | Path | Checks | On failure |\n|---|---|---|---|\n| Startup | `/healthz/startup` | Init complete | K8s waits, then restarts |\n| Liveness | `/healthz/live` | Process not deadlocked | Restart pod |\n| Readiness | `/healthz/ready` | Deps healthy + ready flag set | Stop routing traffic |\n\nLiveness MUST NEVER check downstream dependencies (a DB outage shouldn't\nrestart your replicas). Readiness checks dependencies AND requires an\nexplicit `set_ready()` call — cleared during graceful shutdown.\n\n## Self-regulation (default vertical scaling)\n\nA rustlib data-plane app regulates its own intake. Sized for steady state, a\npod slows down or speeds up WITHIN itself first -- the default, fast, local\nresponse to a burst, a stalled upstream, or a transform that balloons memory.\nOnly when that vertical headroom is exhausted does it escalate to horizontal\nscale (KEDA adding pods), driven by the same pressure signal. Memory is the\nhard, never-OOM authority; CPU is left to the kernel scheduler (CFS), which\nthe byte-budget loop reads through longer process times. It is ON by default\nand opt-out via `self_regulation.enabled = false`. See\n[docs/SELF-REGULATION.md](docs/SELF-REGULATION.md).\n\n## Architecture\n\nSee [docs/](docs/README.md) for the full documentation index —\n[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the module map and layering,\nand [docs/core-pillars/CONFIG.md](docs/core-pillars/CONFIG.md) for the 8-layer\nconfig cascade reference.\n\n## License\n\n[BUSL-1.1](LICENSE) — Business Source License 1.1, transitions to Apache 2.0 after 3 years.\n\n## Related\n\n- **[hyperi-pylib](https://github.com/hyperi-io/hyperi-pylib)** — sister\n  library for Python services. Same opinions, same patterns, expressive\n  Python ergonomics for control planes, APIs, and integration layers.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperi-io%2Fhyperi-rustlib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhyperi-io%2Fhyperi-rustlib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperi-io%2Fhyperi-rustlib/lists"}