{"id":43612029,"url":"https://github.com/bigbrotr/bigbrotr","last_synced_at":"2026-04-08T11:00:59.443Z","repository":{"id":290649894,"uuid":"975096301","full_name":"BigBrotr/bigbrotr","owner":"BigBrotr","description":"Modular Nostr network observatory — relay discovery, health monitoring, event archiving, analytics, and data access","archived":false,"fork":false,"pushed_at":"2026-04-04T16:02:03.000Z","size":26145,"stargazers_count":7,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-04T18:33:12.494Z","etag":null,"topics":["data-mining","docker","events-syncronizer","nostr","postgresql","python","relay-discovery","relays-monitor"],"latest_commit_sha":null,"homepage":"https://bigbrotr.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BigBrotr.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-04-29T19:21:48.000Z","updated_at":"2026-03-31T18:04:28.000Z","dependencies_parsed_at":"2025-04-29T23:21:16.107Z","dependency_job_id":"e005d778-14c5-474c-9e7d-91625721cd13","html_url":"https://github.com/BigBrotr/bigbrotr","commit_stats":null,"previous_names":["vincenzoimp/bigbrotr","bigbrotr/bigbrotr"],"tags_count":33,"template":false,"template_full_name":null,"purl":"pkg:github/BigBrotr/bigbrotr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BigBrotr%2Fbigbrotr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BigBrotr%2Fbigbrotr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BigBrotr%2Fbigbrotr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BigBrotr%2Fbigbrotr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BigBrotr","download_url":"https://codeload.github.com/BigBrotr/bigbrotr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BigBrotr%2Fbigbrotr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31551891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T10:21:54.569Z","status":"ssl_error","status_checked_at":"2026-04-08T10:21:38.171Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-mining","docker","events-syncronizer","nostr","postgresql","python","relay-discovery","relays-monitor"],"created_at":"2026-02-04T12:09:06.883Z","updated_at":"2026-04-08T11:00:59.435Z","avatar_url":"https://github.com/BigBrotr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/BigBrotr/bigbrotr/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://img.shields.io/github/actions/workflow/status/BigBrotr/bigbrotr/ci.yml?branch=develop\u0026label=CI\u0026logo=github\" alt=\"CI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/BigBrotr/bigbrotr/actions/workflows/codeql.yml\"\u003e\u003cimg src=\"https://img.shields.io/github/actions/workflow/status/BigBrotr/bigbrotr/codeql.yml?branch=develop\u0026label=CodeQL\u0026logo=github\" alt=\"CodeQL\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://codecov.io/gh/Bigbrotr/bigbrotr\"\u003e\u003cimg src=\"https://img.shields.io/codecov/c/github/Bigbrotr/bigbrotr?token=LM9D3ABW0L\u0026logo=codecov\u0026label=coverage\" alt=\"Coverage\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://bigbrotr.github.io/bigbrotr/\"\u003e\u003cimg src=\"https://img.shields.io/badge/docs-latest-blue?logo=readthedocs\u0026logoColor=white\" alt=\"Docs\"\u003e\u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/badge/python-3.11+-3776AB?logo=python\u0026logoColor=white\" alt=\"Python 3.11+\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/postgresql-18-4169E1?logo=postgresql\u0026logoColor=white\" alt=\"PostgreSQL 18\"\u003e\n  \u003ca href=\"https://github.com/BigBrotr/bigbrotr/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/github/license/BigBrotr/bigbrotr\" alt=\"License\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003eBigBrotr\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003eModular Nostr Network Observatory\u003c/strong\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  Discovers relays across clearnet, Tor, I2P, and Lokinet. Validates connectivity, runs NIP-11 and NIP-66 health checks, archives events, materializes analytics views, and exposes data through a REST API and a NIP-90 Data Vending Machine.\n\u003c/p\u003e\n\n---\n\n## What It Does\n\nBigBrotr answers three questions about the Nostr network:\n\n1. **What relays exist?** — Seeder bootstraps from a seed file, Finder discovers new relays from event tag values and external APIs.\n2. **How healthy are they?** — Validator confirms WebSocket connectivity, Monitor runs 7 health checks (RTT, SSL, DNS, Geo, Net, HTTP, NIP-11) and publishes NIP-66 events.\n3. **What events are they publishing?** — Synchronizer connects to relays, streams events, and archives them with cursor-based resumption.\n\nEight **independent** async services share a PostgreSQL database. Each runs on its own schedule, can be started or stopped individually, and has no direct dependency on any other service.\n\n```text\n                    ┌──────────────────────────────────────────────────────┐\n                    │                    PostgreSQL Database               │\n                    │                                                      │\n                    │         relay ─── event_relay ─── event              │\n                    │         metadata ─── relay_metadata                  │\n                    │         service_state     11 materialized views      │\n                    └──┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──┘\n                       │      │      │      │      │      │      │      │\n                       ▼      ▼      ▼      ▼      ▼      ▼      ▼      ▼\n                    Seeder Finder Valid. Monitor Sync. Refresh. Api    Dvm\n                       │      │      │      │      │      │      │      │\n                       ▼      ▼      ▼      ▼      ▼      │      ▼      ▼\n                    seed   HTTP   Relays Relays  Relays (no I/O) HTTP  Nostr\n                    file   APIs   (WS)  (NIP-11, (fetch               clients\n                                         NIP-66)  events)               │\n                                           │                            ▼\n                                           ▼                       Nostr Network\n                                      Nostr Network               (kind 5050/6050)\n                                    (kind 10166/30166)\n```\n\n### Services\n\n| Service | What it does | External I/O |\n|---------|-------------|-------------|\n| **Seeder** | Loads relay URLs from a seed file (one-shot) | Seed file |\n| **Finder** | Discovers relay URLs from event tag values and external APIs | HTTP (nostr.watch) |\n| **Validator** | Tests candidates via WebSocket handshake, promotes valid relays | WebSocket |\n| **Monitor** | Runs NIP-11 + 6 NIP-66 health checks, publishes kind 10166/30166 events | HTTP, WS, DNS, SSL, GeoIP |\n| **Synchronizer** | Connects to relays, streams and archives signed events with cursor-based resumption | WebSocket |\n| **Refresher** | Refreshes 11 materialized views in dependency order | None |\n| **Api** | Read-only REST API with auto-generated paginated endpoints | HTTP (FastAPI) |\n| **Dvm** | NIP-90 Data Vending Machine for database queries over Nostr | WebSocket (Nostr) |\n\nAll continuous services default to a 5-minute cycle interval (`interval=300.0`), configurable per deployment.\n\nServices are **loosely coupled through the database**: Seeder and Finder populate candidates, Validator promotes them to relays, Monitor and Synchronizer operate on validated relays, Refresher materializes analytics. Stopping one does not break the others.\n\n---\n\n## Architecture\n\n### Code Organization (Diamond DAG)\n\nImports flow strictly downward:\n\n```text\n              services         src/bigbrotr/services/\n             /   |   \\\n          core  nips  utils    src/bigbrotr/{core,nips,utils}/\n             \\   |   /\n              models           src/bigbrotr/models/\n```\n\n- **models** — Pure frozen dataclasses (Relay, Event, Metadata, ServiceState). Zero I/O, stdlib logging only.\n- **core** — Pool (asyncpg with retry), Brotr (DB facade), BaseService (lifecycle), Logger (structured kv/JSON), Metrics (Prometheus), YAML loader.\n- **nips** — NIP-11 relay info fetch/parse, NIP-66 health checks (RTT, SSL, DNS, Geo, Net, HTTP). Never raises — errors captured in structured logs.\n- **utils** — DNS resolution, Nostr key management, WebSocket/HTTP transport, SSL fallback, SOCKS5 proxy support, event streaming with binary-split windowing.\n- **services** — 8 independent services + shared queries, configs, mixins (ConcurrentStream, NetworkSemaphores, GeoReader, Clients, CatalogAccess).\n\n### Database Schema\n\n```text\n┌─────────────────────┐         ┌──────────────────────────────────────┐\n│      relay          │         │              event                   │\n│─────────────────────│         │──────────────────────────────────────│\n│ url          PK     │◄──┐ ┌──►│ id             PK  (BYTEA, 32B)      │\n│ network      TEXT   │   │ │   │ pubkey         BYTEA (32B)           │\n│ discovered_at BIGINT│   │ │   │ created_at     BIGINT                │\n└─────────┬───────────┘   │ │   │ kind           INTEGER               │\n          │               │ │   │ tags           JSONB                 │\n          │               │ │   │ tagvalues      TEXT[]                │\n          │               │ │   │ content        TEXT                  │\n          │               │ │   │ sig            BYTEA (64B)           │\n          │               │ │   └──────────────────────────────────────┘\n          │               │ │\n          │    ┌──────────┴─┴──────────────────┐\n          │    │          event_relay          │\n          │    │───────────────────────────────│\n          ├───►│ relay_url    FK ──► relay.url |\n          │    │ event_id     FK ──► event.id  |\n          │    │ seen_at      BIGINT           |\n          │    │ PK(event_id, relay_url)       |\n          │    └───────────────────────────────┘\n          │\n          │    ┌───────────────────────────────────────────────┐\n          │    │                   relay_metadata              │\n          │    │───────────────────────────────────────────────│\n          └───►│ relay_url    FK ──► relay.url                 |\n               │ metadata_id  FK ──► metadata.id               |\n               │ metadata_type FK ──► metadata.type            |\n               │ generated_at BIGINT                           |\n               │ PK(relay_url, generated_at, metadata_type)    |\n               └──────────┬────────────────────────────────────┘\n                          │\n               ┌──────────┴────────────────────┐\n               │          metadata             │\n               │───────────────────────────────│\n               │ id       PK  (BYTEA, SHA-256) |\n               │ type     PK  (TEXT, 7 types)  |\n               │ data     JSONB                |\n               └───────────────────────────────┘\n\n\n               ┌───────────────────────┐\n               │    service_state      │\n               │───────────────────────│\n               │ service_name PK (TEXT)│\n               │ state_type   PK (TEXT)│\n               │ state_key    PK (TEXT)│\n               │ state_value  JSONB    │\n               └───────────────────────┘\n```\n\n**Key relationships**:\n- `relay` is the central entity. Cascade deletes propagate to `event_relay` and `relay_metadata`.\n- `metadata` is content-addressed: SHA-256 hash of canonical JSON + type as composite PK. Same data = same hash.\n- `service_state` is a generic key-value store used by Finder (cursors), Validator (candidates), Monitor (checkpoints), Synchronizer (cursors).\n- `event.tagvalues` is computed at insert time by `event_insert()` (from `tags_to_tagvalues(tags)`) and indexed with GIN for fast containment queries.\n\n### Service-Database Interaction Map\n\n```text\n                 relay    event   event_   meta-   relay_    service_  materialized\n                                  relay    data    metadata  state     views (11)\n  ─────────────┬────────┬───────┬────────┬───────┬─────────┬─────────┬────────────\n  Seeder       │  W(1)  │       │        │       │         │  W      │\n  Finder       │  R     │       │  R     │       │         │  R/W    │\n  Validator    │  W     │       │        │       │         │  R/W    │\n  Monitor      │  R     │       │        │  W    │  W      │  R/W    │\n  Synchronizer │  R     │  W    │  W     │       │         │  R/W    │\n  Refresher    │        │       │        │       │         │         │  W\n  Api          │  R     │  R    │  R     │  R    │  R      │  R      │  R\n  Dvm          │  R     │  R    │  R     │  R    │  R      │  R      │  R\n  ─────────────┴────────┴───────┴────────┴───────┴─────────┴─────────┴────────────\n\n  R = reads    W = writes    (1) = only when to_validate=False\n```\n\n---\n\n## Quick Start\n\n### Prerequisites\n\n- Docker and Docker Compose\n- (Optional) Python 3.11+ and [uv](https://docs.astral.sh/uv/) for local development\n\n### Deploy with Docker Compose\n\n```bash\ngit clone https://github.com/BigBrotr/bigbrotr.git\ncd bigbrotr/deployments/bigbrotr\n\n# Configure secrets\ncp .env.example .env\n# Edit .env: set DB_ADMIN_PASSWORD, DB_WRITER_PASSWORD, DB_REFRESHER_PASSWORD, DB_READER_PASSWORD, NOSTR_PRIVATE_KEY, GRAFANA_PASSWORD\n\n# Start everything\ndocker compose up -d\n\n# Watch services start\ndocker compose logs -f seeder\n```\n\nThis starts PostgreSQL 18, PGBouncer, Tor proxy, all 8 services, Prometheus, Alertmanager, and Grafana.\n\n| Endpoint | URL |\n|----------|-----|\n| Grafana | `http://localhost:3000` |\n| Prometheus | `http://localhost:9090` |\n| Alertmanager | `http://localhost:9093` |\n| PostgreSQL | `localhost:5432` |\n| PGBouncer | `localhost:6432` |\n\n### Run a Single Service Locally\n\n```bash\nuv sync --group dev\ncd deployments/bigbrotr\n\n# One cycle\npython -m bigbrotr seeder --once\n\n# Continuous with debug logging\npython -m bigbrotr finder --log-level DEBUG\n```\n\n---\n\n## Deployments\n\nBigBrotr supports multiple deployment configurations from the same codebase via a single parametric Dockerfile (`deployments/Dockerfile` with `ARG DEPLOYMENT`).\n\n### BigBrotr (Full Archive)\n\nStores complete Nostr events (id, pubkey, created_at, kind, tags, content, sig).\n\n```bash\ncd deployments/bigbrotr \u0026\u0026 docker compose up -d\n```\n\n### LilBrotr (Lightweight)\n\nSame eight services and schema, but tags, content, and sig columns are nullable and never populated — approximately 60% disk savings while retaining all metadata and relay health data.\n\n```bash\ncd deployments/lilbrotr \u0026\u0026 docker compose up -d\n```\n\n### Custom Deployment\n\n```bash\ncp -r deployments/bigbrotr deployments/myrelay\n# Edit config, SQL schema, docker-compose.yaml\ncd deployments/myrelay \u0026\u0026 docker compose up -d\n```\n\n---\n\n## Database\n\nPostgreSQL 18 with PGBouncer (transaction-mode pooling) and asyncpg async driver. All mutations via stored functions with bulk array parameters.\n\n### Schema\n\n| Table | Purpose |\n|-------|---------|\n| `relay` | Validated relay URLs with network type and discovery timestamp |\n| `event` | Nostr events (BYTEA ids/pubkeys/sigs for space efficiency) |\n| `event_relay` | Junction: which events were seen at which relays (with `seen_at`) |\n| `metadata` | Content-addressed NIP-11/NIP-66 documents (SHA-256 dedup, composite PK `(id, type)`) |\n| `relay_metadata` | Time-series snapshots linking relays to metadata records |\n| `service_state` | Per-service operational data (candidates, cursors, checkpoints) |\n\n### Stored Functions (25)\n\n- **1 utility**: `tags_to_tagvalues` (extracts key-prefixed single-char tag values for GIN indexing)\n- **10 CRUD**: `relay_insert`, `event_insert`, `metadata_insert`, `event_relay_insert`, `relay_metadata_insert`, `event_relay_insert_cascade`, `relay_metadata_insert_cascade`, `service_state_upsert`, `service_state_get`, `service_state_delete`\n- **2 cleanup**: `orphan_event_delete`, `orphan_metadata_delete` (batched)\n- **12 refresh**: one per materialized view + `all_statistics_refresh`\n\nAll functions use `SECURITY INVOKER`, bulk array parameters, and `ON CONFLICT DO NOTHING`.\n\n### Materialized Views (11)\n\n`relay_metadata_latest`, `event_stats`, `relay_stats`, `kind_counts`, `kind_counts_by_relay`, `pubkey_counts`, `pubkey_counts_by_relay`, `network_stats`, `relay_software_counts`, `supported_nip_counts`, `event_daily_counts` — all support `REFRESH CONCURRENTLY` via unique indexes.\n\n---\n\n## Monitoring\n\n### Prometheus Metrics\n\nEvery service exposes `/metrics` on its configured port with four metric types:\n\n| Metric | Type | Description |\n|--------|------|-------------|\n| `service_info` | Info | Static service metadata |\n| `service_gauge` | Gauge | Point-in-time state (consecutive_failures, last_cycle_timestamp, progress) |\n| `service_counter` | Counter | Cumulative totals (cycles_success, cycles_failed, errors by type) |\n| `cycle_duration_seconds` | Histogram | Cycle latency with 10 buckets (1s to 1h) |\n\n### Alert Rules (7)\n\n| Alert | Condition | Severity |\n|-------|-----------|----------|\n| ServiceDown | `up == 0` for 5m | critical |\n| HighFailureRate | error rate \u003e 0.1/s for 5m | warning |\n| ConsecutiveFailures | 5+ consecutive cycle failures for 2m | critical |\n| SlowCycles | p99 cycle duration \u003e 300s for 5m | warning |\n| DatabaseConnectionsHigh | \u003e 80 active connections for 5m | warning |\n| CacheHitRatioLow | buffer cache hit ratio \u003c 95% for 10m | warning |\n| RefresherViewsFailing | view refresh failures for 10m | warning |\n\n### Grafana Dashboard\n\nAuto-provisioned dashboard with per-service panels: cycle duration, error counts, consecutive failures, and service-specific progress metrics.\n\n### Structured Logging\n\n```text\ninfo finder cycle_completed relay_count=100 duration=2.5\nerror validator retry_failed attempt=3 url=\"wss://relay.example.com\"\n```\n\nJSON mode available for cloud aggregation:\n\n```json\n{\"timestamp\": \"2026-02-09T12:34:56+00:00\", \"level\": \"info\", \"service\": \"finder\", \"message\": \"cycle_completed\", \"relay_count\": 100}\n```\n\n---\n\n## Nostr Protocol Support\n\n### NIPs Implemented\n\n| NIP | Usage |\n|-----|-------|\n| **NIP-01** | Event model, relay communication |\n| **NIP-11** | Relay information document fetch and parse |\n| **NIP-42** | Relay authentication (Synchronizer auth, Validator detection) |\n| **NIP-66** | Relay monitoring and discovery (kinds 10166, 22456, 30166) |\n| **NIP-89** | Handler information (DVM announcement, kind 31990) |\n| **NIP-90** | Data Vending Machine (DVM job requests/results, kinds 5050/6050) |\n\n### Event Kinds\n\n| Kind | Direction | Purpose |\n|------|-----------|---------|\n| 0 | Published | Monitor profile metadata |\n| 5050 | Consumed | NIP-90 DVM job request |\n| 6050 | Published | NIP-90 DVM job result |\n| 10166 | Published | Monitor announcement (capabilities, networks, timeouts) |\n| 22456 | Published | NIP-66 ephemeral relay test |\n| 30166 | Published | Relay discovery (addressable, one per relay, health check tags) |\n| 31990 | Published | NIP-89 handler information (DVM announcement) |\n\n### NIP-66 Health Checks\n\n| Check | What It Measures | Networks |\n|-------|-----------------|----------|\n| RTT | WebSocket open/read/write latency (ms), 3-phase with verification | All |\n| SSL | Certificate validity, expiry, issuer, SANs, cipher, fingerprint | Clearnet |\n| DNS | A/AAAA/CNAME/NS/PTR records, TTL | Clearnet |\n| Geo | Country, city, coordinates, timezone, geohash (GeoLite2 City) | Clearnet |\n| Net | IP address, ASN, organization, network ranges (GeoLite2 ASN) | Clearnet |\n| HTTP | Server header, X-Powered-By (from WebSocket handshake) | All |\n\n---\n\n## Configuration\n\n### Environment Variables\n\n| Variable | Required | Description |\n|----------|----------|-------------|\n| `DB_ADMIN_PASSWORD` | Yes | PostgreSQL admin password |\n| `DB_WRITER_PASSWORD` | Yes | Writer role password (Seeder, Finder, Validator, Monitor, Synchronizer) |\n| `DB_REFRESHER_PASSWORD` | Yes | Refresher role password (matview ownership) |\n| `DB_READER_PASSWORD` | Yes | Reader role password (Api, Dvm, postgres-exporter) |\n| `NOSTR_PRIVATE_KEY` | For Monitor, Validator, Synchronizer, Dvm | Nostr private key (hex or nsec) for event signing and NIP-42 auth |\n| `GRAFANA_PASSWORD` | For Grafana | Grafana admin password |\n\n### Configuration Files\n\n```text\ndeployments/bigbrotr/config/\n├── brotr.yaml                  # Pool, batch size, timeouts\n└── services/\n    ├── seeder.yaml             # Seed file path, validate mode\n    ├── finder.yaml             # API sources (JMESPath), event scanning, concurrency\n    ├── validator.yaml          # Networks, cleanup, processing chunk size\n    ├── monitor.yaml            # Health checks, retry per type, publishing, GeoIP\n    ├── synchronizer.yaml       # Networks, filter, time range, per-relay overrides\n    ├── refresher.yaml          # View list, refresh interval\n    ├── api.yaml                # Host, port, pagination, CORS\n    └── dvm.yaml                # NIP-90 kind, relay list, response format\n```\n\nAll configs use Pydantic v2 validation with typed defaults and constraints.\n\n---\n\n## Development\n\n### Setup\n\n```bash\ngit clone https://github.com/BigBrotr/bigbrotr.git \u0026\u0026 cd bigbrotr\ncurl -LsSf https://astral.sh/uv/install.sh | sh  # install uv (one-time)\nuv sync --group dev\npre-commit install\n```\n\n### Quality Checks\n\n```bash\nmake lint             # ruff check src/ tests/\nmake format           # ruff format src/ tests/\nmake typecheck        # mypy src/bigbrotr (strict mode)\nmake test             # pytest unit tests (~2,737 tests)\nmake test-integration # pytest integration tests (~216 tests, requires Docker)\nmake test-fast        # pytest -m \"not slow\"\nmake coverage         # pytest --cov with HTML report (80% branch minimum)\nmake ci               # all checks: lint + format-check + typecheck + test + sql-check + audit\nmake docs             # build MkDocs documentation site\nmake docs-serve       # serve docs locally with live reload\nmake build            # build Python package (sdist + wheel)\nmake docker-build     # build Docker image (DEPLOYMENT=bigbrotr)\nmake docker-up        # start Docker stack\nmake docker-down      # stop Docker stack\nmake clean            # remove build artifacts and caches\n```\n\n### Test Suite\n\n- ~2,737 unit tests + ~216 integration tests (testcontainers PostgreSQL)\n- `asyncio_mode = \"auto\"` — no `@pytest.mark.asyncio` needed\n- Global timeout: 120s per test\n- Shared fixtures via `tests/fixtures/relays.py` (registered as pytest plugin)\n- Coverage threshold: 80% (branch coverage enabled)\n\n### CI/CD Pipeline\n\n| Stage | Tool | Purpose |\n|-------|------|---------|\n| Pre-commit | ruff, mypy, yamllint, detect-secrets, markdownlint, hadolint, sqlfluff, codespell | Code quality gates (23 hooks) |\n| Unit Test | pytest (Python 3.11–3.14 matrix) | Unit tests + coverage |\n| Integration Test | pytest + testcontainers | PostgreSQL integration tests |\n| Build | Docker Buildx (matrix) | Multi-deployment image builds + Trivy scan |\n| Security | uv-secure, Trivy, CodeQL | Dependency vulns, container scanning, static analysis |\n| Release | PyPI (OIDC) + GHCR | Package + Docker image publishing, SBOM generation |\n| Docs | MkDocs Material | Auto-generated API docs deployed to GitHub Pages |\n| Dependencies | Dependabot | Weekly updates for uv, Docker, GitHub Actions |\n\n---\n\n## Project Structure\n\n```text\nbigbrotr/\n├── src/bigbrotr/                    # Main package\n│   ├── __main__.py                  # CLI entry point (service registry)\n│   ├── core/                        # Infrastructure\n│   │   ├── pool.py                  # asyncpg connection pool with retry/backoff\n│   │   ├── brotr.py                 # DB facade (stored procedures, bulk inserts)\n│   │   ├── base_service.py          # Abstract service with run_forever loop\n│   │   ├── logger.py                # Structured key=value / JSON logging\n│   │   ├── metrics.py               # Prometheus metrics server\n│   │   └── yaml.py                  # YAML config loader\n│   ├── models/                      # Pure frozen dataclasses (zero I/O)\n│   │   ├── relay.py                 # URL validation (rfc3986), network detection\n│   │   ├── event.py                 # Nostr event wrapper (nostr_sdk.Event)\n│   │   ├── metadata.py              # Content-addressed metadata (SHA-256)\n│   │   ├── event_relay.py           # Event-relay junction (cascade insert)\n│   │   ├── relay_metadata.py        # Relay-metadata junction (cascade insert)\n│   │   ├── service_state.py         # Operational state persistence\n│   │   ├── constants.py             # NetworkType, ServiceName, EventKind enums\n│   │   └── _validation.py           # Shared validation and sanitization\n│   ├── nips/                        # NIP protocol implementations (I/O)\n│   │   ├── base.py                  # Base data, logs, metadata models\n│   │   ├── parsing.py               # Declarative field parsing (FieldSpec)\n│   │   ├── event_builders.py        # Kind 0/10166/30166 event construction\n│   │   ├── nip11/                   # Relay information document\n│   │   └── nip66/                   # Health checks: rtt, ssl, dns, geo, net, http\n│   ├── utils/                       # Network primitives\n│   │   ├── protocol.py              # Nostr client, relay connection, broadcasting\n│   │   ├── transport.py             # Insecure WebSocket transport, stderr filter\n│   │   ├── streaming.py             # Event streaming with binary-split windowing\n│   │   ├── dns.py                   # Async hostname resolution (A/AAAA)\n│   │   ├── keys.py                  # Nostr key loading from environment\n│   │   ├── http.py                  # Bounded HTTP response reading\n│   │   └── parsing.py               # Tolerant model factory parsing\n│   └── services/                    # Business logic\n│       ├── seeder/                  # Seed file loading (one-shot)\n│       ├── finder/                  # Relay discovery (APIs + event scanning)\n│       ├── validator/               # WebSocket protocol validation\n│       ├── monitor/                 # Health check orchestration + publishing\n│       ├── synchronizer/            # Event collection (cursor-based)\n│       ├── refresher/               # Materialized view refresh\n│       ├── api/                     # REST API (FastAPI, read-only)\n│       ├── dvm/                     # NIP-90 Data Vending Machine\n│       └── common/                  # Shared queries, configs, mixins\n├── deployments/\n│   ├── Dockerfile                   # Single parametric (ARG DEPLOYMENT)\n│   ├── bigbrotr/                    # Full archive deployment\n│   │   ├── config/                  # YAML configs (brotr + 8 services)\n│   │   ├── postgres/init/           # SQL schema (10 files, 25 functions)\n│   │   ├── monitoring/              # Prometheus + Alertmanager + Grafana\n│   │   └── docker-compose.yaml      # 15 containers, 2 networks\n│   └── lilbrotr/                    # Lightweight deployment\n├── tests/\n│   ├── fixtures/relays.py           # Shared relay fixtures\n│   ├── unit/                        # ~2,737 tests (mirrors src/ structure)\n│   └── integration/                 # ~216 tests (testcontainers PostgreSQL)\n├── docs/                            # MkDocs Material documentation\n├── Makefile                         # Development targets\n└── pyproject.toml                   # All config: deps, ruff, mypy, pytest, coverage\n```\n\n---\n\n## Docker Infrastructure\n\n### Container Stack\n\n| Container | Image | Purpose |\n|-----------|-------|---------|\n| postgres | `postgres:18-alpine` | Primary storage |\n| pgbouncer | `edoburu/pgbouncer:v1.25.1-p0` | Transaction-mode connection pooling |\n| tor | `osminogin/tor-simple:0.4.8.10` | SOCKS5 proxy for .onion relays |\n| seeder | bigbrotr (parametric) | Relay bootstrapping (one-shot) |\n| finder | bigbrotr (parametric) | Relay discovery |\n| validator | bigbrotr (parametric) | Candidate validation |\n| monitor | bigbrotr (parametric) | Health monitoring + event publishing |\n| synchronizer | bigbrotr (parametric) | Event archiving |\n| refresher | bigbrotr (parametric) | Materialized view refresh |\n| api | bigbrotr (parametric) | REST API (FastAPI) |\n| dvm | bigbrotr (parametric) | NIP-90 Data Vending Machine |\n| postgres-exporter | `prometheuscommunity/postgres-exporter:v0.16.0` | PostgreSQL metrics |\n| prometheus | `prom/prometheus:v2.51.0` | Metrics collection (30d retention) |\n| alertmanager | `prom/alertmanager:v0.27.0` | Alert routing and grouping |\n| grafana | `grafana/grafana:10.4.1` | Dashboards |\n\n### Networks\n\n- `data-network` — postgres, pgbouncer, tor, all services\n- `monitoring-network` — prometheus, grafana, alertmanager, postgres-exporter, all services\n\n### Security\n\n- All ports bound to `127.0.0.1` (no external exposure)\n- Non-root container execution (UID 1000)\n- `tini` as PID 1 for proper signal handling\n- SCRAM-SHA-256 authentication (PostgreSQL + PGBouncer)\n- Healthchecks via `pg_isready` and `/metrics` HTTP endpoint\n\n---\n\n## Technology Stack\n\n| Category | Technologies |\n|----------|-------------|\n| Language | Python 3.11+ (fully typed, strict mypy) |\n| Database | PostgreSQL 18, asyncpg, PGBouncer |\n| Async | asyncio, aiohttp, aiohttp-socks |\n| Nostr | nostr-sdk (Rust FFI via UniFFI) |\n| Web Framework | FastAPI, uvicorn |\n| Validation | Pydantic v2, rfc3986 |\n| Monitoring | Prometheus, Grafana, Alertmanager, structured logging |\n| Networking | dnspython, geoip2, geohash2, tldextract, cryptography |\n| Testing | pytest, pytest-asyncio, pytest-cov, testcontainers |\n| Quality | ruff (lint+format), mypy (strict), pre-commit (23 hooks) |\n| CI/CD | GitHub Actions, uv-secure, Trivy, CodeQL, Dependabot |\n| Containers | Docker, Docker Compose, tini |\n| Build | uv (dependency management + build) |\n\n---\n\n## Documentation\n\nFull documentation is available at **[bigbrotr.github.io/bigbrotr](https://bigbrotr.github.io/bigbrotr/)**.\n\n| Section | Description |\n|---------|-------------|\n| [Getting Started](https://bigbrotr.github.io/bigbrotr/getting-started/) | Installation, quick start tutorial, first deployment |\n| [User Guide](https://bigbrotr.github.io/bigbrotr/user-guide/) | Architecture, services, configuration, database, monitoring |\n| [How-to Guides](https://bigbrotr.github.io/bigbrotr/how-to/) | Docker deploy, manual deploy, Tor setup, troubleshooting |\n| [Development](https://bigbrotr.github.io/bigbrotr/development/) | Setup, testing, contributing |\n| [API Reference](https://bigbrotr.github.io/bigbrotr/reference/) | Auto-generated Python API docs |\n| [Changelog](CHANGELOG.md) | Version history and migration guides |\n\n---\n\n## Contributing\n\nSee the [Contributing Guide](https://bigbrotr.github.io/bigbrotr/development/contributing/) for detailed instructions.\n\n1. Fork and clone\n2. `uv sync --group dev` and `pre-commit install`\n3. Write tests for new functionality\n4. `make ci` — all checks must pass\n5. Submit a pull request\n\nConventional commits: `feat:`, `fix:`, `refactor:`, `docs:`, `test:`, `chore:`\n\n---\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n\n---\n\n## Links\n\n- [Full Documentation](https://bigbrotr.github.io/bigbrotr/)\n- [Changelog](CHANGELOG.md)\n- [Nostr Protocol](https://nostr.com)\n- [NIP-11: Relay Information Document](https://github.com/nostr-protocol/nips/blob/master/11.md)\n- [NIP-42: Authentication of Clients to Relays](https://github.com/nostr-protocol/nips/blob/master/42.md)\n- [NIP-66: Relay Discovery and Monitoring](https://github.com/nostr-protocol/nips/blob/master/66.md)\n- [NIP-89: Recommended Application Handlers](https://github.com/nostr-protocol/nips/blob/master/89.md)\n- [NIP-90: Data Vending Machines](https://github.com/nostr-protocol/nips/blob/master/90.md)\n- [NIPs Repository](https://github.com/nostr-protocol/nips)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbigbrotr%2Fbigbrotr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbigbrotr%2Fbigbrotr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbigbrotr%2Fbigbrotr/lists"}