{"id":31557003,"url":"https://github.com/ampyfin/ampy-config","last_synced_at":"2025-10-04T23:21:14.159Z","repository":{"id":313688350,"uuid":"1052216025","full_name":"AmpyFin/ampy-config","owner":"AmpyFin","description":"Typed config + secrets façade for AmpyFin: schema-validated, layered, redacted, and hot-reloaded via NATS/JetStream (Python + optional Go).","archived":false,"fork":false,"pushed_at":"2025-09-12T18:55:08.000Z","size":15269,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-14T23:20:21.873Z","etag":null,"topics":["aws-secrets-manager","configuration-management","control-plane","devops","gcp-secret-manager","hashicorp-vault","hot-reload","jsonschema","observability","pydantic","runtime-configuration","secrets-management","trading-systems"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/ampy-config/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AmpyFin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-07T16:36:16.000Z","updated_at":"2025-09-12T18:35:04.000Z","dependencies_parsed_at":"2025-09-07T21:27:31.964Z","dependency_job_id":"ac14ea30-0081-45cb-a919-ccbd1d12ed61","html_url":"https://github.com/AmpyFin/ampy-config","commit_stats":null,"previous_names":["ampyfin/ampy-config"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AmpyFin/ampy-config","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmpyFin%2Fampy-config","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmpyFin%2Fampy-config/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmpyFin%2Fampy-config/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmpyFin%2Fampy-config/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AmpyFin","download_url":"https://codeload.github.com/AmpyFin/ampy-config/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmpyFin%2Fampy-config/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278386651,"owners_count":25978217,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-secrets-manager","configuration-management","control-plane","devops","gcp-secret-manager","hashicorp-vault","hot-reload","jsonschema","observability","pydantic","runtime-configuration","secrets-management","trading-systems"],"created_at":"2025-10-04T23:21:13.039Z","updated_at":"2025-10-04T23:21:14.153Z","avatar_url":"https://github.com/AmpyFin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# 🚀 ampy-config\n\n**Typed Configuration \u0026 Secrets Façade for AmpyFin**\n\n[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)\n[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)\n[![Tests](https://github.com/AmpyFin/ampy-config/workflows/ci/badge.svg)](https://github.com/AmpyFin/ampy-config/actions)\n[![PyPI](https://img.shields.io/pypi/v/ampy-config.svg)](https://pypi.org/project/ampy-config/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n\u003e 🎯 **Single, safe source of truth** for configuration and secrets across AmpyFin services  \n\u003e 🔗 Built to integrate with **ampy-bus** (control plane over NATS/JetStream) and **ampy-proto** (payload contracts)\n\n[📖 Documentation](#-highlights-what-you-get) • [🚀 Quick Start](#-install-python--pypi) • [🔧 Usage](#-cli-usage) • [🤝 Contributing](#-contributing)\n\n\u003c/div\u003e\n\n---\n\n## 📋 Table of Contents\n\n- [🎯 Why this exists](#-why-this-exists-the-problem)\n- [✨ Highlights](#-highlights-what-you-get)\n- [🚀 Quick Start](#-quick-start)\n- [🚀 Install](#-install)\n- [🎯 Basic Usage Examples](#-basic-usage-examples)\n- [🔗 Integration Examples](#-integration-examples)\n- [✅ Configuration Validation](#-configuration-validation)\n- [🎮 Control Plane](#-control-plane-natsjetstream)\n- [📚 Layering Model](#-layering-model)\n- [🔐 Secrets](#-secrets-indirection-caching-rotation-redaction)\n- [💻 CLI Usage](#-cli-usage)\n- [🐍 Python Integration](#-use-from-a-service-python-example)\n- [🐹 Go Client Usage](#-go-client-usage)\n- [📊 Schema Examples](#-schema-notes-metrics-example)\n- [🌍 Environment Variables](#-environment-variables)\n- [🔧 Troubleshooting](#-troubleshooting)\n- [🛡️ Security](#-security-notes)\n- [🤝 Contributing](#-contributing)\n\n---\n\n## 🎯 Why this exists (the problem)\n\nWithout a unified configuration layer, distributed trading systems tend to develop:\n\n\u003e ⚠️ **Common Issues:**\n\u003e - **ENV/YAML sprawl** → drift, surprises, outages\n\u003e - **Secret handling risks** → credentials in logs, brittle rotations, no redaction\n\u003e - **Non-reproducibility** → can't reconstruct exactly which parameters were live for a given trade/run\n\u003e - **Inconsistent runtime behavior** → some services reload, others require restarts\n\n**ampy-config** provides a single, typed, validated, observable configuration view with clean secret indirection and a runtime control plane for safe updates.\n\n---\n\n## ✨ Highlights (what you get)\n\n| Feature | Description |\n|---------|-------------|\n| 🔍 **Typed schema + validation** | JSON Schema + semantic cross-field checks |\n| 📚 **Layering \u0026 precedence** | defaults → environment profile → overlays → ENV allowlist → runtime overrides |\n| 🔐 **Secret indirection** | `secret://…`, `aws-sm://…`, `gcp-sm://…` with caching, rotation, and universal redaction |\n| 🎮 **Control plane for updates** | `config_preview` → `config_apply` → `config_applied` events on NATS (JetStream) |\n| 📊 **Auditability \u0026 observability** | provenance for each key; logs/metrics/traces (no secrets) |\n| 🌐 **Language-agnostic** | produces plain YAML effective config for Python, Go, C++, etc. |\n\n---\n\n## 🚀 Quick Start\n\n### ⚠️ Prerequisites - NATS Server Required\n\n**ampy-config requires a running NATS server for configuration management:**\n\n```bash\n# Start NATS with JetStream (required)\ndocker run --rm -d --name nats -p 4222:4222 nats:2.10 -js\n\n# Or install NATS server locally\ngo install github.com/nats-io/nats-server/v2@latest\nnats-server -js\n```\n\n**Verify NATS is running:**\n```bash\n# Test connection\nnats --server \"nats://localhost:4222\" server info\n```\n\n\u003e 🚨 **Common Issue**: If you get `nats: no servers available for connection` error, NATS server is not running. Start it with the command above.\n\n## 🚀 Install\n\n### 🐍 Python / PyPI\n\n```bash\npip install ampy-config\n```\n\n**Developer mode** (local repo):\n```bash\npip install -e .\n```\n\n### 🐹 Go Client\n\n**Library:**\n```bash\ngo get github.com/AmpyFin/ampy-config/go/ampyconfig@v1.1.5\n```\n\n**Binaries:**\n```bash\ncd go/ampyconfig\nmake     # builds bin/ampyconfig-{ops,agent,listener}\n```\n\n\u003e 📦 **Available on [pkg.go.dev](https://pkg.go.dev/github.com/AmpyFin/ampy-config/go/ampyconfig)**\n\n### 🔧 Optional secret backends\n\n| Backend | Install Command | Use Case |\n|---------|----------------|----------|\n| 🔐 **HashiCorp Vault** | `pip install hvac` | Enterprise secret management |\n| ☁️ **AWS Secrets Manager** | `pip install boto3` | AWS-native secret storage |\n| 🌐 **GCP Secret Manager** | `pip install google-cloud-secret-manager` | Google Cloud secret storage |\n\n\u003e 💡 **Tip:** You **do not** need to sign up for all of these. Choose one or more real backends for your deployment; the library gracefully falls back to a local JSON file in development.\n\n---\n\n## 🎯 Basic Usage Examples\n\n### 🐍 Python Quick Start\n\n```python\nimport asyncio\nfrom ampy_config.layering import build_effective_config\n\n# Build effective configuration\ncfg, _ = build_effective_config(\n    schema_path=\"schema/ampy-config.schema.json\",\n    defaults_path=\"config/defaults.yaml\",\n    profile_yaml=\"examples/dev.yaml\",\n    overlays=[],\n    service_overrides=[],\n    env_allowlist_path=\"env_allowlist.txt\",\n    env_file=None,\n    runtime_overrides_path=\"runtime/overrides.yaml\",\n)\n\n# Get configuration values\nnats_url = cfg[\"bus\"][\"nats_url\"]\ntopic_prefix = cfg[\"bus\"][\"topic_prefix\"]\nrisk_limit = cfg[\"oms\"][\"risk\"][\"max_order_notional_usd\"]\n\nprint(f\"NATS URL: {nats_url}\")\nprint(f\"Topic Prefix: {topic_prefix}\")\nprint(f\"Risk Limit: {risk_limit}\")\n```\n\n### 🐹 Go Quick Start\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"log\"\n    \n    \"github.com/AmpyFin/ampy-config/go/ampyconfig\"\n    \"github.com/nats-io/nats.go\"\n)\n\nfunc main() {\n    // Connect to NATS\n    nc, err := nats.Connect(\"nats://localhost:4222\")\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer nc.Close()\n    \n    // Create ampy-config client\n    client := ampyconfig.New(\"nats://localhost:4222\", \"ampy.dev\", \"runtime/overrides.yaml\")\n    \n    // Get configuration values\n    natsURL, err := client.Get(\"bus.nats_url\")\n    if err != nil {\n        log.Fatal(err)\n    }\n    \n    topicPrefix, err := client.Get(\"bus.topic_prefix\")\n    if err != nil {\n        log.Fatal(err)\n    }\n    \n    fmt.Printf(\"NATS URL: %s\\n\", natsURL)\n    fmt.Printf(\"Topic Prefix: %s\\n\", topicPrefix)\n}\n```\n\n### 🧪 Test Your Setup\n\n**Python validation test:**\n```bash\npython -c \"\nfrom ampy_config.layering import build_effective_config\ntry:\n    cfg, _ = build_effective_config(\n        'schema/ampy-config.schema.json',\n        'config/defaults.yaml',\n        'examples/dev.yaml',\n        [], [], 'env_allowlist.txt', None, 'runtime/overrides.yaml'\n    )\n    print('✅ ampy-config working correctly!')\n    print(f'Bus config: {cfg[\\\"bus\\\"]}')\nexcept Exception as e:\n    print(f'❌ Error: {e}')\n\"\n```\n\n**Go validation test:**\n```bash\ngo run - \u003c\u003c 'EOF'\npackage main\n\nimport (\n    \"fmt\"\n    \"log\"\n    \n    \"github.com/AmpyFin/ampy-config/go/ampyconfig\"\n)\n\nfunc main() {\n    client := ampyconfig.New(\"nats://localhost:4222\", \"ampy.dev\", \"runtime/overrides.yaml\")\n    \n    // Test getting a value\n    value, err := client.Get(\"test.key\")\n    if err != nil {\n        log.Printf(\"Expected error for missing key: %v\", err)\n    }\n    \n    fmt.Println(\"✅ ampy-config working correctly!\")\n}\nEOF\n```\n\n---\n\n## 🔗 Integration Examples\n\n### With ampy-bus\n\n**Python integration:**\n```python\nimport asyncio\nfrom ampy_config.layering import build_effective_config\nfrom ampy_config.bus.ampy_bus import AmpyBus\n\nasync def create_bus_from_config():\n    # Build configuration\n    cfg, _ = build_effective_config(\n        schema_path=\"schema/ampy-config.schema.json\",\n        defaults_path=\"config/defaults.yaml\",\n        profile_yaml=\"examples/dev.yaml\",\n        overlays=[], service_overrides=[],\n        env_allowlist_path=\"env_allowlist.txt\",\n        env_file=None,\n        runtime_overrides_path=\"runtime/overrides.yaml\",\n    )\n    \n    # Get bus configuration\n    nats_url = cfg[\"bus\"][\"nats_url\"]\n    stream_name = cfg[\"bus\"][\"stream_name\"]\n    topic_prefix = cfg[\"bus\"][\"topic_prefix\"]\n    \n    # Create bus\n    bus = AmpyBus(nats_url)\n    await bus.connect()\n    \n    return bus, topic_prefix\n```\n\n**Go integration:**\n```go\nimport (\n    \"github.com/AmpyFin/ampy-config/go/ampyconfig\"\n    \"github.com/AmpyFin/ampy-bus/pkg/ampybus/natsbinding\"\n)\n\nfunc createBusFromConfig() (*natsbinding.Bus, error) {\n    // Create config client\n    client := ampyconfig.New(\"nats://localhost:4222\", \"ampy.dev\", \"runtime/overrides.yaml\")\n    \n    // Get configuration values\n    natsURL, err := client.Get(\"bus.nats_url\")\n    if err != nil {\n        return nil, err\n    }\n    \n    streamName, err := client.Get(\"bus.stream_name\")\n    if err != nil {\n        return nil, err\n    }\n    \n    topicPrefix, err := client.Get(\"bus.topic_prefix\")\n    if err != nil {\n        return nil, err\n    }\n    \n    // Create bus configuration\n    config := natsbinding.Config{\n        URLs:          []string{natsURL},\n        StreamName:    streamName,\n        Subjects:      []string{topicPrefix + \".\u003e\"},\n        DurablePrefix: \"ampy-trading\",\n    }\n    \n    return natsbinding.NewBus(config)\n}\n```\n\n### With ampy-proto\n\n**Go integration:**\n```go\nimport (\n    \"github.com/AmpyFin/ampy-config/go/ampyconfig\"\n    bars \"github.com/AmpyFin/ampy-proto/v2/gen/go/ampy/bars/v1\"\n    common \"github.com/AmpyFin/ampy-proto/v2/gen/go/ampy/common/v1\"\n)\n\nfunc createBarFromConfig(client *ampyconfig.Client) (*bars.Bar, error) {\n    // Get trading configuration\n    symbol, err := client.Get(\"trading.symbol\")\n    if err != nil {\n        return nil, err\n    }\n    \n    mic, err := client.Get(\"trading.mic\")\n    if err != nil {\n        return nil, err\n    }\n    \n    // Create bar with config values\n    bar := \u0026bars.Bar{\n        Security: \u0026common.SecurityId{\n            Symbol: symbol,\n            Mic:    mic,\n        },\n        // ... other fields\n    }\n    \n    return bar, nil\n}\n```\n\n### Dynamic Configuration Updates\n\n**Python:**\n```python\nasync def setup_config_updates(bus, topic_prefix):\n    async def on_config_apply(subject, data):\n        print(f\"Config applied: {data}\")\n        # Reload your application configuration\n        reload_config()\n    \n    async def on_config_preview(subject, data):\n        print(f\"Config preview: {data}\")\n    \n    # Subscribe to configuration events\n    await bus.subscribe_json(f\"{topic_prefix}.control.v1.config_apply\", on_config_apply)\n    await bus.subscribe_json(f\"{topic_prefix}.control.v1.config_preview\", on_config_preview)\n\ndef reload_config():\n    # Reload your application configuration\n    print(\"Reloading configuration...\")\n```\n\n**Go:**\n```go\nfunc setupConfigUpdates(client *ampyconfig.Client) error {\n    // Subscribe to configuration changes\n    err := client.Subscribe(func(event ampyconfig.ConfigEvent) {\n        switch event.Type {\n        case \"config_preview\":\n            fmt.Printf(\"Config preview: %s\\n\", event.Data)\n        case \"config_apply\":\n            fmt.Printf(\"Config applied: %s\\n\", event.Data)\n            // Reload configuration\n            reloadConfig()\n        }\n    })\n    \n    return err\n}\n\nfunc reloadConfig() {\n    // Reload your application configuration\n    fmt.Println(\"Reloading configuration...\")\n}\n```\n\n---\n\n## ✅ Configuration Validation\n\n### Required Configuration Keys\n\n**Python validation:**\n```python\ndef validate_config(cfg):\n    \"\"\"Validate that all required configuration keys are present\"\"\"\n    required_keys = [\n        \"bus.nats_url\",\n        \"bus.stream_name\", \n        \"bus.topic_prefix\",\n        \"oms.risk.max_order_notional_usd\",\n        \"trading.symbol\",\n        \"trading.mic\",\n    ]\n    \n    missing_keys = []\n    for key_path in required_keys:\n        keys = key_path.split('.')\n        current = cfg\n        try:\n            for key in keys:\n                current = current[key]\n        except (KeyError, TypeError):\n            missing_keys.append(key_path)\n    \n    if missing_keys:\n        raise ValueError(f\"Missing required config keys: {missing_keys}\")\n    \n    return True\n\n# Usage\ntry:\n    cfg, _ = build_effective_config(...)\n    validate_config(cfg)\n    print(\"✅ Configuration validation passed\")\nexcept ValueError as e:\n    print(f\"❌ Configuration validation failed: {e}\")\n```\n\n**Go validation:**\n```go\nfunc validateConfig(client *ampyconfig.Client) error {\n    requiredKeys := []string{\n        \"bus.nats_url\",\n        \"bus.stream_name\",\n        \"bus.topic_prefix\",\n        \"trading.symbol\",\n        \"trading.mic\",\n    }\n    \n    for _, key := range requiredKeys {\n        if _, err := client.Get(key); err != nil {\n            return fmt.Errorf(\"missing required config key: %s\", key)\n        }\n    }\n    \n    return nil\n}\n\n// Usage\nif err := validateConfig(client); err != nil {\n    log.Fatalf(\"Configuration validation failed: %v\", err)\n}\n```\n\n### Schema Validation\n\n**Validate configuration files:**\n```bash\n# Validate single file\npython tools/validate.py examples/dev.yaml\n\n# Validate multiple files\npython tools/validate.py examples/*.yaml\n\n# Validate with explicit schema\npython tools/validate.py --schema schema/ampy-config.schema.json examples/dev.yaml\n```\n\n**Programmatic validation:**\n```python\nfrom ampy_config.layering import build_effective_config\n\ntry:\n    cfg, provenance = build_effective_config(\n        schema_path=\"schema/ampy-config.schema.json\",  # Enables schema validation\n        defaults_path=\"config/defaults.yaml\",\n        profile_yaml=\"examples/dev.yaml\",\n        overlays=[],\n        service_overrides=[],\n        env_allowlist_path=\"env_allowlist.txt\",\n        env_file=None,\n        runtime_overrides_path=\"runtime/overrides.yaml\",\n    )\n    print(\"✅ Schema validation passed\")\nexcept Exception as e:\n    print(f\"❌ Schema validation failed: {e}\")\n```\n\n### Configuration File Examples\n\n**defaults.yaml:**\n```yaml\n# Default configuration values\nbus:\n  nats_url: \"nats://localhost:4222\"\n  stream_name: \"AMPY_TRADING\"\n  topic_prefix: \"ampy.dev\"\n  durable_prefix: \"ampy-trading\"\n\ntrading:\n  symbol: \"AAPL\"\n  mic: \"XNAS\"\n  risk_limit: 10000\n  position_limit: 1000\n\nsignals:\n  ma_short_period: 10\n  ma_long_period: 20\n  threshold: 0.5\n\nlogging:\n  level: \"info\"\n  format: \"json\"\n```\n\n**development.yaml:**\n```yaml\n# Development overrides\nbus:\n  nats_url: \"nats://localhost:4222\"\n  topic_prefix: \"ampy.dev\"\n\ntrading:\n  symbol: \"AAPL\"\n  risk_limit: 1000  # Lower limit for dev\n\nlogging:\n  level: \"debug\"\n  format: \"text\"\n```\n\n**production.yaml:**\n```yaml\n# Production overrides\nbus:\n  nats_url: \"nats://prod-nats:4222\"\n  topic_prefix: \"ampy.prod\"\n\ntrading:\n  risk_limit: 100000\n  position_limit: 10000\n\nlogging:\n  level: \"warn\"\n  format: \"json\"\n```\n\n**runtime/overrides.yaml:**\n```yaml\n# Runtime dynamic overrides (can be updated via NATS)\ntrading:\n  risk_limit: 5000  # Updated during runtime\n\nsignals:\n  threshold: 0.7    # Updated during runtime\n```\n\n---\n\n## 🎮 Control plane (NATS/JetStream)\n\n**Start a local NATS with JetStream:**\n```bash\ndocker run --rm -d --name nats -p 4222:4222 nats:2.10 -js\nexport NATS_URL=\"nats://127.0.0.1:4222\"\n```\n\n**Provision the stream and durable consumers** (once). Using the `nats` CLI:\n\n```bash\n# Stream to cover all control-plane subjects\nnats --server \"$NATS_URL\" stream add ampy-control \\\n  --subjects \"ampy.*.control.v1.*\" \\\n  --retention limits --max-age 24h --storage file \\\n  --max-msgs 10000 --max-bytes 100MB --discard old --defaults\n\n# Agent durables (pull + explicit ack)\nnats --server \"$NATS_URL\" consumer add ampy-control ampy-config-agent-ampy-dev-control-v1-config-preview \\\n  --filter \"ampy.dev.control.v1.config_preview\" --pull --deliver all --ack explicit --defaults\nnats --server \"$NATS_URL\" consumer add ampy-control ampy-config-agent-ampy-dev-control-v1-config-apply \\\n  --filter \"ampy.dev.control.v1.config_apply\" --pull --deliver all --ack explicit --defaults\nnats --server \"$NATS_URL\" consumer add ampy-control ampy-config-agent-ampy-dev-control-v1-secret-rotated \\\n  --filter \"ampy.dev.control.v1.secret_rotated\" --pull --deliver all --ack explicit --defaults\n```\n\n**Verify setup:**\n```bash\nnats --server \"$NATS_URL\" stream ls\nnats --server \"$NATS_URL\" consumer ls ampy-control\n```\n\n\u003e ⚡ **Note:** The library can also auto-provision if permitted, but explicit creation is more predictable for local dev and CI.\n\n---\n\n## 📚 Layering model\n\nEffective config = **merge** in this order (later overrides earlier):\n\n```mermaid\ngraph TD\n    A[📄 Defaults\u003cbr/\u003econfig/defaults.yaml] --\u003e B[🌍 Environment Profile\u003cbr/\u003eexamples/dev.yaml]\n    B --\u003e C[📋 Overlays\u003cbr/\u003e--overlay path]\n    C --\u003e D[🔧 ENV Allowlist\u003cbr/\u003eenv_allowlist.txt]\n    D --\u003e E[⚡ Runtime Overrides\u003cbr/\u003eruntime/overrides.yaml]\n    E --\u003e F[✅ Final Config]\n```\n\n| Layer | Description | Example |\n|-------|-------------|---------|\n| 1️⃣ **Defaults** | Checked-in base config | `config/defaults.yaml` |\n| 2️⃣ **Environment profile** | Environment-specific settings | `examples/dev.yaml`, `examples/paper.yaml`, `examples/prod.yaml` |\n| 3️⃣ **Overlays** | Region/cluster/service YAMLs | `--overlay path` (repeatable) |\n| 4️⃣ **ENV allowlist** | Environment variable mapping | `env_allowlist.txt` maps allowed env keys |\n| 5️⃣ **Runtime overrides** | Live configuration updates | `runtime/overrides.yaml` (written by agent) |\n\nEach key tracks **provenance**: where it came from (defaults/profile/overlay/ENV/runtime).\n\n### 📏 Units \u0026 types\n\n| Type | Format | Examples |\n|------|--------|----------|\n| ⏱️ **Durations** | String format | `150ms`, `2s`, `5m`, `1h` |\n| 📊 **Sizes** | String format | `128KiB`, `1MiB` |\n| 🏷️ **Domains** | Explicit prefixes | `oms.*`, `ingest.*`, `broker.*`, `ml.*`, `warehouse.*`, `fx.*`, `metrics`, `logging`, `tracing`, `security.*`, `feature_flags.*` |\n\n---\n\n## 🔐 Secrets (indirection, caching, rotation, redaction)\n\nUse **references**, not literal values:\n\n| Backend | Format | Example |\n|---------|--------|---------|\n| 🔐 **Vault** | `secret://vault/\u003cpath\u003e#\u003ckey\u003e` | `secret://vault/tiingo#token` |\n| ☁️ **AWS SM** | `aws-sm://\u003cname\u003e?versionStage=AWSCURRENT` | `aws-sm://ALPACA_SECRET?versionStage=AWSCURRENT` |\n| 🌐 **GCP SM** | `gcp-sm://projects/\u003cproject\u003e/secrets/\u003cname\u003e/versions/latest` | `gcp-sm://projects/demo/secrets/AMPY_API/versions/latest` |\n\n**Local development fallback file** (`.secrets.local.json`):\n```json\n{\n  \"secret://vault/tiingo#token\": \"TIINGO_LOCAL_DEV_TOKEN\",\n  \"aws-sm://ALPACA_SECRET?versionStage=AWSCURRENT\": \"ALPACA_LOCAL_DEV_SECRET\",\n  \"gcp-sm://projects/demo/secrets/AMPY_API/versions/latest\": \"AMPY_LOCAL_DEV_API\"\n}\n```\n\n\u003e 🔒 **Security:** Secrets are **always redacted** in logs/metrics/traces; rotation is signaled via `secret_rotated` events.\n\n---\n\n## 💻 CLI usage\n\nAll commands are available via `python -m ampy_config.cli …` (works without global entrypoints).\n\n### 🎨 Render effective config\n\n```bash\npython -m ampy_config.cli render \\\n  --profile dev \\\n  --resolve-secrets redacted \\\n  --provenance\n```\n\n**Write it to a file:**\n```bash\npython -m ampy_config.cli render \\\n  --profile dev \\\n  --resolve-secrets redacted \\\n  --output /tmp/effective.yaml\n```\n\n**Resolve values** (dev only; requires `.secrets.local.json` or configured backends):\n```bash\nAMPY_CONFIG_LOCAL_SECRETS=.secrets.local.json \\\npython -m ampy_config.cli render --profile dev --resolve-secrets values\n```\n\n### ✅ Validate (schema + semantic checks)\n\n```bash\npython tools/validate.py examples/dev.yaml\n# Or explicitly:\npython tools/validate.py --schema schema/ampy-config.schema.json examples/*.yaml\n```\n\n### 🔧 Secrets utilities\n\n```bash\n# Resolve (redacted by default)\npython -m ampy_config.cli secret get \"aws-sm://ALPACA_SECRET?versionStage=AWSCURRENT\"\n\n# Print plain (development only)\npython -m ampy_config.cli secret get --plain \"secret://vault/tiingo#token\"\n\n# Invalidate cache entry\npython -m ampy_config.cli secret rotate \"gcp-sm://projects/demo/secrets/AMPY_API/versions/latest\"\n```\n\n### 🤖 Run the agent\n\n```bash\nexport NATS_URL=\"nats://127.0.0.1:4222\"\nexport AMPY_CONFIG_SERVICE=\"ampy-config-agent\"\n\npython -m ampy_config.cli agent --profile dev\n```\n\n**It subscribes to:**\n```\nampy.dev.control.v1.config_preview\nampy.dev.control.v1.config_apply\nampy.dev.control.v1.secret_rotated\n```\n\n### ⚡ Ops: preview \u0026 apply a runtime override\n\n**Create an overlay:**\n```bash\ncat \u003e/tmp/overlay.yaml \u003c\u003c'YAML'\noms:\n  risk:\n    max_order_notional_usd: 77777\nYAML\n```\n\n**Preview** (validate only):\n```bash\npython -m ampy_config.cli ops preview \\\n  --profile dev \\\n  --overlay-file /tmp/overlay.yaml \\\n  --expires-at \"2025-12-31T23:59:59Z\" \\\n  --reason \"intraday risk tightening\" \\\n  --dry-run\n```\n\n**Apply** (persist) and **wait** until it's effective in the resolved view:\n```bash\npython -m ampy_config.cli ops apply \\\n  --profile dev \\\n  --overlay-file /tmp/overlay.yaml \\\n  --wait-applied --timeout 20\n```\n\n**Then verify:**\n```bash\npython -m ampy_config.cli render \\\n  --profile dev \\\n  --runtime runtime/overrides.yaml \\\n  --resolve-secrets redacted \\\n  --provenance\n```\n\n---\n\n## 🐍 Use from a service (Python example)\n\n```python\n# examples/service_skel.py\nimport asyncio, os\nfrom ampy_config.layering import build_effective_config\nfrom ampy_config.bus.ampy_bus import AmpyBus\nfrom ampy_config.control.events import subjects\n\nasync def main():\n    cfg, _ = build_effective_config(\n        schema_path=\"schema/ampy-config.schema.json\",\n        defaults_path=\"config/defaults.yaml\",\n        profile_yaml=\"examples/dev.yaml\",\n        overlays=[],\n        service_overrides=[],\n        env_allowlist_path=\"env_allowlist.txt\",\n        env_file=None,\n        runtime_overrides_path=\"runtime/overrides.yaml\",\n    )\n    print(\"[service] max_order_notional_usd =\", cfg[\"oms\"][\"risk\"][\"max_order_notional_usd\"])\n\n    bus = AmpyBus(os.environ.get(\"NATS_URL\"))\n    await bus.connect()\n    subs = subjects(cfg[\"bus\"][\"topic_prefix\"])\n\n    async def on_apply(subject, data):\n        # Re-build after apply; in real code, you’d update state atomically \u0026 validate\n        new_cfg, _ = build_effective_config(\n            \"schema/ampy-config.schema.json\",\n            \"config/defaults.yaml\",\n            \"examples/dev.yaml\",\n            [], [], \"env_allowlist.txt\", None, \"runtime/overrides.yaml\"\n        )\n        print(\"[service] updated max_order_notional_usd =\", new_cfg[\"oms\"][\"risk\"][\"max_order_notional_usd\"])\n\n    await bus.subscribe_json(subs[\"apply\"], on_apply)\n    while True:\n        await asyncio.sleep(1)\n\nif __name__ == \"__main__\":\n    os.environ.setdefault(\"AMPY_CONFIG_SERVICE\", \"ampy-service-demo\")\n    os.environ.setdefault(\"NATS_URL\", \"nats://127.0.0.1:4222\")\n    asyncio.run(main())\n```\n\n### 🐹 Go Client Usage\n\n#### 📚 API Reference\n\n**Core Types:**\n```go\n// Client structure\ntype Client struct {\n    // ... internal fields\n}\n\n// Configuration event\ntype ConfigEvent struct {\n    Type string\n    Data string\n}\n\n// Configuration methods\nfunc New(natsURL, topicPrefix, configPath string) *Client\nfunc (c *Client) Get(key string) (string, error)\nfunc (c *Client) GetAll() (map[string]interface{}, error)\nfunc (c *Client) Subscribe(handler func(ConfigEvent)) error\nfunc (c *Client) Close() error\n```\n\n**Key Methods:**\n```go\n// Create client\nfunc New(natsURL, topicPrefix, configPath string) *Client\n\n// Get configuration value\nfunc (c *Client) Get(key string) (string, error)\n\n// Get all configuration\nfunc (c *Client) GetAll() (map[string]interface{}, error)\n\n// Subscribe to config events\nfunc (c *Client) Subscribe(handler func(ConfigEvent)) error\n\n// Close client\nfunc (c *Client) Close() error\n```\n\n#### 🚀 Basic Usage\n\n**Create and use client:**\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"log\"\n    \n    \"github.com/AmpyFin/ampy-config/go/ampyconfig\"\n)\n\nfunc main() {\n    // Create ampy-config client\n    client := ampyconfig.New(\"nats://localhost:4222\", \"ampy.dev\", \"runtime/overrides.yaml\")\n    defer client.Close()\n    \n    // Get configuration values\n    natsURL, err := client.Get(\"bus.nats_url\")\n    if err != nil {\n        log.Fatal(err)\n    }\n    \n    topicPrefix, err := client.Get(\"bus.topic_prefix\")\n    if err != nil {\n        log.Fatal(err)\n    }\n    \n    fmt.Printf(\"NATS URL: %s\\n\", natsURL)\n    fmt.Printf(\"Topic Prefix: %s\\n\", topicPrefix)\n}\n```\n\n**Subscribe to configuration changes:**\n```go\nfunc setupConfigUpdates(client *ampyconfig.Client) error {\n    // Subscribe to configuration changes\n    err := client.Subscribe(func(event ampyconfig.ConfigEvent) {\n        switch event.Type {\n        case \"config_preview\":\n            fmt.Printf(\"Config preview: %s\\n\", event.Data)\n        case \"config_apply\":\n            fmt.Printf(\"Config applied: %s\\n\", event.Data)\n            // Reload configuration\n            reloadConfig()\n        }\n    })\n    \n    return err\n}\n\nfunc reloadConfig() {\n    // Reload your application configuration\n    fmt.Println(\"Reloading configuration...\")\n}\n```\n\n#### 🛠️ Command Line Tools\n\n**Start the agent:**\n```bash\n./bin/ampyconfig-agent \\\n  -nats \"$NATS_URL\" \\\n  -topic ampy/dev \\\n  -runtime runtime/overrides.yaml \\\n  -service ampy-config-agent \\\n  -log info\n```\n\n**Apply configuration changes:**\n```bash\ncat \u003e/tmp/overlay.yaml \u003c\u003c'YAML'\noms:\n  risk:\n    max_order_notional_usd: 123456\nYAML\n\n./bin/ampyconfig-ops \\\n  -nats \"$NATS_URL\" \\\n  -topic ampy/dev \\\n  -overlay-file /tmp/overlay.yaml \\\n  -wait-applied -timeout 20 \\\n  -runtime runtime/overrides.yaml\n```\n\n**Available binaries:**\n- `ampyconfig-ops` — publish `config_preview`, `config_apply`, `secret_rotated`\n- `ampyconfig-agent` — consume control events and persist `runtime/overrides.yaml`\n- `ampyconfig-listener` — example service listener that reacts to changes\n\n\u003e 📝 **Status:** v0 thin client — Python `ampy-config` remains the source of truth for schema validation and layering. This Go module focuses on control-plane parity and operational UX.\n\n### 🌐 Go / C++ services\n\n- Parse the **effective YAML** (rendered by ops at boot or on a schedule)\n- Subscribe to the same control-plane subjects and re-load your resolved config (or just read `runtime/overrides.yaml`) when a `config_apply` is observed\n- Keep reloads **transactional** for safety-critical domains\n\n---\n\n## 📊 Schema notes (metrics example)\n\nThe schema allows **either** OTLP (with endpoint) **or** Prometheus (with port):\n\n```json\n\"metrics\": {\n  \"type\": \"object\",\n  \"additionalProperties\": false,\n  \"properties\": {\n    \"exporter\": { \"type\": \"string\", \"enum\": [\"otlp\", \"prom\"] },\n    \"endpoint\": { \"type\": \"string\" },\n    \"sampling_ratio\": { \"type\": \"number\", \"minimum\": 0, \"maximum\": 1 },\n    \"port\": { \"type\": \"integer\", \"minimum\": 1, \"maximum\": 65535 }\n  },\n  \"required\": [\"exporter\"]\n}\n```\n\n**Examples:**\n\n```yaml\n# OTLP\nmetrics:\n  exporter: otlp\n  endpoint: https://otel.dev.ampyfin.com:4317\n  sampling_ratio: 0.25\n\n# Prometheus\nmetrics:\n  exporter: prom\n  port: 9464\n```\n\n---\n\n## 🌍 Environment variables\n\n| Variable | Description | Example |\n|----------|-------------|---------|\n| `NATS_URL` | NATS server URL | `nats://127.0.0.1:4222` |\n| `AMPY_CONFIG_SERVICE` | Logical service name (used to derive durable names) | `ampy-config-agent` |\n| `AMPY_CONFIG_RUNTIME_OVERRIDES` | Path for persisted runtime overrides | `runtime/overrides.yaml` |\n| `AMPY_CONFIG_LOCAL_SECRETS` | Path to local dev secrets JSON | `.secrets.local.json` |\n| `AMPY_CONFIG_SECRET_TTL_MS` | Secrets cache TTL in milliseconds | `120000` |\n| `AMPY_CONFIG_JS_FALLBACK` | Force direct NATS subscription fallback | `1` (skip JetStream) |\n\n**Secret Backend Variables:**\n- **🔐 Vault**: `VAULT_ADDR`, `VAULT_TOKEN` (if using `secret://`)\n- **☁️ AWS**: `AWS_DEFAULT_REGION` + credentials (if using `aws-sm://`)\n- **🌐 GCP**: `GOOGLE_APPLICATION_CREDENTIALS` (if using `gcp-sm://`)\n\n---\n\n## 🔧 Troubleshooting\n\n### 🚨 Common Issues \u0026 Solutions\n\n| Issue | Cause | Solution |\n|-------|-------|----------|\n| **`nats: no servers available for connection`** | NATS server is not running | Start NATS: `docker run --rm -d --name nats -p 4222:4222 nats:2.10 -js` |\n| **`config key not found`** | Configuration key doesn't exist | Check your YAML files and ensure the key path is correct (e.g., `bus.nats_url`) |\n| **`yaml: unmarshal errors`** | Invalid YAML syntax | Validate YAML syntax in your configuration files |\n| **`permission denied`** | File permission issues | Check file permissions for configuration files and runtime directory |\n| **Configuration not updating** | Not subscribing to config events | Ensure you're subscribing to `config_apply` events and handling updates |\n| **🤖 Agent only shows one subscription** | Blocked while initializing a secret backend | Unset or configure that backend properly, or run with only local secrets in dev |\n| **⏰ No messages consumed / timeouts** | `NATS_URL` points to wrong port, JetStream disabled | Check `NATS_URL`, enable JetStream, verify `ampy-control` stream \u0026 consumers exist |\n| **❌ Apply says OK but value didn't change** | Agent didn't write `runtime/overrides.yaml` | Verify file path via `AMPY_CONFIG_RUNTIME_OVERRIDES` and service reloads on `config_apply` |\n| **⚠️ Schema validation passes but semantic check fails** | Semantic checks run after schema validation | Fix the offending values called out in the error |\n\n### 🔍 Debug Mode\n\n**Python debug:**\n```python\n# Enable debug logging\nimport logging\nlogging.basicConfig(level=logging.DEBUG)\n\n# Add debug logging to your config loading\ndef debug_config(cfg):\n    if cfg.get(\"debug\", {}).get(\"enabled\") == True:\n        print(\"🔍 DEBUG: Configuration loaded successfully\")\n        print(f\"🔍 DEBUG: All config: {cfg}\")\n\n# Usage\ncfg, _ = build_effective_config(...)\ndebug_config(cfg)\n```\n\n**Go debug:**\n```go\n// Enable debug logging\nclient := ampyconfig.New(\"nats://localhost:4222\", \"ampy.dev\", \"runtime/overrides.yaml\")\n\n// Add debug logging\nfunc debugConfig(client *ampyconfig.Client) {\n    if value, err := client.Get(\"debug.enabled\"); err == nil \u0026\u0026 value == \"true\" {\n        fmt.Println(\"🔍 DEBUG: Configuration loaded successfully\")\n        \n        // Print all configuration\n        if allConfig, err := client.GetAll(); err == nil {\n            fmt.Printf(\"🔍 DEBUG: All config: %+v\\n\", allConfig)\n        }\n    }\n}\n```\n\n### 🧪 Quick Health Check\n\n**Test NATS connection:**\n```bash\n# Test NATS server\nnats --server \"nats://localhost:4222\" server info\n\n# Test JetStream\nnats --server \"nats://localhost:4222\" stream ls\n```\n\n**Test configuration loading:**\n```bash\n# Python\npython -c \"\nfrom ampy_config.layering import build_effective_config\ntry:\n    cfg, _ = build_effective_config(\n        'schema/ampy-config.schema.json',\n        'config/defaults.yaml', \n        'examples/dev.yaml',\n        [], [], 'env_allowlist.txt', None, 'runtime/overrides.yaml'\n    )\n    print('✅ Configuration loading works!')\nexcept Exception as e:\n    print(f'❌ Error: {e}')\n\"\n\n# Go\ngo run - \u003c\u003c 'EOF'\npackage main\nimport (\n    \"fmt\"\n    \"github.com/AmpyFin/ampy-config/go/ampyconfig\"\n)\nfunc main() {\n    client := ampyconfig.New(\"nats://localhost:4222\", \"ampy.dev\", \"runtime/overrides.yaml\")\n    fmt.Println(\"✅ ampy-config client created successfully!\")\n}\nEOF\n```\n\n---\n\n## 🛡️ Security notes\n\n- 🔒 **Secrets are never logged**; redaction is enforced throughout the library\n- 🚨 **Prefer fail-shut** for safety-critical domains (OMS risk, broker creds) and **fail-open** for low-risk knobs (metric sampling)\n- 🔐 **Ensure access to secret backends** is locked down with least privilege\n\n---\n\n## 🤝 Contributing\n\nPRs welcome! Please include tests for new config keys, validation rules, and control-plane flows.\n\n**Before submitting:**\n```bash\npytest -q\npython tools/validate.py examples/*.yaml\n```\n\n---\n\n## 📄 License\n\nApache-2.0 (proposed). See `LICENSE` for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fampyfin%2Fampy-config","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fampyfin%2Fampy-config","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fampyfin%2Fampy-config/lists"}