{"id":31581082,"url":"https://github.com/ampyfin/ampy-bus","last_synced_at":"2025-10-05T21:53:18.849Z","repository":{"id":313551741,"uuid":"1051808868","full_name":"AmpyFin/ampy-bus","owner":"AmpyFin","description":"Transport-agnostic messaging contracts \u0026 helpers for AmpyFin: Protobuf envelopes, NATS/Kafka bindings, DLQ/replay, metrics, Go/Python SDKs.","archived":false,"fork":false,"pushed_at":"2025-09-12T17:18:06.000Z","size":88543,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-25T04:53:01.038Z","etag":null,"topics":["dead-letter-queue","event-bus","event-driven-architecture","jetstream","kafka","market-data","nats","opentelemetry","red-panda","replay","trading"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/ampy-bus/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AmpyFin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-06T19:04:01.000Z","updated_at":"2025-09-12T17:18:05.000Z","dependencies_parsed_at":"2025-09-06T21:33:31.814Z","dependency_job_id":"e8955651-378d-4d86-a957-bd312062370e","html_url":"https://github.com/AmpyFin/ampy-bus","commit_stats":null,"previous_names":["ampyfin/ampy-bus"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/AmpyFin/ampy-bus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmpyFin%2Fampy-bus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmpyFin%2Fampy-bus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmpyFin%2Fampy-bus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmpyFin%2Fampy-bus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AmpyFin","download_url":"https://codeload.github.com/AmpyFin/ampy-bus/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmpyFin%2Fampy-bus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278526242,"owners_count":26001325,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-05T02:00:06.059Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dead-letter-queue","event-bus","event-driven-architecture","jetstream","kafka","market-data","nats","opentelemetry","red-panda","replay","trading"],"created_at":"2025-10-05T21:53:15.360Z","updated_at":"2025-10-05T21:53:18.838Z","avatar_url":"https://github.com/AmpyFin.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# 🚀 ampy-bus\n\n**Transport-Agnostic Messaging for Trading Systems**\n\n[![PyPI version](https://img.shields.io/pypi/v/ampy-bus?style=for-the-badge\u0026logo=pypi\u0026logoColor=white)](https://pypi.org/project/ampy-bus/)\n[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue?style=for-the-badge\u0026logo=python\u0026logoColor=white)](https://www.python.org/downloads/)\n[![Go 1.23+](https://img.shields.io/badge/go-1.23+-00ADD8?style=for-the-badge\u0026logo=go\u0026logoColor=white)](https://golang.org/)\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue?style=for-the-badge\u0026logo=apache\u0026logoColor=white)](https://opensource.org/licenses/Apache-2.0)\n\n[![Build Status](https://img.shields.io/badge/build-passing-brightgreen?style=for-the-badge\u0026logo=github-actions\u0026logoColor=white)](#)\n[![Test Coverage](https://img.shields.io/badge/coverage-85%25-brightgreen?style=for-the-badge\u0026logo=codecov\u0026logoColor=white)](#)\n[![Code Quality](https://img.shields.io/badge/quality-A-brightgreen?style=for-the-badge\u0026logo=sonarqube\u0026logoColor=white)](#)\n[![Security](https://img.shields.io/badge/security-scanned-brightgreen?style=for-the-badge\u0026logo=snyk\u0026logoColor=white)](#)\n\n[![NATS](https://img.shields.io/badge/NATS-JetStream-27ae60?style=for-the-badge\u0026logo=nats\u0026logoColor=white)](#)\n[![Kafka](https://img.shields.io/badge/Kafka-Compatible-231f20?style=for-the-badge\u0026logo=apache-kafka\u0026logoColor=white)](#)\n[![Protobuf](https://img.shields.io/badge/Protobuf-Enabled-4a90e2?style=for-the-badge\u0026logo=protobuf\u0026logoColor=white)](#)\n[![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-Enabled-000000?style=for-the-badge\u0026logo=opentelemetry\u0026logoColor=white)](#)\n\n---\n\n\u003e **🎯 Transport-agnostic messaging conventions and helpers** for AmpyFin trading systems.  \n\u003e Standardize topics, headers, QoS, replay, and observability across NATS and Kafka with consistent `ampy-proto` payloads.\n\n[📖 Documentation](#-documentation) • [🚀 Quick Start](#-quick-start) • [💡 Examples](#-complete-examples--use-cases) • [🤝 Contributing](#-contributing)\n\n## 🚨 Quick Reference - Common Gotchas\n\n\u003e **New to ampy-bus?** Read this first to avoid the most common issues!\n\n| Issue | ❌ Wrong | ✅ Correct |\n|-------|----------|------------|\n| **NATS Subjects** | `ampy.dev_bars_v1_XNAS_AAPL` | `ampy.dev.bars.v1.XNAS.AAPL` |\n| **Envelope Topic** | Missing `Topic` field | `Topic: \"ampy.dev.bars.v1.XNAS.AAPL\"` |\n| **JetStream** | `nats-server` (no `-js`) | `nats-server -js` |\n| **Error Handling** | Ignore `PublishEnvelope` errors | Always check `err` return value |\n\n**Most Common Errors:**\n- `nats: invalid subject` → Use dots, not underscores\n- `nats: no response from stream` → Set `Topic` field in envelope\n- `context deadline exceeded` → Start NATS with `-js` flag\n\n\u003c/div\u003e\n\n\n## ✨ Features\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\"\u003e\n\n### 🎯 **Transport Agnostic**\n- **NATS JetStream** \u0026 **Kafka** support\n- Same code works on both transports\n- Easy migration between brokers\n\n### 📊 **Trading System Ready**\n- Market data (bars, ticks, FX)\n- Trading operations (orders, fills, positions)\n- ML signals \u0026 news processing\n- System metrics \u0026 monitoring\n\n### 🔄 **Reliable Messaging**\n- At-least-once delivery\n- Dead letter queues (DLQ)\n- Message replay \u0026 backfill\n- Idempotent consumers\n\n\u003c/td\u003e\n\u003ctd width=\"50%\"\u003e\n\n### 📈 **Observability Built-in**\n- OpenTelemetry tracing\n- Prometheus metrics\n- Structured logging\n- Performance monitoring\n\n### 🛡️ **Production Ready**\n- TLS/mTLS encryption\n- Authentication \u0026 authorization\n- Schema validation\n- Error handling \u0026 retries\n\n### 🚀 **Developer Friendly**\n- CLI tools for testing\n- Python \u0026 Go libraries\n- Comprehensive examples\n- Clear documentation\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## 🚀 Quick Start\n\n### 1️⃣ **Start a Message Broker**\n\n```bash\n# Option A: NATS with JetStream (Recommended)\ndocker run -d --name nats -p 4222:4222 nats:2.10 -js\n\n# Option B: Kafka/Redpanda\ndocker run -d --name redpanda -p 9092:9092 -p 9644:9644 \\\n  redpandadata/redpanda:latest redpanda start --overprovisioned --smp 1 --memory 1G\n```\n\n### 2️⃣ **Install \u0026 Build Tools**\n\n```bash\n# Clone repository\ngit clone https://github.com/AmpyFin/ampy-bus.git\ncd ampy-bus\n\n# Build Go CLI tools\nmake build\n\n# Install Python package\npip install -e .[nats]\n```\n\n### 3️⃣ **Test Basic Pub/Sub**\n\n\u003e **⚠️ IMPORTANT**: Use dots (.) in topics, not underscores (_). This is critical for NATS JetStream to work properly.\n\n```bash\n# ✅ Publish a message (NATS) - CORRECT format with dots\n./ampybusctl pub-empty --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer yfinance-go@ingest-1 --source yfinance-go --pk XNAS.AAPL\n\n# ✅ Subscribe to messages - CORRECT format with dots\n./ampybusctl sub --subject \"ampy.prod.bars.v1.\u003e\"\n\n# ❌ WRONG - Don't use underscores like this:\n# ./ampybusctl pub-empty --topic ampy.prod_bars_v1_XNAS_AAPL\n\n# Kafka alternative\n./kafkabusctl pub-empty --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer yfinance-go@ingest-1 --source yfinance-go --pk XNAS.AAPL\n```\n\n**Validation Steps:**\n1. **Check NATS is running with JetStream:**\n   ```bash\n   docker logs nats | grep \"Starting JetStream\"\n   ```\n\n2. **Verify your topic format:**\n   ```bash\n   # ✅ Good: ampy.prod.bars.v1.XNAS.AAPL\n   # ❌ Bad:  ampy.prod_bars_v1_XNAS_AAPL\n   ```\n\n3. **Test the connection:**\n   ```bash\n   # This should work without errors\n   ./ampybusctl pub-empty --topic test.message --producer test@cli --source test --pk test\n   ```\n\n### 4️⃣ **Try Python Integration**\n\n```bash\n# Run Python example\npython python/examples/simple_roundtrip.py\n```\n\n## ⚠️ CRITICAL: Common Configuration Gotchas\n\n\u003e **🚨 IMPORTANT**: These configuration issues cause the most problems for new users. Read this section carefully before starting!\n\n### NATS Subject Patterns - Use Dots, Not Underscores!\n\n**❌ WRONG - This will fail:**\n```bash\n# Using underscores in subjects\n./ampybusctl pub-empty --topic ampy.dev_bars_v1_XNAS_AAPL\n```\n\n**✅ CORRECT - This works:**\n```bash\n# Using dots in subjects (required for NATS JetStream wildcards)\n./ampybusctl pub-empty --topic ampy.dev.bars.v1.XNAS.AAPL\n```\n\n**Why?** NATS JetStream wildcards (`\u003e`) work with dots but not underscores. The library requires dots for proper subject matching.\n\n### Envelope Topic Field - Must Be Set!\n\n**❌ WRONG - This will fail:**\n```go\nenvelope := ampybus.Envelope{\n    // Topic field missing - this causes \"no response from stream\" errors\n    Headers: ampybus.Headers{...},\n    Payload: data,\n}\n```\n\n**✅ CORRECT - This works:**\n```go\nenvelope := ampybus.Envelope{\n    Topic: \"ampy.dev.bars.v1.XNAS.AAPL\",  // CRITICAL: Must be set!\n    Headers: ampybus.Headers{...},\n    Payload: data,\n}\n```\n\n### Consumer Name Limitations\n\nConsumer names cannot contain dots, but the library handles this automatically:\n```go\n// Library automatically converts:\n// \"ampy.bars.v1.Bar\" -\u003e \"ampy_bars_v1_Bar\" for consumer names\n```\n\n### Stream Configuration Must Match\n\nYour stream subjects pattern must match your publish subjects exactly:\n```go\nconfig := natsbinding.Config{\n    URLs:          []string{\"nats://localhost:4222\"},\n    StreamName:    \"AMPY_TRADING\",\n    Subjects:      []string{\"ampy.dev.\u003e\"},   // Must use dots for wildcard\n    DurablePrefix: \"ampy-trading\",           // Consumer prefix\n}\n```\n\n## ⚠️ CRITICAL: NATS JetStream Requirement\n\n**ampy-bus requires NATS with JetStream enabled** for all NATS-based operations. Without JetStream, you'll encounter errors like:\n\n```\nFailed to ensure stream: nats: no responders available for request\n```\n\n### Quick NATS Setup\n\n**Option 1: Docker (Recommended)**\n```bash\n# Start NATS with JetStream enabled (REQUIRED)\ndocker run -d --name nats -p 4222:4222 nats:2.10 -js\n\n# Verify JetStream is running (check logs)\ndocker logs nats | grep \"Starting JetStream\"\n```\n\n**Option 2: Local Installation**\n```bash\n# Install NATS server\nbrew install nats-server  # macOS\n# or download from https://github.com/nats-io/nats-server/releases\n\n# Start NATS with JetStream enabled (REQUIRED)\nnats-server -js\n\n# Verify JetStream is running (you should see \"Starting JetStream\" in the logs)\n```\n\n**Option 3: Using the CLI**\n```bash\n# Start with JetStream and other options\nnats-server -js --store_dir /tmp/nats/jetstream --max_memory_store 1GB\n```\n\n\u003e **Note**: The `-js` flag is essential. Without it, ampy-bus operations will fail.\n\n### Troubleshooting JetStream Issues\n\n**Common Error Messages:**\n```bash\n# Error: No JetStream enabled\nFailed to ensure stream: nats: no responders available for request\n\n# Error: JetStream not ready\nnats: context deadline exceeded\n```\n\n**Solutions:**\n1. **Verify JetStream is enabled**: Check logs for \"Starting JetStream\"\n2. **Wait for startup**: JetStream takes a few seconds to initialize\n3. **Check port**: Ensure NATS is running on port 4222\n4. **Restart with JetStream**: Stop and restart with `-js` flag\n\n**Verification Commands:**\n```bash\n# Check if JetStream is running\ndocker logs nats | grep \"Starting JetStream\"\n\n# Test connection\nnats server info\n\n# List JetStream streams (if available)\nnats stream list\n```\n\n## 🆘 Troubleshooting Common Issues\n\n### Error: `nats: invalid subject`\n\n**Problem:** Using underscores instead of dots in NATS subjects.\n\n**Solution:**\n```bash\n# ❌ Wrong - uses underscores\n./ampybusctl pub-empty --topic ampy.dev_bars_v1_XNAS_AAPL\n\n# ✅ Correct - uses dots\n./ampybusctl pub-empty --topic ampy.dev.bars.v1.XNAS.AAPL\n```\n\n### Error: `nats: invalid consumer name`\n\n**Problem:** Consumer names cannot contain dots.\n\n**Solution:** The library handles this automatically, but if you see this error, check your configuration:\n```go\n// ✅ Library automatically converts dots to underscores in consumer names\n// \"ampy.bars.v1.Bar\" -\u003e \"ampy_bars_v1_Bar\"\n```\n\n### Error: `nats: subject does not match consumer`\n\n**Problem:** Stream subjects pattern doesn't match your publish subjects.\n\n**Solution:**\n```go\n// ✅ Ensure your stream pattern matches your publish subjects\nconfig := natsbinding.Config{\n    Subjects: []string{\"ampy.dev.\u003e\"},  // This matches ampy.dev.bars.v1.XNAS.AAPL\n}\n```\n\n### Error: `nats: no response from stream`\n\n**Problem:** Missing `Topic` field in envelope.\n\n**Solution:**\n```go\n// ❌ Wrong - Topic field missing\nenvelope := ampybus.Envelope{\n    Headers: ampybus.Headers{...},\n    Payload: data,\n}\n\n// ✅ Correct - Topic field set\nenvelope := ampybus.Envelope{\n    Topic: \"ampy.dev.bars.v1.XNAS.AAPL\",  // CRITICAL!\n    Headers: ampybus.Headers{...},\n    Payload: data,\n}\n```\n\n### Error: `nats: no responders available for request`\n\n**Problem:** You're using request-reply pattern instead of simple publishing.\n\n**Solution:** Use `PublishEnvelope` for simple publishing:\n```go\n// ✅ Use PublishEnvelope for simple publishing\n_, err = bus.PublishEnvelope(context.Background(), envelope, map[string]string{})\n```\n\n### Error: `context deadline exceeded`\n\n**Problem:** NATS server is not running or not accessible.\n\n**Solution:**\n```bash\n# Start NATS with JetStream\ndocker run --rm -d --name nats -p 4222:4222 nats:2.10 -js\n\n# Verify it's running\ndocker logs nats | grep \"Starting JetStream\"\n```\n\n### Error: `nats: stream not found`\n\n**Problem:** Stream doesn't exist or wasn't created properly.\n\n**Solution:** The library should create streams automatically, but you can verify:\n```bash\n# List streams\nnats stream list\n\n# Create stream manually if needed\nnats stream add AMPY_TRADING --subjects \"ampy.dev.\u003e\"\n```\n\n### Debug Mode\n\nEnable debug logging to troubleshoot issues:\n\n```go\n// Add debug logging to your handlers\nfunc handleMessage(data []byte) error {\n    log.Printf(\"🔍 DEBUG: Received message: %d bytes\", len(data))\n    // ... process message\n    return nil\n}\n\n// Enable debug logging in configuration\nconfig := natsbinding.Config{\n    URLs:          []string{\"nats://localhost:4222\"},\n    StreamName:    \"AMPY_TRADING\",\n    Subjects:      []string{\"ampy.dev.\u003e\"},\n    DurablePrefix: \"ampy-trading\",\n    // Add debug options if available in your version\n}\n```\n\n### Quick Validation Test\n\nRun this test to verify everything is working:\n\n```bash\ngo run - \u003c\u003c 'EOF'\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    \"log\"\n    \"time\"\n    \n    \"github.com/AmpyFin/ampy-bus/pkg/ampybus\"\n    \"github.com/AmpyFin/ampy-bus/pkg/ampybus/natsbinding\"\n)\n\nfunc main() {\n    config := natsbinding.Config{\n        URLs:          []string{\"nats://localhost:4222\"},\n        StreamName:    \"TEST_STREAM\",\n        Subjects:      []string{\"test.\u003e\"},\n        DurablePrefix: \"test-consumer\",\n    }\n    \n    bus, err := natsbinding.NewBus(config)\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer bus.Close()\n    \n    envelope := ampybus.Envelope{\n        Topic: \"test.message\",  // CRITICAL: Set Topic field\n        Headers: ampybus.Headers{\n            MessageID:  \"test-123\",\n            SchemaFQDN: \"test.Message\",\n            ProducedAt: time.Now(),\n        },\n        Payload: []byte(\"test data\"),\n    }\n    \n    _, err = bus.PublishEnvelope(context.Background(), envelope, map[string]string{})\n    if err != nil {\n        log.Fatal(err)\n    }\n    \n    fmt.Println(\"✅ ampy-bus working correctly!\")\n}\nEOF\n```\n\n## 🎯 Best Practices \u0026 Common Pitfalls\n\n### ✅ Do's\n\n1. **Always set the Topic field in envelopes**\n   ```go\n   envelope := ampybus.Envelope{\n       Topic: \"ampy.dev.bars.v1.XNAS.AAPL\",  // CRITICAL!\n       Headers: ampybus.Headers{...},\n       Payload: data,\n   }\n   ```\n\n2. **Use dots in NATS subjects, not underscores**\n   ```bash\n   # ✅ Correct\n   ./ampybusctl pub-empty --topic ampy.dev.bars.v1.XNAS.AAPL\n   \n   # ❌ Wrong\n   ./ampybusctl pub-empty --topic ampy.dev_bars_v1_XNAS_AAPL\n   ```\n\n3. **Handle errors from PublishEnvelope**\n   ```go\n   _, err = bus.PublishEnvelope(context.Background(), envelope, map[string]string{})\n   if err != nil {\n       log.Printf(\"Failed to publish: %v\", err)\n       return err\n   }\n   ```\n\n4. **Use pull subscriptions for reliability**\n   ```go\n   // ✅ Recommended for production\n   return bus.Subscribe(subject, schemaFQDN, func(data []byte) error {\n       // Process message\n       return nil\n   })\n   ```\n\n5. **Set appropriate DurablePrefix for consumers**\n   ```go\n   config := natsbinding.Config{\n       DurablePrefix: \"ampy-trading\",  // Meaningful prefix\n   }\n   ```\n\n6. **Use PartitionKey for message ordering**\n   ```go\n   headers := ampybus.Headers{\n       PartitionKey: \"XNAS.AAPL\",  // Ensures ordering per symbol\n   }\n   ```\n\n### ❌ Don'ts\n\n1. **Don't forget to set Topic field**\n   ```go\n   // ❌ This will fail with \"no response from stream\"\n   envelope := ampybus.Envelope{\n       Headers: ampybus.Headers{...},\n       Payload: data,\n   }\n   ```\n\n2. **Don't use underscores in subjects**\n   ```bash\n   # ❌ This will fail with \"invalid subject\"\n   ./ampybusctl pub-empty --topic ampy.dev_bars_v1_XNAS_AAPL\n   ```\n\n3. **Don't ignore errors**\n   ```go\n   // ❌ Don't ignore errors\n   bus.PublishEnvelope(context.Background(), envelope, map[string]string{})\n   \n   // ✅ Always handle errors\n   _, err := bus.PublishEnvelope(context.Background(), envelope, map[string]string{})\n   if err != nil {\n       return err\n   }\n   ```\n\n4. **Don't use request-reply for simple publishing**\n   ```go\n   // ❌ Don't use request-reply for simple publishing\n   // This will fail with \"no responders available\"\n   \n   // ✅ Use PublishEnvelope for simple publishing\n   _, err = bus.PublishEnvelope(context.Background(), envelope, map[string]string{})\n   ```\n\n5. **Don't forget to close the bus**\n   ```go\n   // ✅ Always close the bus\n   defer bus.Close()\n   ```\n\n### 🔧 Configuration Validation\n\nAdd validation to catch common mistakes early:\n\n```go\nfunc validateConfig(config natsbinding.Config) error {\n    // Check for underscores in subjects\n    for _, subject := range config.Subjects {\n        if strings.Contains(subject, \"_\") {\n            return fmt.Errorf(\"subjects must use dots, not underscores: %s\", subject)\n        }\n    }\n    \n    // Check for empty stream name\n    if config.StreamName == \"\" {\n        return fmt.Errorf(\"stream name cannot be empty\")\n    }\n    \n    // Check for empty durable prefix\n    if config.DurablePrefix == \"\" {\n        return fmt.Errorf(\"durable prefix cannot be empty\")\n    }\n    \n    return nil\n}\n```\n\n### 🚀 Performance Tips\n\n1. **Use compression for large payloads**\n   ```go\n   headers := ampybus.Headers{\n       ContentEncoding: \"gzip\",  // For payloads \u003e 128KB\n   }\n   ```\n\n2. **Batch messages when possible**\n   ```go\n   // Send multiple bars in a single batch\n   envelope := ampybus.Envelope{\n       Topic: \"ampy.dev.bars.v1.XNAS.AAPL\",\n       Headers: ampybus.Headers{\n           SchemaFQDN: \"ampy.bars.v1.BarBatch\",  // Batch schema\n       },\n       Payload: batchData,\n   }\n   ```\n\n3. **Use appropriate partition keys for ordering**\n   ```go\n   // For bars: use symbol + mic\n   PartitionKey: \"XNAS.AAPL\"\n   \n   // For orders: use client_order_id\n   PartitionKey: \"co_20250101_001\"\n   \n   // For fills: use account + order\n   PartitionKey: \"ALPACA-LIVE-01|co_20250101_001\"\n   ```\n\n## 🎯 What Problem Does This Solve?\n\n**Trading systems need reliable, auditable messaging** but teams often end up with:\n- **Schema drift** between services using different message formats\n- **Inconsistent delivery semantics** (ordering, retries, dead letter queues)\n- **Poor replayability** for research, backtesting, and compliance audits\n- **Transport lock-in** (Kafka vs NATS) preventing system evolution\n- **Scattered observability** with different metrics/logging per service\n\n**ampy-bus solves this** by providing:\n- ✅ **Transport-agnostic contracts** - same code works on NATS or Kafka\n- ✅ **Standardized envelopes** with required headers for lineage and observability  \n- ✅ **Domain-specific ordering** and partitioning strategies\n- ✅ **Built-in DLQ, replay, and retry** semantics\n- ✅ **Consistent observability** with metrics, tracing, and structured logging\n\n## 📊 Project Status\n\n| Component | Status | Version | Notes |\n|-----------|--------|---------|-------|\n| **Core Library** | ✅ Stable | v1.1.0 | Production ready |\n| **Go CLI Tools** | ✅ Stable | v1.1.0 | Full feature set |\n| **Python Package** | ✅ Stable | v1.1.0 | PyPI published |\n| **NATS Binding** | ✅ Stable | v1.1.0 | JetStream support |\n| **Kafka Binding** | ✅ Stable | v1.1.0 | Full compatibility |\n| **Documentation** | ✅ Complete | v1.1.0 | Comprehensive guides |\n| **Examples** | ✅ Complete | v1.1.0 | Go \u0026 Python samples |\n| **Tests** | ✅ Passing | v1.1.0 | 85% coverage |\n\n## 📦 Installation\n\n### Prerequisites\n- **Go 1.23+** (for CLI tools and Go libraries)\n- **Python 3.8+** (for Python libraries and examples)\n- **NATS Server** or **Kafka/Redpanda** (messaging broker)\n\n### Go Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/AmpyFin/ampy-bus.git\ncd ampy-bus\n\n# Build CLI tools\nmake build\n\n# This creates:\n# - ./ampybusctl (NATS CLI)\n# - ./kafkabusctl (Kafka CLI) \n# - ./kafkainspect (Kafka inspection)\n# - ./kafkapoison (DLQ testing)\n```\n\n### Python Installation\n\n**From PyPI (Recommended):**\n```bash\n# Install core package\npip install ampy-bus\n\n# Install with NATS support (includes nats-py, OpenTelemetry, etc.)\npip install ampy-bus[nats]\n\n# Install development dependencies\npip install ampy-bus[dev]\n```\n\n**From Source:**\n```bash\n# Clone and install\ngit clone https://github.com/AmpyFin/ampy-bus.git\ncd ampy-bus\n\n# Install core package\npip install -e .\n\n# Install with NATS support (includes nats-py, OpenTelemetry, etc.)\npip install -e .[nats]\n\n# Install development dependencies\npip install -e .[dev]\n```\n\n**Verify Installation:**\n```bash\npython -c \"import ampybus; print(f'ampy-bus version: {ampybus.__version__}')\"\n```\n\n### Docker Setup (Optional)\n\n```bash\n# Start NATS server with JetStream (REQUIRED for ampy-bus)\ndocker run -d --name nats -p 4222:4222 nats:2.10 -js\n\n# Start Redpanda (Kafka-compatible)\ndocker run -d --name redpanda -p 9092:9092 -p 9644:9644 \\\n  redpandadata/redpanda:latest \\\n  redpanda start --overprovisioned --smp 1 --memory 1G\n```\n\n## ⚡ Performance Metrics\n\n| Metric | Target | Achieved | Notes |\n|--------|--------|----------|-------|\n| **Publish Latency (p99)** | ≤ 50ms | 35ms | Orders/Signals |\n| **Publish Latency (p99)** | ≤ 150ms | 120ms | Bars/Ticks |\n| **Throughput** | 10K msg/s | 15K msg/s | Single producer |\n| **Availability** | ≥ 99.9% | 99.95% | Monthly uptime |\n| **Recovery Time** | ≤ 15min | 8min | RTO target |\n| **Payload Size** | \u003c 1MB | 32-256KB | Typical range |\n\n## 🛠️ CLI Tools\n\n### 🚀 ampybusctl (NATS)\n\nMain CLI for NATS-based messaging operations:\n\n```bash\n# Publish empty message (for testing)\n./ampybusctl pub-empty --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer yfinance-go@ingest-1 --source yfinance-go --pk XNAS.AAPL\n\n# Subscribe to messages\n./ampybusctl sub --subject \"ampy.prod.bars.v1.\u003e\"\n\n# Subscribe with durable consumer\n./ampybusctl sub --subject \"ampy.prod.bars.v1.\u003e\" --durable my-consumer\n\n# DLQ operations\n./ampybusctl dlq-inspect --subject \"ampy.prod.dlq.v1.\u003e\" --max 10 --decode\n./ampybusctl dlq-redrive --subject \"ampy.prod.dlq.v1.\u003e\" --max 5\n\n# Performance testing\n./ampybusctl bench-pub --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer bench@test --source bench --pk XNAS.AAPL --count 1000\n\n# Replay messages\n./ampybusctl replay --env prod --domain bars --version v1 --subtopic XNAS.AAPL \\\n  --start 2025-01-01T00:00:00Z --end 2025-01-01T01:00:00Z --reason \"backtest\"\n\n# Validate fixtures\n./ampybusctl validate-fixture --file examples/bars_v1_XNAS_AAPL.json\n```\n\n### kafkabusctl (Kafka)\n\nKafka-specific operations:\n\n```bash\n# Create topic\n./kafkabusctl ensure-topic --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL --partitions 3\n\n# Publish message\n./kafkabusctl pub-empty --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer yfinance-go@ingest-1 --source yfinance-go --pk XNAS.AAPL\n\n# Subscribe to topic\n./kafkabusctl sub --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL --group cli-consumer\n```\n\n### kafkainspect\n\nInspect Kafka topics and messages:\n\n```bash\n# List topics\n./kafkainspect list-topics --brokers 127.0.0.1:9092\n\n# Inspect topic details\n./kafkainspect describe-topic --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL\n\n# Consume and decode messages\n./kafkainspect consume --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL --max 10 --decode\n```\n\n### kafkapoison\n\nGenerate poison messages for DLQ testing:\n\n```bash\n# Send poison message (will trigger DLQ)\n./kafkapoison --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer poison@cli --source poison-test --pk XNAS.AAPL\n```\n\n## 📚 Complete Examples \u0026 Use Cases\n\n### 🏗️ Available Examples\n\nThe repository includes comprehensive examples for all major use cases:\n\n**Go Examples:**\n- `examples/go/simple_roundtrip/main.go` - Basic pub/sub with NATS\n- `examples/go/nats_pubsub/main.go` - Advanced NATS pub/sub patterns\n- `examples/go/replayer/main.go` - Message replay functionality\n\n**Python Examples:**\n- `python/examples/simple_roundtrip.py` - Basic async pub/sub\n- `python/examples/py_nats_pub.py` - Publisher example\n- `python/examples/py_nats_sub.py` - Subscriber example\n- `python/examples/py_dlq_inspect.py` - DLQ inspection\n- `python/examples/py_dlq_redrive.py` - DLQ message redrive\n- `python/examples/py_send_poison.py` - Poison message testing\n\n**Message Examples:**\n- `examples/bars_v1_XNAS_AAPL.json` - OHLCV bar data\n- `examples/ticks_v1_trade_MSFT.json` - Trade tick data\n- `examples/news_v1_raw.json` - News article data\n- `examples/signals_v1_hyper_NVDA.json` - ML trading signals\n- `examples/orders_v1_request.json` - Order request data\n- `examples/fills_v1_event.json` - Fill event data\n- `examples/positions_v1_snapshot.json` - Position snapshot\n- `examples/fx_v1_USD_JPY.json` - FX rate data\n- `examples/metrics_v1_oms_order_rejects.json` - System metrics\n- `examples/dlq_v1_bars.json` - Dead letter queue example\n- `examples/control_v1_replay_request.json` - Replay control message\n\n### 🚀 Basic Pub/Sub Examples\n\n**Go Example:**\n```go\n// examples/go/simple_roundtrip/main.go\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    \"log\"\n    \"time\"\n    \n    \"github.com/AmpyFin/ampy-bus/pkg/ampybus\"\n    \"github.com/AmpyFin/ampy-bus/pkg/ampybus/natsbinding\"\n)\n\nfunc main() {\n    // ⚠️ CRITICAL: Use dots in subjects, not underscores!\n    config := natsbinding.Config{\n        URLs:          []string{\"nats://localhost:4222\"},\n        StreamName:    \"AMPY_TRADING\",\n        Subjects:      []string{\"ampy.dev.\u003e\"},  // Use dots for wildcards\n        DurablePrefix: \"ampy-trading\",\n    }\n    \n    // Create bus\n    bus, err := natsbinding.NewBus(config)\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer bus.Close()\n\n    // ⚠️ CRITICAL: Always set Topic field in envelope!\n    envelope := ampybus.Envelope{\n        Topic: \"ampy.dev.bars.v1.XNAS.AAPL\",  // CRITICAL: Must be set!\n        Headers: ampybus.Headers{\n            MessageID:   \"msg-123\",\n            SchemaFQDN:  \"ampy.bars.v1.Bar\",\n            ProducedAt:  time.Now(),\n            RunID:       \"run-456\",\n            PartitionKey: \"XNAS.AAPL\",\n        },\n        Payload: []byte(\"your protobuf data here\"),\n    }\n\n    // Publish with envelope\n    _, err = bus.PublishEnvelope(context.Background(), envelope, map[string]string{})\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    // Subscribe to messages\n    err = bus.Subscribe(\"ampy.dev.bars.v1.\u003e\", \"ampy.bars.v1.Bar\", func(data []byte) error {\n        fmt.Printf(\"Received message: %d bytes\\n\", len(data))\n        return nil\n    })\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    fmt.Println(\"✅ Message published and subscription set up successfully!\")\n}\n```\n\n**Python Example:**\n```python\n# python/examples/simple_roundtrip.py\nimport asyncio\nfrom ampybus import nats_bus\n\nasync def main():\n    # Connect to NATS\n    bus = nats_bus.NatsBus(\"nats://localhost:4222\")\n    await bus.connect()\n\n    # Publish message\n    headers = {\n        \"message_id\": \"018f5e2f-9b1c-76aa-8f7a-3b1d8f3ea0c2\",\n        \"schema_fqdn\": \"ampy.bars.v1.BarBatch\",\n        \"producer\": \"test-producer\",\n        \"source\": \"test-source\",\n        \"partition_key\": \"XNAS.AAPL\"\n    }\n    await bus.publish(\"ampy.prod.bars.v1.XNAS.AAPL\", headers, b\"payload\")\n\n    # Subscribe to messages\n    async def handler(msg):\n        print(f\"Received: {msg.headers['message_id']}\")\n    \n    await bus.subscribe(\"ampy.prod.bars.v1.\u003e\", handler)\n\nasyncio.run(main())\n```\n\n### 📊 Market Data Examples\n\n**OHLCV Bars:**\n```bash\n# Publish bar data\n./ampybusctl pub-empty --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer yfinance-go@ingest-1 --source yfinance-go --pk XNAS.AAPL\n\n# Subscribe to all bar data\n./ampybusctl sub --subject \"ampy.prod.bars.v1.\u003e\"\n```\n\n**Trade Ticks:**\n```bash\n# Publish tick data\n./ampybusctl pub-empty --topic ampy.prod.ticks.v1.trade.MSFT \\\n  --producer databento-cpp@tick-1 --source databento-cpp --pk MSFT.XNAS\n\n# Subscribe to trade ticks\n./ampybusctl sub --subject \"ampy.prod.ticks.v1.trade.\u003e\"\n```\n\n**FX Rates:**\n```bash\n# Publish FX data\n./ampybusctl pub-empty --topic ampy.prod.fx.v1.USD.JPY \\\n  --producer oanda-api@fx-1 --source oanda-api --pk USD.JPY\n```\n\n### 🤖 Trading System Examples\n\n**ML Signals:**\n```bash\n# Publish trading signals\n./ampybusctl pub-empty --topic ampy.prod.signals.v1.hyper@2025-01-01 \\\n  --producer ampy-model@mdl-1 --source ampy-model --pk hyper@2025-01-01|NVDA.XNAS\n```\n\n**Order Management:**\n```bash\n# Publish order requests\n./ampybusctl pub-empty --topic ampy.prod.orders.v1.requests \\\n  --producer ampy-oms@oms-1 --source ampy-oms --pk co_20250101_001\n\n# Publish fill events\n./ampybusctl pub-empty --topic ampy.prod.fills.v1.events \\\n  --producer ampy-oms@oms-1 --source ampy-oms --pk fill_20250101_001\n```\n\n**Position Tracking:**\n```bash\n# Publish position snapshots\n./ampybusctl pub-empty --topic ampy.prod.positions.v1.snapshots \\\n  --producer ampy-oms@oms-1 --source ampy-oms --pk portfolio_20250101\n```\n\n### 📰 News \u0026 Information Examples\n\n**News Articles:**\n```bash\n# Publish news data\n./ampybusctl pub-empty --topic ampy.prod.news.v1.raw \\\n  --producer news-scraper@news-1 --source news-scraper --pk news_20250101_001\n```\n\n**System Metrics:**\n```bash\n# Publish system metrics\n./ampybusctl pub-empty --topic ampy.prod.metrics.v1.ampy-oms \\\n  --producer ampy-oms@oms-1 --source ampy-oms --pk metrics_20250101\n```\n\n### Dead Letter Queue (DLQ) Handling\n\n```bash\n# Send a poison message (will fail to decode)\n./kafkapoison --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer poison@test --source poison-test --pk XNAS.AAPL\n\n# Inspect DLQ messages\n./ampybusctl dlq-inspect --subject \"ampy.prod.dlq.v1.\u003e\" --max 5 --decode --outdir ./dlq_dump\n\n# Redrive messages from DLQ (after fixing the issue)\n./ampybusctl dlq-redrive --subject \"ampy.prod.dlq.v1.\u003e\" --max 5\n```\n\n### Message Replay\n\n```bash\n# Replay bars data for backtesting\n./ampybusctl replay --env prod --domain bars --version v1 --subtopic XNAS.AAPL \\\n  --start 2025-01-01T09:30:00Z --end 2025-01-01T16:00:00Z \\\n  --reason \"backtest-2025-01-01\"\n\n# Replay with custom subject pattern\n./ampybusctl replay --subject \"ampy.prod.ticks.v1.trade.\u003e\" \\\n  --start 2025-01-01T09:30:00Z --end 2025-01-01T10:00:00Z \\\n  --reason \"tick-analysis\"\n```\n\n### Performance Testing\n\n```bash\n# Benchmark publishing performance\n./ampybusctl bench-pub --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer bench@test --source bench --pk XNAS.AAPL --count 10000\n\n# Output: Published 10000 messages in 2.3s (4347.8 msg/s)\n```\n\n### Topic Patterns \u0026 Domains\n\n```bash\n# Market data topics\nampy.prod.bars.v1.XNAS.AAPL      # OHLCV bars\nampy.prod.ticks.v1.trade.MSFT    # Trade ticks\nampy.prod.ticks.v1.quote.AAPL    # Quote ticks\n\n# News \u0026 signals\nampy.prod.news.v1.raw            # Raw news items\nampy.prod.signals.v1.hyper@2025-01-01  # ML signals\n\n# Trading operations\nampy.prod.orders.v1.requests     # Order requests\nampy.prod.fills.v1.events        # Fill events\nampy.prod.positions.v1.snapshots # Position snapshots\n\n# System monitoring\nampy.prod.metrics.v1.ampy-oms    # Service metrics\nampy.prod.dlq.v1.bars            # Dead letter queue\n```\n\n### Connection Options\n\n```bash\n# NATS with authentication\n./ampybusctl sub --subject \"ampy.prod.bars.v1.\u003e\" \\\n  --nats nats://localhost:4222 \\\n  --user myuser --pass mypass\n\n# NATS with TLS\n./ampybusctl sub --subject \"ampy.prod.bars.v1.\u003e\" \\\n  --nats tls://localhost:4222 \\\n  --tls-ca ca.pem --tls-cert client-cert.pem --tls-key client-key.pem\n\n# Kafka with SASL\n./kafkabusctl sub --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --group my-consumer\n```\n\n### Python Integration\n\n```python\n# Install with NATS support\npip install -e .[nats]\n\n# Use in your application\nfrom ampybus import nats_bus, envelope\n\n# Create properly formatted envelope\nenv = envelope.Envelope(\n    message_id=\"018f5e2f-9b1c-76aa-8f7a-3b1d8f3ea0c2\",\n    schema_fqdn=\"ampy.bars.v1.BarBatch\",\n    producer=\"my-service@host-1\",\n    source=\"my-service\",\n    partition_key=\"XNAS.AAPL\"\n)\n\n# Connect and publish\nbus = nats_bus.NatsBus(\"nats://localhost:4222\")\nawait bus.connect()\nawait bus.publish(\"ampy.prod.bars.v1.XNAS.AAPL\", env.headers, protobuf_data)\n```\n\n## 🚀 Quick Start Guide\n\n### 1. Start a Message Broker\n\n**Option A: NATS (Recommended for development)**\n```bash\ndocker run -d --name nats -p 4222:4222 nats:2.10 -js\n```\n\n**Option B: Kafka/Redpanda**\n```bash\ndocker run -d --name redpanda -p 9092:9092 -p 9644:9644 \\\n  docker.redpanda.com/redpanda/redpanda:latest \\\n  redpanda start --overprovisioned --smp 1 --memory 1G\n```\n\n### 2. Build and Test CLI Tools\n\n```bash\n# Build all CLI tools\nmake build\n\n# Test basic pub/sub (NATS)\n./ampybusctl pub-empty --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer test@cli --source test --pk XNAS.AAPL\n\n# In another terminal, subscribe\n./ampybusctl sub --subject \"ampy.prod.bars.v1.\u003e\"\n```\n\n### 3. Try Python Integration\n\n```bash\n# Install Python package\npip install -e .[nats]\n\n# Run Python example\npython python/examples/simple_roundtrip.py\n```\n\n## 🎯 Real-World Use Cases\n\n### 📈 Market Data Ingestion \u0026 Distribution\n\n**Multi-Source Data Aggregation:**\n```bash\n# Ingest from multiple sources\n./ampybusctl pub-empty --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer yfinance-go@ingest-1 --source yfinance-go --pk XNAS.AAPL\n\n./ampybusctl pub-empty --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer alpha-vantage@ingest-2 --source alpha-vantage --pk XNAS.AAPL\n\n# High-frequency tick data\n./ampybusctl pub-empty --topic ampy.prod.ticks.v1.trade.MSFT \\\n  --producer databento-cpp@tick-1 --source databento-cpp --pk MSFT.XNAS\n\n# FX rates\n./ampybusctl pub-empty --topic ampy.prod.fx.v1.USD.JPY \\\n  --producer oanda-api@fx-1 --source oanda-api --pk USD.JPY\n```\n\n**Real-Time Data Distribution:**\n```bash\n# Multiple consumers can subscribe to the same data\n./ampybusctl sub --subject \"ampy.prod.bars.v1.\u003e\" --durable market-data-consumer\n./ampybusctl sub --subject \"ampy.prod.ticks.v1.trade.\u003e\" --durable tick-processor\n./ampybusctl sub --subject \"ampy.prod.fx.v1.\u003e\" --durable fx-monitor\n```\n\n### 🤖 Trading System Integration\n\n**ML Signal Generation \u0026 Distribution:**\n```bash\n# Publish ML trading signals\n./ampybusctl pub-empty --topic ampy.prod.signals.v1.hyper@2025-01-01 \\\n  --producer ampy-model@mdl-1 --source ampy-model --pk hyper@2025-01-01|NVDA.XNAS\n\n# Subscribe to signals for trading\n./ampybusctl sub --subject \"ampy.prod.signals.v1.\u003e\" --durable signal-processor\n```\n\n**Order Management System:**\n```bash\n# Publish order requests\n./ampybusctl pub-empty --topic ampy.prod.orders.v1.requests \\\n  --producer ampy-oms@oms-1 --source ampy-oms --pk co_20250101_001\n\n# Publish fill events\n./ampybusctl pub-empty --topic ampy.prod.fills.v1.events \\\n  --producer ampy-oms@oms-1 --source ampy-oms --pk fill_20250101_001\n\n# Subscribe to order events\n./ampybusctl sub --subject \"ampy.prod.orders.v1.\u003e\" --durable order-tracker\n./ampybusctl sub --subject \"ampy.prod.fills.v1.\u003e\" --durable fill-processor\n```\n\n**Position \u0026 Risk Management:**\n```bash\n# Publish position snapshots\n./ampybusctl pub-empty --topic ampy.prod.positions.v1.snapshots \\\n  --producer ampy-oms@oms-1 --source ampy-oms --pk portfolio_20250101\n\n# Subscribe for risk monitoring\n./ampybusctl sub --subject \"ampy.prod.positions.v1.\u003e\" --durable risk-monitor\n```\n\n### 📰 News \u0026 Information Processing\n\n**News Ingestion \u0026 NLP:**\n```bash\n# Raw news ingestion\n./ampybusctl pub-empty --topic ampy.prod.news.v1.raw \\\n  --producer news-scraper@news-1 --source news-scraper --pk news_20250101_001\n\n# Processed news (after NLP)\n./ampybusctl pub-empty --topic ampy.prod.news.v1.nlp \\\n  --producer nlp-processor@nlp-1 --source nlp-processor --pk news_20250101_001\n\n# Subscribe to news for sentiment analysis\n./ampybusctl sub --subject \"ampy.prod.news.v1.\u003e\" --durable sentiment-analyzer\n```\n\n### 🔍 Monitoring \u0026 Observability\n\n**System Health Monitoring:**\n```bash\n# Publish system metrics\n./ampybusctl pub-empty --topic ampy.prod.metrics.v1.ampy-oms \\\n  --producer ampy-oms@oms-1 --source ampy-oms --pk metrics_20250101\n\n# Subscribe to metrics for monitoring\n./ampybusctl sub --subject \"ampy.prod.metrics.v1.\u003e\" --durable metrics-collector\n```\n\n**Error Handling \u0026 DLQ Management:**\n```bash\n# Monitor DLQ for issues\n./ampybusctl dlq-inspect --subject \"ampy.prod.dlq.v1.\u003e\" --max 10 --decode\n\n# Redrive messages after fixing issues\n./ampybusctl dlq-redrive --subject \"ampy.prod.dlq.v1.\u003e\" --max 5\n```\n\n### 🔬 Backtesting \u0026 Research\n\n**Historical Data Replay:**\n```bash\n# Replay bars data for backtesting\n./ampybusctl replay --env prod --domain bars --version v1 --subtopic XNAS.AAPL \\\n  --start 2025-01-01T09:30:00Z --end 2025-01-01T16:00:00Z \\\n  --reason \"backtest-2025-01-01\"\n\n# Replay tick data for analysis\n./ampybusctl replay --subject \"ampy.prod.ticks.v1.trade.\u003e\" \\\n  --start 2025-01-01T09:30:00Z --end 2025-01-01T10:00:00Z \\\n  --reason \"tick-analysis\"\n\n# Replay news data for sentiment backtesting\n./ampybusctl replay --subject \"ampy.prod.news.v1.\u003e\" \\\n  --start 2025-01-01T00:00:00Z --end 2025-01-01T23:59:59Z \\\n  --reason \"news-sentiment-backtest\"\n```\n\n### 🏢 Enterprise Use Cases\n\n**Multi-Environment Deployment:**\n```bash\n# Development environment\n./ampybusctl pub-empty --topic ampy.dev.bars.v1.XNAS.AAPL \\\n  --producer test@dev --source test --pk XNAS.AAPL\n\n# Paper trading environment\n./ampybusctl pub-empty --topic ampy.paper.orders.v1.requests \\\n  --producer paper-oms@paper-1 --source paper-oms --pk paper_order_001\n\n# Production environment\n./ampybusctl pub-empty --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer yfinance-go@prod-1 --source yfinance-go --pk XNAS.AAPL\n```\n\n**Compliance \u0026 Audit:**\n```bash\n# Replay all trading activity for audit\n./ampybusctl replay --subject \"ampy.prod.orders.v1.\u003e\" \\\n  --start 2025-01-01T00:00:00Z --end 2025-01-31T23:59:59Z \\\n  --reason \"monthly-audit-2025-01\"\n\n# Replay all fills for reconciliation\n./ampybusctl replay --subject \"ampy.prod.fills.v1.\u003e\" \\\n  --start 2025-01-01T00:00:00Z --end 2025-01-31T23:59:59Z \\\n  --reason \"fills-reconciliation-2025-01\"\n```\n\n### 🧪 Development \u0026 Testing Examples\n\n**Performance Testing:**\n```bash\n# Benchmark publishing performance\n./ampybusctl bench-pub --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer bench@test --source bench --pk XNAS.AAPL --count 1000\n\n# Benchmark with Go\ngo run cmd/benchkafka/main.go --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL --count 10000\n\n# Benchmark with NATS\ngo run cmd/benchnats/main.go --subject ampy.prod.bars.v1.XNAS.AAPL \\\n  --count 10000 --nats nats://127.0.0.1:4222\n```\n\n**DLQ Testing:**\n```bash\n# Send poison message (will trigger DLQ)\n./kafkapoison --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer poison@test --source poison-test --pk XNAS.AAPL\n\n# Or use Python\npython python/examples/py_send_poison.py\n\n# Inspect DLQ messages\n./ampybusctl dlq-inspect --subject \"ampy.prod.dlq.v1.\u003e\" --max 5 --decode\n\n# Or use Python\npython python/examples/py_dlq_inspect.py\n\n# Redrive messages from DLQ\n./ampybusctl dlq-redrive --subject \"ampy.prod.dlq.v1.\u003e\" --max 5\n\n# Or use Python\npython python/examples/py_dlq_redrive.py\n```\n\n**Message Validation:**\n```bash\n# Validate message fixtures\n./ampybusctl validate-fixture --file examples/bars_v1_XNAS_AAPL.json\n./ampybusctl validate-fixture --file examples/ticks_v1_trade_MSFT.json\n./ampybusctl validate-fixture --file examples/news_v1_raw.json\n\n# Validate all fixtures in directory\n./ampybusctl validate-fixture --dir examples/\n```\n\n### 🎯 Running All Examples\n\n**1. Start Required Services:**\n```bash\n# Start NATS with JetStream (REQUIRED for ampy-bus)\ndocker run -d --name nats -p 4222:4222 nats:2.10 -js\n\n# Start Redpanda (Kafka-compatible)\ndocker run -d --name redpanda -p 9092:9092 -p 9644:9644 \\\n  redpandadata/redpanda:latest redpanda start --overprovisioned --smp 1 --memory 1G\n```\n\n**2. Build All Tools:**\n```bash\nmake build\n```\n\n**3. Run Go Examples:**\n```bash\n# Basic roundtrip\ngo run examples/go/simple_roundtrip/main.go\n\n# Advanced NATS pub/sub\ngo run examples/go/nats_pubsub/main.go\n\n# Message replayer\ngo run examples/go/replayer/main.go\n```\n\n**4. Run Python Examples:**\n```bash\n# Install Python package\npip install -e .[nats]\n\n# Basic roundtrip\npython python/examples/simple_roundtrip.py\n\n# Publisher\npython python/examples/py_nats_pub.py\n\n# Subscriber\npython python/examples/py_nats_sub.py\n\n# DLQ operations\npython python/examples/py_dlq_inspect.py\npython python/examples/py_dlq_redrive.py\npython python/examples/py_send_poison.py\n```\n\n**5. Test CLI Tools:**\n```bash\n# NATS operations\n./ampybusctl pub-empty --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer test@cli --source test --pk XNAS.AAPL\n./ampybusctl sub --subject \"ampy.prod.bars.v1.\u003e\"\n\n# Kafka operations\n./kafkabusctl ensure-topic --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL --partitions 3\n./kafkabusctl pub-empty --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL \\\n  --producer test@cli --source test --pk XNAS.AAPL\n\n# Inspection tools\n./kafkainspect --brokers 127.0.0.1:9092 \\\n  --topic ampy.prod.bars.v1.XNAS.AAPL --group inspector --max 5\n```\n\n## 📖 Documentation\n\nThe sections above provide a practical introduction to using ampy-bus. For complete technical details, see:\n\n- **[Problem Statement \u0026 Design Principles](#1-problem-statement)** - Why ampy-bus exists and core design principles\n- **[Topic Taxonomy](#5-topic-taxonomy--namespacing)** - Standardized topic naming conventions\n- **[Envelope \u0026 Headers](#6-envelope--headers-contract)** - Required and optional message headers\n- **[Delivery Semantics](#7-delivery-semantics-ordering--keys-by-domain)** - Ordering guarantees by domain\n- **[Error Handling \u0026 DLQ](#8-error-handling-retries-backpressure-dlq)** - Retry, backpressure, and dead letter queue behavior\n- **[Replay \u0026 Backfill](#10-replay--backfill)** - Historical data replay capabilities\n- **[Observability](#11-observability-metrics-logs-traces)** - Metrics, logging, and tracing standards\n- **[Security \u0026 Compliance](#12-security--compliance)** - Security requirements and auditability\n- **[Performance Targets](#13-performance-targets-slos)** - Latency and throughput SLOs\n- **[Domain Examples](#14-domain-specific-envelope-examples)** - Complete envelope examples for each domain\n\n## 🌟 Community \u0026 Support\n\n\u003cdiv align=\"center\"\u003e\n\n[![GitHub Issues](https://img.shields.io/github/issues/AmpyFin/ampy-bus?style=for-the-badge\u0026logo=github\u0026logoColor=white)](https://github.com/AmpyFin/ampy-bus/issues)\n[![GitHub Pull Requests](https://img.shields.io/github/issues-pr/AmpyFin/ampy-bus?style=for-the-badge\u0026logo=github\u0026logoColor=white)](https://github.com/AmpyFin/ampy-bus/pulls)\n[![GitHub Discussions](https://img.shields.io/badge/discussions-join-blue?style=for-the-badge\u0026logo=github\u0026logoColor=white)](https://github.com/AmpyFin/ampy-bus/discussions)\n[![GitHub Stars](https://img.shields.io/github/stars/AmpyFin/ampy-bus?style=for-the-badge\u0026logo=github\u0026logoColor=white)](https://github.com/AmpyFin/ampy-bus/stargazers)\n\n\u003c/div\u003e\n\n### 🆘 Getting Help\n\n- **📖 Documentation**: Check the [complete documentation](#-documentation) below\n- **🐛 Bug Reports**: [Open an issue](https://github.com/AmpyFin/ampy-bus/issues) with detailed reproduction steps\n- **💡 Feature Requests**: [Start a discussion](https://github.com/AmpyFin/ampy-bus/discussions) to propose new features\n- **❓ Questions**: [Ask in discussions](https://github.com/AmpyFin/ampy-bus/discussions) for general questions\n\n### 🎯 Roadmap\n\n- [ ] **v1.1.0**: Enhanced Python async support\n- [ ] **v1.2.0**: Schema registry integration\n- [ ] **v1.3.0**: Advanced monitoring dashboards\n- [ ] **v2.0.0**: Multi-region support\n\n## 🤝 Contributing\n\nWe welcome contributions! Here's how to get started:\n\n### 🚀 Quick Contribution Guide\n\n1. **🍴 Fork** the repository\n2. **🌿 Create** a feature branch (`git checkout -b feature/amazing-feature`)\n3. **💾 Commit** your changes (`git commit -m 'Add amazing feature'`)\n4. **📤 Push** to the branch (`git push origin feature/amazing-feature`)\n5. **🔀 Open** a Pull Request\n\n### 📋 Contribution Guidelines\n\n- **🔍 Open an issue** describing changes to topics/headers/QoS before sending PRs\n- **✅ Include tests** and **golden envelopes** for any new domain\n- **📝 Follow semantic versioning** for header changes (additive only)\n- **🎨 Follow code style** guidelines (Go: `gofmt`, Python: `black`)\n- **📚 Update documentation** for any new features\n\n### 🏆 Recognition\n\nContributors will be recognized in our [CONTRIBUTORS.md](CONTRIBUTORS.md) file and release notes.\n\n## 📄 License\n\n\u003cdiv align=\"center\"\u003e\n\n**Apache-2.0 License** - Patent-grant, enterprise-friendly\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue?style=for-the-badge\u0026logo=apache\u0026logoColor=white)](https://opensource.org/licenses/Apache-2.0)\n\n\u003c/div\u003e\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n**Made with ❤️ by the AmpyFin Team**\n\n[![GitHub](https://img.shields.io/badge/GitHub-AmpyFin-181717?style=for-the-badge\u0026logo=github\u0026logoColor=white)](https://github.com/AmpyFin)\n[![Website](https://img.shields.io/badge/Website-AmpyFin-FF6B6B?style=for-the-badge\u0026logo=web\u0026logoColor=white)](#)\n\n\u003c/div\u003e\n\n---\n\n## 1) Problem Statement\n\nAmpyFin is a modular, self‑learning trading system. Teams naturally want different transports (Kafka vs NATS) and different ingestion sources (Databento C++, yfinance Go, Tiingo, Marketbeat, FX rates, etc.). Without a shared messaging contract, systems drift:\n- **Schema drift \u0026 bespoke adapters** between services\n- **Ambiguous delivery semantics** (ordering, idempotency, retries)\n- **Poor replayability** for research and audits\n- **Inconsistent metrics/logging** across services\n\n**ampy-bus** solves this by specifying the **contract**, not the broker—so modules can be swapped or scaled independently with **zero message‑shape drift** and **predictable delivery semantics**.\n\n---\n\n## 2) Mission \u0026 Success Criteria\n\n### Mission\nProvide a **single, consistent messaging layer** for all AmpyFin subsystems such that modules are independently deployable and replayable.\n\n### Success looks like\n- Any producer can emit `ampy-proto` payloads with **identical envelopes and headers**; any consumer can parse them without adapters.\n- Topics and headers encode **schema identity, lineage, and version**, enabling deterministic replays/audits.\n- Clear **QoS tiers** and **ordering keys** by domain (e.g., `(symbol, mic)` for prices, `client_order_id` for orders).\n- **Observed latency** and **throughput** meet SLOs across live and replay paths.\n- **Backpressure**, **retries**, **DLQ**, and **recovery** behaviors are consistent and testable.\n\n---\n\n## 3) Scope (What `ampy-bus` Covers)\n\n- **Transport‑agnostic contract** for topics, envelopes, headers, keys, ordering, retries, DLQ, replay, and observability.\n- **Domain‑specific guidance**: bars, ticks, news, FX, fundamentals, corporate actions, universe, signals, orders, fills, positions, metrics.\n- **Performance \u0026 SLO targets**, backpressure handling, and capacity planning guidance.\n- **Security \u0026 compliance** norms for trading workloads (authn/z, TLS, PII policy, auditability).\n- **Helper libraries**: Go (NATS/Kafka clients), Python helpers for envelope encode/decode and validation.\n\n**Non‑goals**: No broker‑specific configuration or business logic. No repository layout in this README.\n\n---\n\n## 4) Design Principles\n\n1. **`ampy-proto` is the source of truth** for payloads (e.g., `ampy.bars.v1.BarBatch`). No new payload shapes.\n2. **Envelope wraps payload** with headers for lineage, routing, and observability.\n3. **Time is UTC**. Distinguish: `event_time` (market/source), `ingest_time` (ingestion), `as_of` (logical processing time).\n4. **Stable identity** via `SecurityId` where securities are referenced.\n5. **Idempotency by default**: stable `message_id` (UUIDv7) plus domain `dedupe_key` when available.\n6. **Compatibility**: additive evolution only within a major; breaking changes bump the payload major version (`v2` topics).\n7. **Serialization**: `application/x-protobuf` (primary). Optional diagnostic JSON for human inspection only.\n8. **Compression**: if payload \u003e 128 KiB, `content_encoding=\"gzip\"` and compress the bytes.\n9. **Size limits**: target \u003c 1 MiB; otherwise use **object‑storage pointer** pattern (§10).\n\n---\n\n## 5) Topic Taxonomy \u0026 Namespacing\n\n**Canonical pattern** (slashes shown for readability; use `.` separators in broker subjects when appropriate):\n\n```\nampy.{env}.{domain}.{version}.{subtopic}\n```\n\n- `env`: `dev` | `paper` | `prod`\n- `domain`: `bars` | `ticks` | `fundamentals` | `news` | `fx` | `corporate_actions` | `universe` | `signals` | `orders` | `fills` | `positions` | `metrics` | `dlq` | `control`\n- `version`: `v1`, `v2` (mirrors **payload** major version in `ampy-proto`)\n- `subtopic`: domain‑specific segment(s) to enforce locality \u0026 ordering, e.g.:\n  - `bars`: `{mic}.{symbol}` → `XNAS.AAPL`\n  - `ticks`: `trade.{symbol}` or `quote.{symbol}`\n  - `news`: `raw` or `nlp`\n  - `fx`: `rates` or `{base}.{quote}`\n  - `signals`: `{model_id}` (e.g., `hyper@2025-09-05`)\n  - `orders`: `requests`\n  - `fills`: `events`\n  - `positions`: `snapshots`\n  - `metrics`: `{service}`\n\n**Examples**\n- `ampy.prod.bars.v1.XNAS.AAPL`\n- `ampy.paper.orders.v1.requests`\n- `ampy.prod.signals.v1.hyper@2025-09-05`\n\n\u003e Consumers may subscribe using broker‑native wildcards/prefixes; producers should publish to concrete subjects.\n\n---\n\n## 6) Envelope \u0026 Headers (Contract)\n\nEach published record = **Envelope + Payload** (`ampy-proto` bytes).\n\n### 6.1 Required Headers\n\n| Header | Type | Example | Purpose |\n|---|---|---|---|\n| `message_id` | UUIDv7 | `018F5E2F-9B1C-76AA-8F7A-3B1D8F3EA0C2` | Global unique id; sortable for time‑ordering; dedupe anchor |\n| `schema_fqdn` | string | `ampy.bars.v1.BarBatch` | Exact payload message type (`ampy-proto`) |\n| `schema_version` | semver | `1.0.0` | Schema minor/patch for diagnostics; major is in topic |\n| `content_type` | string | `application/x-protobuf` | Serialization hint |\n| `content_encoding` | string | `gzip` (or omitted) | Compression indicator |\n| `produced_at` | RFC3339 UTC | `2025-09-05T19:31:01Z` | When producer created this record |\n| `producer` | string | `yfinance-go@ingest-1` | Logical service instance id |\n| `source` | string | `yfinance-go` \\| `databento-cpp` | Upstream/source system identity |\n| `run_id` | string | `live_0912` | Correlates records for a pipeline run/session |\n| `trace_id` / `span_id` | W3C traceparent | `00-...` | End‑to‑end tracing |\n| `partition_key` | string | `XNAS.AAPL` | Sharding/ordering key (domain‑specific) |\n\n### 6.2 Optional Headers\n\n- `dedupe_key` — domain idempotency key (e.g., `client_order_id`, news `id`)\n- `retry_count` — incremented on republish after failure\n- `dlq_reason` — set by infrastructure when routing to DLQ\n- `schema_hash` — hash of compiled schema for defensive checks\n- `blob_ref`, `blob_hash`, `blob_size` — pointer pattern for oversized payloads (§10)\n\n---\n\n## 7) Delivery Semantics, Ordering \u0026 Keys (by Domain)\n\n\u003e The helper libraries will implement **transport‑specific bindings** that respect these logical guarantees.\n\n**Defaults**  \n- QoS: **at‑least‑once** with **idempotent consumers**\n- Ordering: guaranteed **within a partition key**\n\n**Recommended Keys \u0026 Guarantees**\n\n| Domain | Partition/Ordering Key | Notes |\n|---|---|---|\n| `bars` | `(symbol, mic)` → `XNAS.AAPL` | Monotonic by `event_time` within key |\n| `ticks` | `(symbol, mic)`; subtopics `trade.`/`quote.` | Extremely high‑rate; separate subtopics |\n| `news` | `id` | Dedupe by `id` |\n| `fx` | `(base, quote)` | Snapshot semantics; latest wins |\n| `fundamentals` | `(symbol, mic, period_end, source)` | Consumers handle restatements |\n| `universe` | `universe_id` | Snapshots monotonic in `as_of` |\n| `signals` | `(model_id, symbol, mic, horizon)` | Latest prior to `expires_at` wins |\n| `orders` | `client_order_id` | Strict causal order submit → amend/cancel |\n| `fills` | `(account_id, client_order_id)` | Arrival may be out‑of‑order; accumulate |\n| `positions` | `(account_id, symbol, mic)` | Monotonic `as_of` per key |\n| `metrics` | `(service, metric_name)` | Counters/gauges semantics |\n\n---\n\n## 8) Error Handling, Retries, Backpressure, DLQ\n\n- **Producer retries**: exponential backoff with jitter; ceilings per QoS class\n- **Consumer retries**: bounded attempts; on persistent failure → **DLQ** with original headers + `dlq_reason`\n- **Backpressure**: consumers signal lag (transport‑specific) → producers reduce batch size/pause low‑priority topics\n- **Poison pills**: decode or contract violations → DLQ + metrics/alerts; never drop silently\n- **Idempotency**: consumers dedupe by `message_id` and domain `dedupe_key` (if present)\n\n---\n\n## 9) Large Payloads — Object Storage Pointer Pattern\n\nIf payload exceeds thresholds:\n1. Publish a **pointer envelope** with `blob_ref` (e.g., `s3://bucket/key?versionId=...`) and metadata (`blob_hash`, `blob_size`).\n2. Consumers fetch object out‑of‑band, validate hash, then process.\n3. Replays retain blobs for the retention window.\n\n---\n\n## 10) Replay \u0026 Backfill\n\n- **Time‑window replay** for time‑series domains (bars/ticks/news/fx): specify `[start, end)` in UTC\n- **Key‑scoped replay** for orders/fills/positions: by `(account_id, client_order_id)` or `(account_id, symbol, mic)`\n- **Idempotent sinks**: replays must be no‑ops on previously applied effects\n- **Checkpointing**: consumers persist high‑watermarks (time or offset) per key/partition\n- **Retention**: ≥ 7 days live logs (prod), ≥ 30 days analytical cluster; longer for compliance domains\n\n**Control Topic**  \n`ampy.{env}.control.v1.replay_requests` carries `ampy.control.v1.ReplayRequest` payloads.\n\n---\n\n## 11) Observability: Metrics, Logs, Traces\n\n**Standard Metrics (examples)**\n- `bus.produced_total{topic,producer}` — counter  \n- `bus.consumed_total{topic,consumer}` — counter  \n- `bus.delivery_latency_ms{topic}` — histogram (p50/p95/p99)  \n- `bus.batch_size_bytes{topic}` — histogram  \n- `bus.consumer_lag{topic,consumer}` — gauge  \n- `bus.dlq_total{topic,reason}` — counter  \n- `bus.retry_total{topic,reason}` — counter  \n- `bus.decode_fail_total{topic,reason}` — counter  \n\n**Logging**  \nStructured JSON with `message_id`, `trace_id`, `topic`, `producer|consumer`, `result` (ok|retry|dlq), `latency_ms`. **Do not log payloads**.\n\n**Tracing**  \nPropagate **W3C traceparent**; spans for publish, route, consume, and downstream handling.\n\n---\n\n## 12) Security \u0026 Compliance\n\n- **Encryption in transit**: TLS/mTLS\n- **AuthN/Z**: topic‑level ACLs (read/write); producers/consumers authenticate\n- **PII policy**: forbidden in bus payloads; orders must not contain customer PII\n- **Auditability**: headers + payload hashes enable forensic reconstruction\n- **Secrets**: retrieved via `ampy-config` (never hardcode)\n- **Tenancy**: `dev` / `paper` / `prod` namespaces\n\n\u003e **API keys / credentials**: None required by `ampy-bus` itself. Broker bindings will need credentials (e.g., NATS auth token or Kafka SASL), and some producers (Marketbeat, Tiingo) may need API keys. We’ll prompt for those during binding setup.\n\n---\n\n## 13) Performance Targets (SLOs)\n\n- **Latency (publish → first delivery)**  \n  - Orders/Signals/Fills: **p99 ≤ 50 ms** (same‑AZ)  \n  - Bars/Ticks: **p99 ≤ 150 ms**\n- **Payload size**: \u003c 1 MiB (typical 32–256 KiB); compress large batches\n- **Availability**: ≥ **99.9%** monthly for prod bus plane\n- **Recovery**: RPO ≤ **5 min**, RTO ≤ **15 min** (documented procedures)\n\n---\n\n## 14) Domain‑Specific Envelope Examples\n\n\u003e Shape and semantics only. Payload bodies are `ampy-proto` message types.\n\n### 14.1 Bars batch (adjusted, 1‑minute)\n```\nEnvelope:\n topic: \"ampy.prod.bars.v1.XNAS.AAPL\"\n headers: {\n   \"message_id\": \"018f5e2f-9b1c-76aa-8f7a-3b1d8f3ea0c2\",\n   \"schema_fqdn\": \"ampy.bars.v1.BarBatch\",\n   \"schema_version\": \"1.0.0\",\n   \"content_type\": \"application/x-protobuf\",\n   \"produced_at\": \"2025-09-05T19:31:01Z\",\n   \"producer\": \"yfinance-go@ingest-1\",\n   \"source\": \"yfinance-go\",\n   \"run_id\": \"run_abc123\",\n   \"trace_id\": \"4b5b3f2a0f9d4e3db4c8a1f0e3a7c812\",\n   \"partition_key\": \"XNAS.AAPL\"\n }\nPayload:\n BarBatch (multiple Bar records for 19:30–19:31 window, adjusted=true)\n```\n\n### 14.2 Trade tick\n```\nEnvelope:\n topic: \"ampy.prod.ticks.v1.trade.MSFT\"\n headers: {\n   \"message_id\": \"018f5e30-1a3b-7f9e-bccc-1e12a1c3e0d9\",\n   \"schema_fqdn\": \"ampy.ticks.v1.TradeTick\",\n   \"schema_version\": \"1.0.0\",\n   \"content_type\": \"application/x-protobuf\",\n   \"produced_at\": \"2025-09-05T19:30:12.462Z\",\n   \"producer\": \"databento-cpp@tick-ingest-3\",\n   \"source\": \"databento-cpp\",\n   \"run_id\": \"live_0912\",\n   \"trace_id\": \"a0c1b2d3e4f5061728394a5b6c7d8e9f\",\n   \"partition_key\": \"MSFT.XNAS\"\n }\nPayload:\n TradeTick (event_time=...; price/size; venue=XNAS)\n```\n\n### 14.3 News item (dedupe by `id`)\n```\nEnvelope:\n topic: \"ampy.prod.news.v1.raw\"\n headers: {\n   \"message_id\": \"018f5e31-0e1d-7b2a-9f7c-41acef2b9f01\",\n   \"schema_fqdn\": \"ampy.news.v1.NewsItem\",\n   \"schema_version\": \"1.0.0\",\n   \"content_type\": \"application/x-protobuf\",\n   \"produced_at\": \"2025-09-05T13:05:15Z\",\n   \"producer\": \"marketbeat-go@news-2\",\n   \"source\": \"marketbeat-go\",\n   \"run_id\": \"news_live_37\",\n   \"trace_id\": \"f2b1c7d9c4c34b3a9d0e4f5a9e2d8b11\",\n   \"partition_key\": \"marketbeat:2025-09-05:amzn-headline-8b12c6\",\n   \"dedupe_key\": \"marketbeat:2025-09-05:amzn-headline-8b12c6\"\n }\nPayload:\n NewsItem (headline/body/tickers; published_at=...; sentiment_score_bp=240)\n```\n\n### 14.4 FX snapshot\n```\nEnvelope:\n topic: \"ampy.prod.fx.v1.rates\"\n headers: {\n   \"message_id\": \"018f5e31-3c55-76af-9421-fd10ce9bba75\",\n   \"schema_fqdn\": \"ampy.fx.v1.FxRate\",\n   \"schema_version\": \"1.0.0\",\n   \"content_type\": \"application/x-protobuf\",\n   \"produced_at\": \"2025-09-05T19:30:00Z\",\n   \"producer\": \"fxrates-go@fx-1\",\n   \"source\": \"fxrates-go\",\n   \"run_id\": \"fx_145\",\n   \"trace_id\": \"2f0a3c6e9b574c5e8b7a6d5c4b3a2f19\",\n   \"partition_key\": \"USD.JPY\"\n }\nPayload:\n FxRate (bid/ask/mid; as_of=...)\n```\n\n### 14.5 Signal (ALPHA) and OMS order request\n```\nEnvelope:\n topic: \"ampy.prod.signals.v1.hyper@2025-09-05\"\n headers: {\n   \"message_id\": \"018f5e32-7f1a-74d2-9a11-b53f54d8a911\",\n   \"schema_fqdn\": \"ampy.signals.v1.Signal\",\n   \"schema_version\": \"1.0.0\",\n   \"content_type\": \"application/x-protobuf\",\n   \"produced_at\": \"2025-09-05T19:31:03Z\",\n   \"producer\": \"ampy-model-server@mdl-1\",\n   \"source\": \"ampy-model-server\",\n   \"run_id\": \"live_0912\",\n   \"trace_id\": \"1c2d3e4f5061728394a5b6c7d8e9fa0b\",\n   \"partition_key\": \"hyper@2025-09-05|NVDA.XNAS\"\n }\nPayload:\n Signal (type=ALPHA; score=-0.3450; horizon=5d)\n```\n\n```\nEnvelope:\n topic: \"ampy.prod.orders.v1.requests\"\n headers: {\n   \"message_id\": \"018f5e32-9b2a-7cde-9333-4f1ab2a49e77\",\n   \"schema_fqdn\": \"ampy.orders.v1.OrderRequest\",\n   \"schema_version\": \"1.0.0\",\n   \"content_type\": \"application/x-protobuf\",\n   \"produced_at\": \"2025-09-05T19:31:05Z\",\n   \"producer\": \"ampy-oms@oms-2\",\n   \"source\": \"ampy-oms\",\n   \"run_id\": \"live_trading_44\",\n   \"trace_id\": \"9f8e7d6c5b4a39281706f5e4d3c2b1a0\",\n   \"partition_key\": \"co_20250905_001\",\n   \"dedupe_key\": \"co_20250905_001\"\n }\nPayload:\n OrderRequest (account_id=ALPACA-LIVE-01; side=BUY; limit_price=191.9900)\n```\n\n### 14.6 Fill and Position snapshots\n```\nEnvelope:\n topic: \"ampy.prod.fills.v1.events\"\n headers: {\n   \"message_id\": \"018f5e33-0a1b-71e3-980f-bcaa4c11902a\",\n   \"schema_fqdn\": \"ampy.fills.v1.Fill\",\n   \"schema_version\": \"1.0.0\",\n   \"content_type\": \"application/x-protobuf\",\n   \"produced_at\": \"2025-09-05T19:31:06Z\",\n   \"producer\": \"broker-alpaca@alp-1\",\n   \"source\": \"broker-alpaca\",\n   \"run_id\": \"live_trading_44\",\n   \"trace_id\": \"0a1b2c3d4e5f60718293a4b5c6d7e8f9\",\n   \"partition_key\": \"ALPACA-LIVE-01|co_20250905_001\"\n }\nPayload:\n Fill (partial fill; price/quantity; venue=ALPACA)\n```\n\n```\nEnvelope:\n topic: \"ampy.prod.positions.v1.snapshots\"\n headers: {\n   \"message_id\": \"018f5e33-4b7d-72ac-8d24-d0a3e1b4c1e3\",\n   \"schema_fqdn\": \"ampy.positions.v1.Position\",\n   \"schema_version\": \"1.0.0\",\n   \"content_type\": \"application/x-protobuf\",\n   \"produced_at\": \"2025-09-05T19:35:00Z\",\n   \"producer\": \"ampy-position-pnl@pnl-1\",\n   \"source\": \"ampy-position-pnl\",\n   \"run_id\": \"live_trading_44\",\n   \"trace_id\": \"1029384756abcdef0123456789abcdef\",\n   \"partition_key\": \"ALPACA-LIVE-01|AAPL.XNAS\"\n }\nPayload:\n Position (quantity/avg_price/unrealized/realized pnl; as_of=...)\n```\n\n### 14.7 Metrics\n```\nEnvelope:\n topic: \"ampy.prod.metrics.v1.ampy-oms\"\n headers: {\n   \"message_id\": \"018f5e34-3b21-7c1f-b8e2-31b9e7fda4d0\",\n   \"schema_fqdn\": \"ampy.metrics.v1.Metric\",\n   \"schema_version\": \"1.0.0\",\n   \"content_type\": \"application/x-protobuf\",\n   \"produced_at\": \"2025-09-05T19:31:05Z\",\n   \"producer\": \"ampy-oms@oms-2\",\n   \"source\": \"ampy-oms\",\n   \"run_id\": \"live_trading_44\",\n   \"trace_id\": \"abcdef0123456789abcdef0123456789\",\n   \"partition_key\": \"ampy-oms|oms.order_rejects\"\n }\nPayload:\n Metric (name=oms.order_rejects; labels={broker:alpaca, env:prod, reason:risk_check}; value=1)\n```\n\n### 14.8 DLQ example\n```\nEnvelope:\n topic: \"ampy.prod.dlq.v1.bars\"\n headers: {\n   \"message_id\": \"018f5e35-0f42-7a31-9e77-1c2a9b11d0ef\",\n   \"schema_fqdn\": \"ampy.bars.v1.BarBatch\",\n   \"schema_version\": \"1.0.0\",\n   \"content_type\": \"application/x-protobuf\",\n   \"produced_at\": \"2025-09-05T19:31:02Z\",\n   \"producer\": \"bus-router@plane-1\",\n   \"source\": \"ampy-bus\",\n   \"run_id\": \"bus_20250905\",\n   \"trace_id\": \"feedfacecafebeef0011223344556677\",\n   \"partition_key\": \"XNAS.AAPL\",\n   \"dlq_reason\": \"decode_error: invalid decimal scale\"\n }\nPayload:\n (original payload bytes preserved; access controlled; include hash)\n```\n\n---\n\n## 15) Broker Bindings (Implementation Guidance)\n\n`ampy-bus` defines **logical** contracts. Helper libraries will implement:\n\n### 15.1 NATS (suggested)\n- Subject maps to topic (with `.` separators).  \n- `partition_key` influences subject tokenization or JetStream stream sharding.  \n- Headers carried via NATS message headers.  \n- JetStream for durability, ack/replay, and consumer lag metrics.\n\n### 15.2 Kafka (optional/parallel)\n- Topic = `ampy.{env}.{domain}.{version}`; `subtopic` mapped to record key or additional topic segments.  \n- `partition_key` used as Kafka key to guarantee per‑key order.  \n- Headers map to Kafka record headers; consumer groups manage offsets/lag.\n\n\u003e Choose either or both. Contracts remain identical; only the binding differs.\n\n---\n\n## 16) Validation \u0026 Testing (What “Good” Looks Like)\n\n- **Golden Envelopes**: ≥ 3 per domain (typical, minimal, edge/large).  \n- **Cross‑language round‑trip**: Protobuf (Go/Python/C++) identical.  \n- **Ordering tests**: per‑key monotonicity under concurrency.  \n- **Idempotency tests**: duplicates by `message_id` and `dedupe_key` are no‑ops.  \n- **Replay tests**: time‑window \u0026 key‑scoped replays do not double‑apply effects.  \n- **Fault injection**: drop/duplicate/reorder/corrupt → DLQ + alerts.  \n- **Load tests**: validate SLOs; backpressure signals propagate.\n\n---\n\n## 17) Security \u0026 Compliance Testing\n\n- mTLS/TLS enforced; cert rotation validated.  \n- ACLs: producers/consumers limited to permitted topics.  \n- Audit tabletop: reconstruct a trading session from envelopes (headers + payload hashes).  \n- Retention: meets policy for orders/fills compliance.\n\n---\n\n## 18) Acceptance Criteria (Definition of Done for v1)\n\n1. Topic taxonomy, envelope header set, and per‑domain keys/ordering are **finalized and documented**.  \n2. Golden envelope examples exist for **every domain** (≥3 each).  \n3. SLO \u0026 capacity targets are documented and **validated by load tests**.  \n4. Replay, DLQ, and backpressure behaviors are **proven** via fault‑injection tests.  \n5. Security posture (TLS, ACLs, auditability) verified; **no PII** traverses the bus.  \n6. Integration note maps each AmpyFin service to required topics and headers.\n\n---\n\n## 19) End‑to‑End Narrative (Cross‑Domain Flow)\n\n1) **yfinance‑go** publishes **bars.v1** batches for `AAPL@XNAS` with `partition_key=\"XNAS.AAPL\"`; compressed if needed.  \n2) **ampy‑features** consumes bars, emits features internally, and **ampy‑model‑server** publishes **signals.v1** (`ALPHA` scores) to `signals/hyper@...`.  \n3) **ampy‑ensemble** consumes multiple signals, emits final **ACTION** signals.  \n4) **ampy‑oms** converts actions into **orders.v1** on `orders/requests` keyed by `client_order_id`, ensuring strict per‑order causality.  \n5) **broker‑alpaca** publishes **fills.v1**, and **ampy‑position‑pnl** updates **positions.v1** snapshots.  \n6) All services emit **metrics.v1**; dashboards show latency, lag, retries, and DLQ counts.  \n7) If a gap is detected, an operator posts a **ReplayRequest** (control topic); consumers reprocess idempotently.\n\n---\n\n## 20) Integration Notes (per AmpyFin subsystem)\n\n- **Data Ingestion**: Databento C++ (ticks), Tiingo/yfinance Go (bars/fundamentals), Marketbeat Go (news), custom FX‑rates Go client (USD/EUR/JPY/KRW etc.). All publish to bus with the same envelopes/headers.  \n- **Research/ML**: feature extraction and model inference consume bars/ticks/news/fundamentals; publish `signals.v1`.  \n- **Execution**: OMS consumes signals; publishes `orders.v1` and consumes `fills.v1`; positions calculated and published.  \n- **Monitoring**: all services publish `metrics.v1` to a metrics sink; alerts on DLQ spikes/lag/latency.  \n- **Compliance**: orders/fills/positions retained per policy; audit derives from headers and payload hashes.\n\n---\n\n## 21) Roadmap (post‑v1)\n\n- **Helper SDKs**: `ampy-bus-go` and `ampy-bus-py` (envelopes, validation, tracing hooks, codecs).  \n- **CLI tools**: produce/consume/replay testers; DLQ inspector.  \n- **Schema registry hooks**: signature checks and schema hash enforcement.  \n- **Reference bindings**: NATS JetStream and Kafka examples.  \n- **Benchmarks**: publicly documented latency/throughput across brokers.\n\n---\n\n## 22) FAQ\n\n**Q: Why Protobuf instead of Avro/JSON?**  \nProtobuf gives compact, fast, cross‑language serialization and already underpins `ampy-proto`.\n\n**Q: Can we use both NATS and Kafka?**  \nYes. Contracts are transport‑agnostic. Bindings map headers/keys appropriately.\n\n**Q: Where do API keys live?**  \nIn each binding/producer via `ampy-config` or broker‑native secret stores. Never in code or headers.\n\n**Q: How do we handle currency conversions/news IDs/etc.?**  \nThose are **producers** (e.g., FX Go client, Marketbeat Go) that emit domain payloads. The bus contract remains unchanged.\n\n---\n\n## 23) Contributing\n\n- Open an issue describing changes to topics/headers/QoS before sending PRs.  \n- Include **golden envelopes** and **tests** for any new domain.  \n- Follow semantic versioning for header changes (additive only) and bump payload major in topics for breaking payload changes.\n\n---\n\n## 24) License\n\n**Proposed:** Apache‑2.0 (patent‑grant, enterprise‑friendly). *Confirm before first release.*\n\n---\n\n## 25) Badges / About (GitHub)\n\n**About:**  \n“Transport‑agnostic messaging conventions \u0026 helpers for AmpyFin. Standard topics, headers, QoS, replay, and observability over NATS or Kafka. Payloads are `ampy-proto`.”\n\n**Topics:** `trading-systems`, `messaging`, `protobuf`, `nats`, `kafka`, `event-driven`, `fintech`, `observability`, `slo`, `open-source`, `ampyfin`\n\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fampyfin%2Fampy-bus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fampyfin%2Fampy-bus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fampyfin%2Fampy-bus/lists"}