{"id":50777044,"url":"https://github.com/jbcom/secrets-sync","last_synced_at":"2026-06-12T00:30:39.758Z","repository":{"id":363749533,"uuid":"1264648800","full_name":"jbcom/secrets-sync","owner":"jbcom","description":"Enterprise-grade secret synchronization pipeline","archived":false,"fork":false,"pushed_at":"2026-06-10T07:55:42.000Z","size":1086,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-10T08:18:26.813Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jbcom.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"docs/SUPPORT.md","governance":null,"roadmap":"docs/ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-10T04:15:14.000Z","updated_at":"2026-06-10T04:47:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jbcom/secrets-sync","commit_stats":null,"previous_names":["jbcom/secrets-sync"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jbcom/secrets-sync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbcom%2Fsecrets-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbcom%2Fsecrets-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbcom%2Fsecrets-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbcom%2Fsecrets-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jbcom","download_url":"https://codeload.github.com/jbcom/secrets-sync/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbcom%2Fsecrets-sync/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34224103,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-12T00:30:34.891Z","updated_at":"2026-06-12T00:30:39.736Z","avatar_url":"https://github.com/jbcom.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# SecretSync\n\n**Enterprise-Grade Secret Synchronization Pipeline**\n\n[![⭐ Star on GitHub](https://img.shields.io/github/stars/jbcom/secrets-sync?style=social)](https://github.com/jbcom/secrets-sync/stargazers)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![GitHub release](https://img.shields.io/github/release/jbcom/secrets-sync.svg)](https://github.com/jbcom/secrets-sync/releases)\n[![Go Report Card](https://goreportcard.com/badge/github.com/jbcom/secrets-sync)](https://goreportcard.com/report/github.com/jbcom/secrets-sync)\n[![Python Bindings](https://img.shields.io/badge/python-bindings-blue.svg)](./python/)\n\n[Quick Start](#quick-start) • [Package Docs](https://extended-data.dev/packages/secretssync/) • [Repo Docs](./docs/) • [Python Bindings](#python-bindings) • [Examples](./examples/) • [GitHub Action](./docs/GITHUB_ACTIONS.md)\n\n\u003c/div\u003e\n\n---\n\nSecretSync provides **fully automated, enterprise-grade secret synchronization** across multiple cloud providers and secret stores. Built for scale with a **two-phase pipeline architecture** (merge → sync), it supports inheritance, dynamic target discovery, and CI/CD-friendly diff reporting.\n\n## 🏢 Part of Extended Data Library\n\nSecretSync is part of the [Extended Data Library](https://github.com/jbcom/secrets-sync) ecosystem - a collection of high-performance, enterprise-grade tools for data management, secret handling, and infrastructure automation.\n\n**🐍 Python Integration**: SecretSync provides Python bindings via [gopy](https://github.com/go-python/gopy), enabling seamless integration with the [extended-data](https://github.com/jbcom/extended-data) library and Python-based AI agents.\n\n**🚀 Perfect for:** Multi-account AWS environments, Kubernetes deployments, CI/CD pipelines, and enterprise secret management at scale.\n\n## 🤔 Why SecretSync?\n\n| Feature | SecretSync | Alternatives |\n|---------|------------|--------------|\n| **Two-Phase Pipeline** | ✅ Merge → Sync with inheritance | ❌ Simple 1:1 sync only |\n| **AWS Organizations** | ✅ Dynamic discovery with tag filtering | ❌ Manual account management |\n| **Secret Versioning** | ✅ Complete audit trail with rollback | ❌ No version tracking |\n| **Enhanced Diff** | ✅ Side-by-side with intelligent masking | ❌ Basic text diff |\n| **Enterprise Scale** | ✅ 1000+ accounts, circuit breakers | ❌ Limited scalability |\n| **CI/CD Integration** | ✅ GitHub Action + exit codes | ❌ Manual scripting required |\n\n## ✨ Key Features\n\n### 🔍 **Advanced Discovery** (v1.2.0)\n- **AWS Organizations Integration**: Discover accounts with tag filtering, wildcards, and OU-based selection\n- **AWS Identity Center**: Permission set discovery and account assignment mapping\n- **Smart Caching**: Multi-level caching for optimal performance at scale\n\n### 📚 **Secret Versioning** (v1.2.0)\n- **Complete Audit Trail**: Track every secret change with metadata\n- **S3-Based Storage**: Reliable, scalable version history\n- **Rollback Capability**: CLI support for version rollback\n- **Retention Policies**: Configurable cleanup of old versions\n\n### 🎨 **Enhanced Diff Output** (v1.2.0)\n- **Side-by-Side Comparison**: Visual diff with aligned columns and color coding\n- **Intelligent Masking**: Automatic detection and masking of sensitive values\n- **Multiple Formats**: Human, JSON, GitHub Actions, and compact outputs\n- **Rich Statistics**: Detailed change counts, sizes, and timing\n\n### 🛡️ **Enterprise Reliability** (v1.1.0)\n- **Circuit Breakers**: Automatic failure detection and recovery\n- **Prometheus Metrics**: Production-ready observability with `/metrics` endpoint\n- **Request Tracking**: Unique request IDs and duration tracking\n- **Race-Free Operations**: Thread-safe with comprehensive testing\n\n### 🏗️ **Pipeline Architecture**\n- **Two-Phase Design**: Merge → Sync for complex inheritance scenarios\n- **DeepMerge Support**: List append, dict merge, scalar override\n- **Target Inheritance**: Hierarchical configuration with circular dependency detection\n- **Dynamic Discovery**: AWS Organizations, Identity Center, and fuzzy matching\n\n## Attribution\n\nSecretSync originated as a fork of [robertlestak/vault-secret-sync](https://github.com/robertlestak/vault-secret-sync) (MIT License). We thank **Robert Lestak** for creating the original codebase.\n\n**SecretSync is an independent product** with its own roadmap and development direction. It has been substantially rewritten with:\n- Two-phase pipeline architecture (merge → sync)\n- S3 merge store support  \n- Dynamic target discovery (AWS Organizations, Identity Center)\n- Comprehensive diff/dry-run system with CI/CD integration\n- DeepMerge semantics for secret aggregation\n- Kubernetes operator with CRD support\n\n## Supported Secret Stores\n\n| Store | Source | Target | Merge Store |\n|-------|--------|--------|-------------|\n| HashiCorp Vault (KV2) | ✅ | ✅ | ✅ |\n| AWS Secrets Manager | ✅ | ✅ | ❌ |\n| AWS S3 | ❌ | ❌ | ✅ |\n| AWS Identity Center | Discovery | ❌ | ❌ |\n\n## Two-Phase Pipeline Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│                    MERGE PHASE (Optional)                        │\n│  Source1 ──┐                                                     │\n│  Source2 ──┼──▶ Merge Store (Vault/S3) ──▶ Aggregated Secrets   │\n│  Source3 ──┘    (deepmerge, inheritance)                         │\n└─────────────────────────────────────────────────────────────────┘\n                              │\n                              ▼\n┌─────────────────────────────────────────────────────────────────┐\n│                        SYNC PHASE                                │\n│  Merge Store ──┬──▶ AWS Account 1 (via STS AssumeRole)          │\n│  (or Source)   ├──▶ AWS Account 2                                │\n│                ├──▶ Vault Cluster                                │\n│                └──▶ GCP Project                                  │\n└─────────────────────────────────────────────────────────────────┘\n```\n\nSee [Two-Phase Architecture](./docs/TWO_PHASE_ARCHITECTURE.md) for detailed documentation.\n\n## Quick Start\n\n### Installation\n\n```bash\n# Go install\ngo install github.com/jbcom/secrets-sync/cmd/secretsync@latest\n\n# Or build from a local checkout\ngit clone https://github.com/jbcom/secrets-sync.git\ncd secrets-sync\nmake build\n```\n\n## Python Bindings\n\nSecretSync provides Python bindings via [gopy](https://github.com/go-python/gopy), enabling integration with Python applications and AI agent frameworks.\n\n### Building Python Bindings\n\n```bash\n# Install prerequisites\npip install pybindgen build\ngo install golang.org/x/tools/cmd/goimports@latest\ngo install github.com/go-python/gopy@latest\n\n# Build Python bindings\nmake python-bindings\n\n# Install locally\nmake python-install\n```\n\n### Using via extended-data\n\nThe recommended way to use SecretSync from Python is via the [extended-data](https://github.com/jbcom/extended-data) library:\n\n```bash\npip install extended-data[secrets]\n```\n\nThis installs the Python connector surface. To execute the full pipeline from\nPython, make sure the `secretsync` CLI is installed or the native bindings have\nbeen built in the current environment.\n\n```python\nfrom extended_data.secrets import SecretsConnector\n\n# Initialize connector\nconnector = SecretsConnector()\n\n# Validate configuration\nis_valid, message = connector.validate_config(\"pipeline.yaml\")\n\n# Dry run to see what would change\nresult = connector.dry_run(\"pipeline.yaml\")\nprint(f\"Would sync {result.secrets_processed} secrets\")\nprint(f\"  Add: {result.secrets_added}\")\nprint(f\"  Modify: {result.secrets_modified}\")\nprint(f\"  Remove: {result.secrets_removed}\")\n\n# Execute the full pipeline\nresult = connector.run_pipeline(\"pipeline.yaml\")\nif result.success:\n    print(f\"Successfully synced {result.secrets_added} secrets\")\n```\n\n### AI Agent Integration\n\nSecretSync tools are available for LangChain, CrewAI, and AWS Strands:\n\n```python\nfrom extended_data.secrets import get_tools\n\n# Auto-detect framework\ntools = get_tools()\n\n# Or specify framework\nlangchain_tools = get_tools(\"langchain\")\ncrewai_tools = get_tools(\"crewai\")\n```\n\n### Basic Usage\n\n```bash\n# Validate configuration\nsecretsync validate --config pipeline.yaml\n\n# Dry run with enhanced diff output (v1.2.0)\nsecretsync pipeline --config pipeline.yaml --dry-run --format side-by-side\n\n# Full pipeline execution with metrics (v1.1.0)\nsecretsync pipeline --config pipeline.yaml --metrics-port 9090\n\n# CI/CD mode (exit codes: 0=no changes, 1=changes, 2=errors)\nsecretsync pipeline --config pipeline.yaml --dry-run --exit-code\n\n# Version management (v1.2.0)\nsecretsync versions --secret-path \"app/database/password\"\nsecretsync sync --version 5 --target production\n```\n\n### Example Configuration\n\n```yaml\n# pipeline.yaml - v1.2.0 with advanced features\nvault:\n  address: \"https://vault.example.com\"\n  namespace: \"admin\"\n\naws:\n  region: \"us-east-1\"\n  execution_role_pattern: \"arn:aws:iam::{account_id}:role/SecretsSync\"\n\n# Advanced discovery (v1.2.0)\ndiscovery:\n  aws_organizations:\n    enabled: true\n    tag_filters:\n      - key: \"Environment\"\n        values: [\"production\", \"staging\"]\n        operator: \"equals\"\n      - key: \"Team\"\n        values: [\"platform*\"]\n        operator: \"contains\"\n    organizational_units:\n      - \"ou-production-12345\"\n    tag_logic: \"AND\"\n    cache_ttl: \"1h\"\n  \n  identity_center:\n    enabled: true\n    region: \"us-east-1\"\n    cache_ttl: \"30m\"\n\n# Secret versioning (v1.2.0)\nversioning:\n  enabled: true\n  s3_bucket: \"company-secretsync-versions\"\n  retention_days: 90\n\n# Observability (v1.1.0)\nobservability:\n  metrics:\n    enabled: true\n    port: 9090\n    address: \"0.0.0.0\"\n\nmerge_store:\n  vault:\n    mount: \"secret/merged\"\n\nsources:\n  api-keys:\n    vault:\n      path: \"secret/api-keys\"\n  database:\n    vault:\n      path: \"secret/database\"\n\ntargets:\n  Staging:\n    imports: [api-keys, database]\n    account_id: \"111111111111\"\n  \n  Production:\n    inherits: Staging\n    imports: [production-overrides]\n    account_id: \"222222222222\"\n```\n\n## GitHub Actions\n\nSecretSync is available as a GitHub Action for seamless CI/CD integration:\n\n```yaml\n- name: Sync Secrets\n  uses: jbcom/secrets-sync@secretssync-v2.0.2\n  with:\n    config: config.yaml\n    dry-run: 'false'\n    output-format: 'github'\n  env:\n    VAULT_ROLE_ID: ${{ secrets.VAULT_ROLE_ID }}\n    VAULT_SECRET_ID: ${{ secrets.VAULT_SECRET_ID }}\n```\n\n**Key Features:**\n- 🔒 Native OIDC support for AWS authentication\n- 📊 GitHub-native diff annotations in PRs\n- 🎯 Exit codes for CI/CD control flow\n- 🔄 Automatic Docker multi-arch builds\n- ⚡ Zero configuration needed beyond config file\n\n**Quick Start:**\n1. Add `config.yaml` to your repository\n2. Configure AWS OIDC and Vault secrets\n3. Use the action in your workflow\n\nSee [GitHub Actions documentation](./docs/GITHUB_ACTIONS.md) for complete usage guide and examples.\n\n## CI/CD Integration (CLI)\n\n### GitHub Actions (CLI)\n\n```yaml\n- name: Validate secrets pipeline\n  run: |\n    secretsync pipeline --config pipeline.yaml --dry-run --output github --exit-code\n  \n- name: Apply secrets (on merge to main)\n  if: github.ref == 'refs/heads/main'\n  run: |\n    secretsync pipeline --config pipeline.yaml\n```\n\n### Output Formats (Enhanced in v1.2.0)\n\n| Format | Use Case | Features |\n|--------|----------|----------|\n| `human` | Interactive terminal output | Color coding, readable layout |\n| `side-by-side` | **NEW** Visual comparison | Aligned columns, intelligent masking |\n| `json` | Machine parsing, logging | Structured data with metadata |\n| `github` | GitHub Actions annotations | PR comments, file annotations |\n| `compact` | One-line CI status | Minimal output for scripts |\n\n**Value Masking (v1.2.0)**: Sensitive values are automatically masked by default. Use `--show-values` flag to display actual values (use with caution in CI/CD).\n\n## 📚 Documentation\n\n### Getting Started\n- [🌐 Published Package Docs](https://extended-data.dev/packages/secretssync/) - Public package overview, installation paths, and Python integration guidance\n- [🚀 Getting Started Guide](./docs/GETTING_STARTED.md) - Step-by-step setup tutorial\n- [❓ FAQ](./docs/FAQ.md) - Frequently asked questions\n- [📋 Examples](./examples/) - Complete configuration examples\n\n### Core Documentation\n- [🏗️ Architecture Overview](./docs/ARCHITECTURE.md) - System design and components\n- [🔄 Two-Phase Pipeline](./docs/TWO_PHASE_ARCHITECTURE.md) - Merge → Sync architecture\n- [⚙️ Pipeline Configuration](./docs/PIPELINE.md) - Configuration reference\n- [🚀 Deployment Guide](./docs/DEPLOYMENT.md) - Production deployment patterns\n\n### Advanced Topics\n- [🔒 Security Configuration](./docs/SECURITY.md) - Security best practices\n- [📊 Observability](./docs/OBSERVABILITY.md) - Monitoring and metrics\n- [🎯 GitHub Actions](./docs/GITHUB_ACTIONS.md) - CI/CD integration guide\n- [📖 Usage Reference](./docs/USAGE.md) - Complete CLI reference\n\n### Community\n- [🗺️ Roadmap](./docs/ROADMAP.md) - Future development plans\n- [🤝 Contributing](./CONTRIBUTING.md) - How to contribute\n- [🛡️ Security Policy](./SECURITY.md) - Security reporting\n- [📜 Code of Conduct](./CODE_OF_CONDUCT.md) - Community guidelines\n\n## Helm Deployment\n\n```bash\n# Add Helm repo\nhelm repo add secretsync https://jbcom.github.io/secrets-sync\n\n# Install\nhelm install secretsync secretsync/secretsync \\\n  --set vault.address=https://vault.example.com\n```\n\n## Docker\n\n```bash\n# Run with config file\ndocker run -v $(pwd)/config.yaml:/config.yaml \\\n  jbcom/secrets-sync-secretssync pipeline --config /config.yaml\n\n# Multi-arch images available: linux/amd64, linux/arm64\n```\n\n## Observability\n\nSecretSync exposes Prometheus metrics for production monitoring and debugging.\n\n### Enabling Metrics\n\n```bash\n# Enable metrics server on port 9090\nsecretsync pipeline --config config.yaml --metrics-port 9090\n\n# Custom address and port\nsecretsync pipeline --config config.yaml --metrics-addr 0.0.0.0 --metrics-port 9090\n```\n\n### Available Metrics\n\n**Vault Metrics:**\n- `secretsync_vault_api_call_duration_seconds` - Vault API call latency\n- `secretsync_vault_secrets_listed_total` - Total secrets listed from Vault\n- `secretsync_vault_traversal_depth` - BFS traversal depth reached\n- `secretsync_vault_queue_size` - Current traversal queue size\n- `secretsync_vault_errors_total` - Vault error count by operation/type\n\n**AWS Metrics:**\n- `secretsync_aws_api_call_duration_seconds` - AWS API call latency\n- `secretsync_aws_pagination_pages` - Number of pagination pages processed\n- `secretsync_aws_cache_hits_total` - Cache hit count\n- `secretsync_aws_cache_misses_total` - Cache miss count\n- `secretsync_aws_secrets_operations_total` - Secret operations (create/update/delete)\n\n**Pipeline Metrics:**\n- `secretsync_pipeline_execution_duration_seconds` - Pipeline phase duration\n- `secretsync_pipeline_targets_processed_total` - Targets processed by phase\n- `secretsync_pipeline_parallel_workers` - Active parallel workers\n- `secretsync_pipeline_errors_total` - Pipeline error count\n\n**S3 Metrics:**\n- `secretsync_s3_operation_duration_seconds` - S3 operation latency\n- `secretsync_s3_object_size_bytes` - S3 object sizes\n\n### Prometheus Configuration\n\n```yaml\nscrape_configs:\n  - job_name: 'secretsync'\n    static_configs:\n      - targets: ['localhost:9090']\n    metrics_path: '/metrics'\n```\n\n### Health Check\n\nThe metrics server also exposes a `/health` endpoint:\n\n```bash\ncurl http://localhost:9090/health\n# Returns: OK\n```\n\n## Development\n\n```bash\n# Clone\ngit clone https://github.com/jbcom/secrets-sync.git\ncd secrets-sync\n\n# Build\ngo build ./...\n\n# Unit tests\ngo test ./...\n\n# Integration tests (requires Docker)\nmake test-integration-docker\n\n# Lint\ngolangci-lint run\n```\n\n### Integration Testing\n\nSecretSync includes comprehensive integration tests that validate the complete pipeline with real Vault and AWS Secrets Manager instances (via LocalStack).\n\n**Quick Start:**\n```bash\n# Run complete integration test suite\nmake test-integration-docker\n```\n\nThis command:\n- Starts Vault and LocalStack in Docker containers\n- Seeds test data automatically\n- Runs all integration tests\n- Cleans up containers\n\n**Manual Testing:**\n```bash\n# Start test environment\nmake test-env-up\n\n# Export environment variables (shown in output)\nexport VAULT_ADDR=http://localhost:8200\nexport VAULT_TOKEN=test-root-token\nexport AWS_ENDPOINT_URL=http://localhost:4566\nexport AWS_ACCESS_KEY_ID=test\nexport AWS_SECRET_ACCESS_KEY=test\n\n# Run tests\ngo test -v -tags=integration ./tests/integration/...\n\n# Cleanup\nmake test-env-down\n```\n\nFor detailed documentation, see [tests/integration/README.md](./tests/integration/README.md).\n\n## 🌟 Community \u0026 Support\n\n### Getting Help\n- **📚 Documentation**: Start with the [published package docs](https://extended-data.dev/packages/secretssync/) and the repo-local [docs folder](./docs/)\n- **🐛 GitHub Issues**: Questions, bug reports, and feature requests\n- **🔒 Security**: Private security vulnerability reporting\n\n### Contributing\nWe welcome contributions! See our [Contributing Guide](./CONTRIBUTING.md) for:\n- 🛠️ Development setup\n- 📝 Code style guidelines  \n- 🧪 Testing requirements\n- 📋 Pull request process\n\n### Community\n- **⭐ Star the repo** to show your support\n- **🐦 Follow updates** on GitHub\n- **📢 Share** your success stories\n- **🤝 Contribute** code, docs, or feedback\n\n## 📄 License\n\n[MIT License](./LICENSE) - Free for commercial and personal use\n\n## 🙏 Attribution\n\nSecretSync originated as a fork of [vault-secret-sync](https://github.com/robertlestak/vault-secret-sync) by **Robert Lestak**. We thank Robert for creating the original foundation.\n\nSecretSync has evolved into an independent project with its own architecture, features, and roadmap, while maintaining the same MIT license and open-source spirit.\n\n**Current Maintainer**: [jbcom](https://github.com/jbcom)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjbcom%2Fsecrets-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjbcom%2Fsecrets-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjbcom%2Fsecrets-sync/lists"}