Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/sethbacon/terraform-registry-backend

Private Terraform registry implementing the HashiCorp module, provider, and network mirror protocols
https://github.com/sethbacon/terraform-registry-backend

gin go infrastructure-as-code postgresql private-registry terraform terraform-registry

Last synced: about 1 month ago
JSON representation

Private Terraform registry implementing the HashiCorp module, provider, and network mirror protocols

Awesome Lists containing this project

README 

          

# Enterprise Terraform Registry — Backend



A fully-featured, enterprise-grade Terraform registry implementing all three HashiCorp protocols with multi-tenancy support.



[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)

[![Go Version](https://img.shields.io/badge/Go-1.26+-00ADD8?logo=go)](https://go.dev/)

[![Coverage](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/sethbacon/59239e8575b4f784f875647e2b344b41/raw/coverage.json)](https://github.com/sethbacon/terraform-registry-backend/actions/workflows/ci.yml)



This repository contains the backend API, database migrations, and deployment infrastructure for the Enterprise Terraform Registry. The frontend UI is a separate React SPA: **[terraform-registry-frontend](https://github.com/sethbacon/terraform-registry-frontend)**.



## Features



### Terraform Protocol Support



- **Module Registry Protocol** — Complete implementation for hosting and discovering Terraform modules

- **Provider Registry Protocol** — Full provider hosting with platform-specific binaries

- **Provider Network Mirror Protocol** — Efficient provider mirroring and caching



### Authentication & Authorization



- **API Key Authentication** — Secure token-based access with scoped permissions

- **OIDC Integration** — Generic OIDC provider support for SSO

- **Azure AD / Entra ID** — Native Azure Active Directory integration

- **Role-Based Access Control (RBAC)** — Fine-grained permissions with organization roles



### Multi-Tenancy



- **Organization Management** — Isolated organization namespaces for teams and projects

- **User Management** — Comprehensive user administration

- **Organization Membership** — Role-based team collaboration (viewer, publisher, devops, user_manager, auditor, admin)

- **Configurable Modes** — Single-tenant or multi-tenant deployment



### Module Source Control (SCM) Integration



- **GitHub Integration** — Connect modules to GitHub repositories with OAuth

- **Azure DevOps Integration** — Native support for Azure Repos

- **GitLab Integration** — Full GitLab repository support

- **Bitbucket Data Center** — Self-hosted Bitbucket with PAT authentication

- **Webhook Support** — Automatic publishing on repository events



### Storage Backends



- **Local Filesystem** — Direct file serving for development and simple deployments

- **Azure Blob Storage** — Cloud storage with SAS tokens, CDN URLs, and flexible access tiers

- **AWS S3 / S3-Compatible** — AWS S3, MinIO, and DigitalOcean Spaces with presigned URLs

- **Google Cloud Storage** — Native GCS integration with signed URLs and resumable uploads

- **Pluggable Architecture** — Extensible storage interface for adding new backends



### Deployment Options



Docker Compose, Kubernetes (Kustomize and Helm), Azure Container Apps, AWS ECS

Fargate, Google Cloud Run, standalone binary + systemd, and full Terraform IaC

for all three major clouds. See the [Deployment Guide](docs/deployment.md) for

the full matrix, recommended targets, and step-by-step instructions.



### Terraform Binary Mirror



- **Multi-Config Mirror** — Multiple named mirror configs coexist; independently mirror HashiCorp Terraform, OpenTofu, or custom upstream sources side-by-side

- **Tool Support** — `terraform`, `opentofu`, and `custom` tool types with per-config upstream URL

- **Supply-Chain Security** — GPG signature verification against the embedded HashiCorp release key (OpenTofu support configurable)

- **Platform Filtering** — Optionally restrict which `os/arch` combinations are downloaded and served

- **Public Download API** — Unauthenticated endpoints at `/terraform/binaries/:name/versions/…` compatible with Terraform's [network mirror protocol](https://developer.hashicorp.com/terraform/internals/provider-network-mirror-protocol)

- **Prometheus Metric** — `terraform_binary_downloads_total` counter with `{version, os, arch}` labels



### Module Security Scanning



- **Automatic Scanning** — Every uploaded module version is automatically queued for a security scan

- **Multiple Scanner Backends** — Trivy (recommended), Checkov, Terrascan, Snyk, or a custom binary

- **Structured Results** — Severity counts (Critical/High/Medium/Low) and raw output stored per version

- **Supply-Chain Protection** — Optional binary version pinning rejects mismatched scanner executables

- **Module Documentation Extraction** — Inputs, outputs, provider requirements, and Terraform version constraints automatically extracted at upload time



### Observability & CI/CD



- **Prometheus Metrics** — Eleven named application metrics on a dedicated scrape port (default 9090)

- **Structured Logging** — stdlib `slog` with JSON (production) and text (development) formats

- **pprof Profiling** — Opt-in profiling server on a configurable port

- **GitHub Actions CI/CD** — Build, vet, race-detector tests, gosec security scan, Docker build, multi-platform releases

- **Bi-weekly Dependabot** — Automated Go module and Actions dependency updates



## Tech Stack



| Concern         | Technology                                                  |

| --------------- | ----------------------------------------------------------- |

| Language        | Go 1.26                                                     |

| HTTP framework  | Gin                                                         |

| Database        | PostgreSQL 14+ (16 recommended)                             |

| Migrations      | `golang-migrate` (file-based, immutable numbered pairs)     |

| Auth            | JWT (HS256), API keys, OIDC, Azure AD                       |

| Storage         | Local FS, Azure Blob, AWS S3 (+compatible), Google Cloud    |

| SCM             | GitHub, Azure DevOps, GitLab, Bitbucket Data Center         |

| Observability   | Prometheus metrics, `slog` JSON logs, pprof                 |

| API docs        | Swagger 2.0 (swag annotations) → served at `/swagger`       |

| Build / release | GoReleaser, GitHub Actions, cosign keyless, SLSA, syft SBOM |



## Architecture



```txt

┌────────────────────────────────────────────────────┐

│              React TypeScript SPA                  │

│  (see terraform-registry-frontend)                 │

└──────────────────┬─────────────────────────────────┘

                   │ REST API / Protocol Endpoints

┌──────────────────▼─────────────────────────────────┐

│              Go 1.26 Backend (Gin)                 │

│  Modules API │ Providers API │ Mirror │ Admin      │

│  Terraform Binary Mirror                           │

│  Auth: JWT │ API Keys │ OIDC │ Azure AD │ RBAC     │

│  SCM: GitHub │ Azure DevOps │ GitLab │ Bitbucket   │

│  Storage: Local │ Azure Blob │ S3 │ GCS            │

└──────────────────┬─────────────────────────────────┘

                   │

          ┌────────┴────────┐

          │   PostgreSQL    │

          └─────────────────┘

```



## Installation



### Prerequisites



- Go 1.26 or later

- PostgreSQL 14 or later (16 recommended; used in Docker Compose and CI)

- Docker & Docker Compose (for containerized deployment)



### Quick Start with Docker Compose



```bash

# Clone the repository

git clone https://github.com/sethbacon/terraform-registry-backend.git

cd terraform-registry-backend



# Start all services (backend + postgres; frontend served separately)

cd deployments

docker-compose up -d



# Backend API: http://localhost:8080

# PostgreSQL:  localhost:5432

# Prometheus metrics: http://localhost:9090/metrics

```



> For the web UI, see [terraform-registry-frontend](https://github.com/sethbacon/terraform-registry-frontend).



### First-Run Setup



On first startup, the server prints a one-time **setup token** to the logs. Use this token to configure the registry through the web wizard at `/setup` or via the setup API. The wizard guides you through:



1. OIDC provider configuration (authentication)

2. Storage backend configuration (where modules/providers are stored)

3. Initial admin user setup



See [Initial Setup Guide](docs/initial-setup.md) for full details, including headless/curl-based setup.



### Manual Setup



```bash

cd backend



# Install dependencies

go mod download



# Set up configuration

cp config.example.yaml config.yaml

# Edit config.yaml with your settings



# Run database migrations

go run cmd/server/main.go migrate up



# Start the server

go run cmd/server/main.go serve

```



## Configuration



All configuration can be set via environment variables (prefix: `TFR_`) or YAML config file.



```bash

# Database

export TFR_DATABASE_HOST=localhost

export TFR_DATABASE_PORT=5432

export TFR_DATABASE_USER=registry

export TFR_DATABASE_PASSWORD=your_password

export TFR_DATABASE_NAME=terraform_registry

export TFR_DATABASE_SSL_MODE=disable



# Server

export TFR_SERVER_PORT=8080

export TFR_SERVER_HOST=0.0.0.0

export TFR_SERVER_BASE_URL=http://localhost:8080



# Storage (choose one: local | azure | s3 | gcs)

export TFR_STORAGE_DEFAULT_BACKEND=local

export TFR_STORAGE_LOCAL_BASE_PATH=/var/lib/terraform-registry



# Authentication

export TFR_AUTH_API_KEYS_ENABLED=true

export TFR_AUTH_OIDC_ENABLED=false

export TFR_AUTH_AZURE_AD_ENABLED=false



# Security

export TFR_JWT_SECRET=your_jwt_secret          # min 32 chars in production

export ENCRYPTION_KEY=your_32_byte_encryption_key

```



See `backend/config.example.yaml` for a complete configuration reference.

See [Configuration Reference](docs/configuration.md) for detailed guidance.



## Monitoring & Observability



The registry exposes Prometheus metrics on a dedicated port (default **9090**).



```bash

# View all metrics

curl -s http://localhost:9090/metrics



# Check HTTP request rates

curl -s http://localhost:9090/metrics | grep '^http_requests_total'

```



### Prometheus + Grafana Stack



```bash

cd deployments

docker-compose --profile monitoring up -d

# Prometheus UI: http://localhost:9091

# Grafana:       http://localhost:3001  (admin / admin)

```



See [Observability Reference](docs/observability.md) for Grafana dashboard setup and alert rules.



## Usage with Terraform



```hcl

terraform {

  required_providers {

    mycloud = {

      source  = "registry.example.com/myorg/mycloud"

      version = "~> 1.0"

    }

  }

}



module "vpc" {

  source  = "registry.example.com/myorg/vpc/aws"

  version = "2.1.0"

}

```



### Publishing Modules



```bash

curl -X POST https://registry.example.com/api/v1/modules \

  -H "Authorization: Bearer YOUR_API_KEY" \

  -F "file=@module.tar.gz" \

  -F "namespace=myorg" \

  -F "name=vpc" \

  -F "system=aws" \

  -F "version=1.0.0"

```



### Publishing Providers



```bash

curl -X POST https://registry.example.com/api/v1/providers \

  -H "Authorization: Bearer YOUR_API_KEY" \

  -F "file=@terraform-provider-mycloud_1.0.0_linux_amd64.zip" \

  -F "namespace=myorg" \

  -F "type=mycloud" \

  -F "version=1.0.0" \

  -F "os=linux" \

  -F "arch=amd64" \

  -F "gpg_public_key=@public_key.asc"

```



## Development



### Running Tests



```bash

cd backend

go test ./...

```



### Building



```bash

cd backend

go build -o terraform-registry cmd/server/main.go

```



## Documentation



- [Getting Started](docs/getting-started.md) — Quick-start tutorial for new users

- [Architecture](docs/architecture.md) — System design and component interactions

- [Architecture Decision Records](docs/adr/) — Key design decisions and rationale

- [API Reference](docs/api-reference.md) — API documentation guide

- [API Migration Guide](docs/api-migration-guide.md) — Breaking changes and migration steps across releases

- [Configuration Reference](docs/configuration.md) — All `TFR_*` environment variables

- [Deployment Guide](docs/deployment.md) — Production deployment for all platforms

- [Kubernetes Cloud Guides](docs/deployment/README.md) — AKS, EKS, and GKE deployment guides

- [Capacity Planning](docs/capacity-planning.md) — Sizing guidance for production deployments

- [Disaster Recovery](docs/disaster-recovery.md) — DR runbook and backup/restore procedures

- [Observability Reference](docs/observability.md) — Prometheus metrics catalogue

- [Troubleshooting](docs/troubleshooting.md) — Common issues and diagnostic tools

- [Development Guide](docs/development.md) — Local development setup

- [OIDC Configuration](docs/OIDC_CONFIGURATION.md) — SSO provider setup

- [Module Security Scanning](docs/module-scanning.md) — Scanner setup (Trivy, Checkov, Terrascan, Snyk, custom)

- [Module Documentation Extraction](docs/module-documentation.md) — Automatic extraction of inputs, outputs, and provider requirements

- [Terraform CLI Configuration](docs/terraform-cli-configuration.md) — Configure the Terraform CLI to use this registry

- [Releasing](RELEASING.md) — Release process and supply-chain verification

- [Security](SECURITY.md) — Reporting vulnerabilities and supported versions

- [Contributing](CONTRIBUTING.md) — How to contribute to this project

- [Code of Conduct](CODE_OF_CONDUCT.md) — Community standards

- [Changelog](CHANGELOG.md) — Version history and changes

- [Frontend Repository](https://github.com/sethbacon/terraform-registry-frontend) — React SPA, accessibility, E2E tests



## References



- [Module Registry Protocol](https://developer.hashicorp.com/terraform/internals/module-registry-protocol)

- [Provider Registry Protocol](https://developer.hashicorp.com/terraform/internals/provider-registry-protocol)

- [Provider Network Mirror Protocol](https://developer.hashicorp.com/terraform/internals/provider-network-mirror-protocol)



**Frontend:** [terraform-registry-frontend](https://github.com/sethbacon/terraform-registry-frontend)



## Supply Chain Security



Every release includes build provenance attestations (via GitHub Artifact Attestations), SBOM generation, and Sigstore cosign signatures published to the [Rekor](https://rekor.sigstore.dev) public transparency log.



### Verify a container image



```bash

# Verify the cosign signature and Sigstore identity

cosign verify \

  --certificate-identity-regexp 'https://github\.com/sethbacon/terraform-registry-backend/' \

  --certificate-oidc-issuer https://token.actions.githubusercontent.com \

  ghcr.io/sethbacon/terraform-registry-backend:latest



# Search the Rekor transparency log for entries

rekor-cli search --sha 

```



### Verify release binaries



```bash

# Verify the checksum file signature

cosign verify-blob \

  --certificate-identity-regexp 'https://github\.com/sethbacon/terraform-registry-backend/' \

  --certificate-oidc-issuer https://token.actions.githubusercontent.com \

  --signature checksums.txt.sig \

  --certificate checksums.txt.pem \

  checksums.txt



# Verify build provenance

gh attestation verify  --repo sethbacon/terraform-registry-backend

```



## Contributing



Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) before submitting pull requests. Key requirements:



- `go fmt ./...` and `go vet ./...` must pass

- New handlers require Swagger annotations

- New features require documentation updates

- Security issues must be reported privately (see CONTRIBUTING.md)



## License



This project is licensed under the Apache License, Version 2.0 — see the [LICENSE](LICENSE) file for details.



## Disclaimer



This software is provided **"AS IS"**, without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose, and non-infringement. In no event shall the authors or copyright holders be liable for any claim, damages, or other liability arising from the use of this software. See the [Apache 2.0 License](LICENSE) (Sections 7–8) for the full warranty disclaimer and limitation of liability.



**Operational security is the responsibility of the deploying organization.** This includes, but is not limited to: securing the deployment environment, managing secrets and credentials, configuring TLS, enforcing network boundaries, rotating API keys, auditing access, keeping dependencies up to date, and validating the fitness of this software for your specific compliance and security requirements. The maintainers make no guarantees regarding the security posture of any deployment.



## Acknowledgments



- [HashiCorp Terraform](https://www.terraform.io/) - Module and Provider protocols

- [Gin Web Framework](https://gin-gonic.com/) - Go HTTP framework



### Trademark Notice



HashiCorp, Terraform, and Vault are trademarks of HashiCorp, Inc. AWS and Amazon Web Services are trademarks of Amazon.com, Inc. or its affiliates. Microsoft Azure is a trademark of Microsoft Corporation. Google Cloud is a trademark of Google LLC. VMware and vSphere are trademarks of Broadcom Inc. Oracle and OCI are trademarks of Oracle Corporation. All other trademarks are the property of their respective owners.



This project is not affiliated with, endorsed by, or sponsored by any of the above-named organizations.