{"id":13515489,"url":"https://github.com/seal-io/hermitcrab","last_synced_at":"2025-04-14T08:40:52.066Z","repository":{"id":214167995,"uuid":"735817312","full_name":"seal-io/hermitcrab","owner":"seal-io","description":"Available Terraform Provider network mirroring service.","archived":false,"fork":false,"pushed_at":"2025-03-13T00:36:07.000Z","size":1046,"stargazers_count":28,"open_issues_count":1,"forks_count":4,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-27T22:11:59.188Z","etag":null,"topics":["mirroring","opentofu","provider","registry","terraform"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/seal-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-26T06:53:25.000Z","updated_at":"2025-03-25T17:32:35.000Z","dependencies_parsed_at":"2024-01-03T03:35:06.889Z","dependency_job_id":"ca8a3e4d-1e6c-41e8-954b-2553c98f38b0","html_url":"https://github.com/seal-io/hermitcrab","commit_stats":null,"previous_names":["seal-io/hermitcrab"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seal-io%2Fhermitcrab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seal-io%2Fhermitcrab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seal-io%2Fhermitcrab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seal-io%2Fhermitcrab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/seal-io","download_url":"https://codeload.github.com/seal-io/hermitcrab/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248848495,"owners_count":21171387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mirroring","opentofu","provider","registry","terraform"],"created_at":"2024-08-01T05:01:11.936Z","updated_at":"2025-04-14T08:40:52.042Z","avatar_url":"https://github.com/seal-io.png","language":"Go","funding_links":[],"categories":["Tools","Go"],"sub_categories":["Registry"],"readme":"# Hermit Crab \n\n\u003e tl;dr: Available Terraform/OpenTofu Provider network mirroring service.\n\n[![](https://goreportcard.com/badge/github.com/seal-io/hermitcrab)](https://goreportcard.com/report/github.com/seal-io/hermitcrab)\n[![](https://img.shields.io/github/actions/workflow/status/seal-io/hermitcrab/ci.yml?label=ci)](https://github.com/seal-io/hermitcrab/actions)\n[![](https://img.shields.io/docker/image-size/sealio/hermitcrab/main?label=docker)](https://hub.docker.com/r/sealio/hermitcrab/tags)\n[![](https://img.shields.io/github/v/tag/seal-io/hermitcrab?label=release)](https://github.com/seal-io/hermitcrab/releases) \n[![](https://img.shields.io/github/license/seal-io/hermitcrab?label=license)](https://github.com/seal-io/hermitcrab#license)\n\nHermit Crab provides a stable and reliable [Terraform](https://registry.terraform.io/browse/providers)/[OpenTofu](https://opentofu.org/registry/) Provider network mirror service. \n\nThis tool is maintained by [Seal](https://github.com/seal-io).\n\n```mermaid\nsequenceDiagram\n    actor tf  as terraform init\n    participant hc as Hermit Crab\n    participant tfreg as Terraform Registry\n    participant stg as Provider Package Storage\n    \n    tf -\u003e\u003e hc: list available versions\n    alt not found\n        hc -\u003e\u003e tfreg: list available versions\n        tfreg --\u003e\u003e hc: \n        hc -\u003e\u003e hc: save\n    end\n    hc --\u003e\u003e tf: {\"versions\":{\"2.0.0\":{},\"2.0.1\":{}}}\n    \n    tf -\u003e\u003e hc: list available installation packages\n    alt not found\n        par\n            hc -\u003e\u003e tfreg: find darwin/amd64 provider package\n            tfreg --\u003e\u003e hc: \n            hc -\u003e\u003e hc: save\n        and\n            hc -\u003e\u003e tfreg: find linux/amd64 provider package\n            tfreg --\u003e\u003e hc: \n            hc -\u003e\u003e hc: save\n        end\n    end\n    hc --\u003e\u003e tf: {\"archives\": {\"darwin_amd64\":{},\"linux_amd64\":{}}\n    \n    tf -\u003e\u003e hc: download platform provider package, like darwin/amd64\n    alt not found\n        par not downloading\n            hc -\u003e\u003e stg: download\n            stg --\u003e\u003e hc: \n            hc -\u003e\u003e hc: store \n        and downloading\n            hc -\u003e\u003e hc: wait until downloading finished\n        end\n    end\n    hc --\u003e\u003e tf: ***.zip\n\n```\n\n## Background\n\nWhen we drive [Terraform](https://www.terraform.io/) at some automation scenarios, like CI, automatic deployment, etc., we need to download the Provider plugins from the internet by `init` command. \n\nDepending on [Terraform Provider Registry Protocol](https://developer.hashicorp.com/terraform/internals/provider-registry-protocol), we may download a plugin that is not cached from an unstable networking remote.\n\nTo mitigate the effect of unstable networking, there are two ways to solve this: [Implied Mirroring](https://developer.hashicorp.com/terraform/cli/config/config-file#implied-local-mirror-directories) and [Network Mirroring](https://developer.hashicorp.com/terraform/cli/config/config-file#network_mirror).\n\nAs far as **Implied Mirroring** is concerned, it works well when a Provider matches the [Version Constraints](https://developer.hashicorp.com/terraform/language/expressions/version-constraints). However, this mode fails when the version is not cached in the local file directory.\n\n```\n╷\n│ Error: Failed to query available provider packages\n│\n│ Could not retrieve the list of available versions for provider\n```\n\nWhat's even more troublesome is that if the version changes, we need to continuously maintain this local file directory.\n\n**Network Mirroring**, different from **Implied Mirroring**, can maintain all versions(including the latest) at a nearby network and allows distributed Terraform working agents to share the same mirroring package.\n\n## Usage\n\nHermit Crab implements the [Terraform](https://developer.hashicorp.com/terraform/internals/provider-registry-protocol)/[OpenTofu](https://opentofu.org/docs/internals/provider-network-mirror-protocol/) Provider Registry Protocol and acts as a mirroring service.\n\nHermit Crab can be easily served through [Docker](https://www.docker.com/).\n\n```shell\ndocker run -d --restart=always -p 80:80 -p 443:443 sealio/hermitcrab\n```\n\nHermit Crab saves the mirrored(downloaded) packages in the `/var/run/hermitcrab` directory by default, which can be persisted by mounting a volume.\n\n```shell\ndocker run -d --restart=always -p 80:80 -p 443:443 \\\n  -v /tmp/hermitcrab:/var/run/hermitcrab \\\n  sealio/hermitcrab\n```\n\nHermit Crab manages the archives as the following layer structure, which is absolutely compatible with the output of [`terraform providers mirror`](https://developer.hashicorp.com/terraform/cli/commands/providers/mirror)/[`tofu providers mirror`](https://opentofu.org/docs/cli/commands/providers/mirror).\n\n```\n/var/run/hermitcrab/data/providers\n├── /\u003cHOSTNAME\u003e\n│   ├── /\u003cNAMESPACE\u003e\n│   │   ├── /\u003cTYPE\u003e\n│   │   │   ├── terraform-provider-\u003cTYPE\u003e_\u003cVERSION\u003e_\u003cOS\u003e_\u003cARCH\u003e.zip\n```\n\nHermit Crab also can reuse the mirroring providers prepared by `terraform providers mirror`/`tofo providers mirror`.\n\n```shell\nterraform providers mirror /tmp/providers-plugins\n\ndocker run -d --restart=always -p 80:80 -p 443:443 \\\n  -v /tmp/providers-plugins:/usr/share/terraform/providers/plugins \\\n  sealio/hermitcrab\n```\n\nTerraform/OpenTofu Provider Network Mirror protocol wants [HTTPS](https://en.wikipedia.org/wiki/HTTPS) access, Hermit Crab provides multiple ways to achieve this.\n\n- Use the default self-signed certificate, no additional configuration is required.\n\n  Since Terraform always verifies the certificate insecure or not, under this mode, we need to import the self-signed certificate into the trusted certificate store.\n\n  ```shell\n  # download the self-signed certificate\n  echo quit | openssl s_client -showcerts -servername \u003cYOUR_SERVER\u003e -connect \u003cYOUR_ADDRESS\u003e 2\u003e/dev/null | openssl x509 -outform PEM \u003eserver.pem\n  ```  \n\n- Using [ACME](https://en.wikipedia.org/wiki/Automatic_Certificate_Management_Environment) to [gain a trusted certificate](https://letsencrypt.org/docs/challenge-types/), need a domain name and a DNS configuration.\n\n  ```shell\n  docker run -d --restart=always -p 80:80 -p 443:443 \\\n    -e SERVER_TLS_AUTO_CERT_DOMAINS=\u003cYOUR_DOMAIN_NAME\u003e \\\n    sealio/hermitcrab\n  ```\n\n- Use a custom certificate.\n  \n  ```shell\n  docker run -d --restart=always -p 80:80 -p 443:443 \\\n    -v /\u003cYOUR_PRIVATE_KEY_FILE\u003e:/etc/hermitcrab/ssl/key.pem \\\n    -v /\u003cYOUR_CERT_FILE\u003e:/etc/hermitcrab/ssl/cert.pem \\\n    -e SERVER_TLS_PRIVATE_KEY_FILE=/etc/hermitcrab/ssl/key.pem \\\n    -e SERVER_TLS_CERT_FILE=/etc/hermitcrab/ssl/cert.pem \\\n    sealio/hermitcrab\n  ```\n\nAlso support to launch from Helm Chart.\n\n```shell\n# latest version\nhelm install my-release oci://ghcr.io/seal-io/helm-charts/hermitcrab\n# with specific version\nhelm install my-release oci://ghcr.io/seal-io/helm-charts/hermitcrab --version \u003cVERSION\u003e\n```\n\nAfter setting up Hermit Crab, you can make the [Terraform](https://developer.hashicorp.com/terraform/cli/config/config-file)/[OpenTofu](https://opentofu.org/docs/cli/config/config-file/) CLI Configuration File as below to use the mirroring service.\n\n```hcl\nprovider_installation {\n  network_mirror {\n   url = \"https://\u003cADDRESS\u003e/v1/providers/\"\n  }\n}\n```\n\n## Notice\n\nHermit Crab is not a [Terraform Registry](https://registry.terraform.io), although implementing these protocols is not difficult, there are many options that you can choose from, like [HashiCorp Terraform Enterprise](https://www.hashicorp.com/products/terraform/pricing/), [JFrog Artifactory](https://jfrog.com/help/r/jfrog-artifactory-documentation/terraform-registry), etc.\n\nHermit Crab cannot mirror [Terraform Module](https://developer.hashicorp.com/terraform/internals/module-registry-protocol), since obtaining Terraform modules is diverse, like [Git](https://developer.hashicorp.com/terraform/language/modules/sources#generic-git-repository), [HTTP URLs](https://developer.hashicorp.com/terraform/language/modules/sources#http-urls), [S3 Bucket](https://developer.hashicorp.com/terraform/language/modules/sources#gcs-bucket) and so on, it's hard to provide a unified way to mirror them.\n\nHermit Crab doesn't support rewriting the provider [hostname](https://developer.hashicorp.com/terraform/internals/provider-network-mirror-protocol#hostname), which is a rare case and may make template/module reusing difficult. One possible scenario is that there is a private Terraform Registry in your network, and you need to use the community template/module without any modification.\n\nHermit Crab automatically synchronizes the in-use versions per 30 minutes, if the information update occurs during sleep, we can manually trigger the synchronization by sending a `PUT` request to `/v1/providers/sync`.\n\nHermit Crab only performs a checksum verification on the downloaded archives. For archives that already exist in the implied or explicit directory, checksum verification is not performed.\n\nHermit Crab allows downloading the archives matching `^terraform-provider-(?P\u003ctype\u003e\\w+)[_-](?P\u003cversion\u003e[\\w|\\\\.]+)[_-](?P\u003cos\u003e[a-z]+)[_-](?P\u003carch\u003e[a-z0-9]+)([_-].*)?\\.zip$`, but it is recommended to follow the [Terraform Release Rules](https://developer.hashicorp.com/terraform/registry/providers/publishing#manually-preparing-a-release).\n\n# License\n\nCopyright (c) 2023 [Seal, Inc.](https://seal.io)\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at [LICENSE](./LICENSE) file for details.\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseal-io%2Fhermitcrab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseal-io%2Fhermitcrab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseal-io%2Fhermitcrab/lists"}