{"id":13413601,"url":"https://github.com/hashicorp/go-getter","last_synced_at":"2025-05-14T09:06:09.199Z","repository":{"id":1763751,"uuid":"44139350","full_name":"hashicorp/go-getter","owner":"hashicorp","description":"Package for downloading things from a string URL using a variety of protocols.","archived":false,"fork":false,"pushed_at":"2025-05-06T06:23:39.000Z","size":2889,"stargazers_count":1700,"open_issues_count":178,"forks_count":250,"subscribers_count":288,"default_branch":"main","last_synced_at":"2025-05-07T08:03:28.040Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hashicorp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-10-12T23:17:07.000Z","updated_at":"2025-05-05T22:02:17.000Z","dependencies_parsed_at":"2023-07-05T20:02:14.778Z","dependency_job_id":"e1d47be1-5438-4dfa-a84d-abdb3bb54997","html_url":"https://github.com/hashicorp/go-getter","commit_stats":{"total_commits":438,"total_committers":78,"mean_commits":5.615384615384615,"dds":0.7351598173515982,"last_synced_commit":"842d6c379e5e70d23905b8f6b5a25a80290acb66"},"previous_names":[],"tags_count":63,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fgo-getter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fgo-getter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fgo-getter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fgo-getter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hashicorp","download_url":"https://codeload.github.com/hashicorp/go-getter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254110373,"owners_count":22016391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:01:44.201Z","updated_at":"2025-05-14T09:06:09.125Z","avatar_url":"https://github.com/hashicorp.png","language":"Go","readme":"# go-getter\n\n[![CircleCI](https://circleci.com/gh/hashicorp/go-getter/tree/main.svg?style=svg)][circleci]\n[![Go Documentation](http://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)][godocs]\n\n[circleci]: https://circleci.com/gh/hashicorp/go-getter/tree/main\n[godocs]: http://godoc.org/github.com/hashicorp/go-getter\n\ngo-getter is a library for Go (golang) for downloading files or directories\nfrom various sources using a URL as the primary form of input.\n\nThe power of this library is being flexible in being able to download\nfrom a number of different sources (file paths, Git, HTTP, Mercurial, etc.)\nusing a single string as input. This removes the burden of knowing how to\ndownload from a variety of sources from the implementer.\n\nThe concept of a _detector_ automatically turns invalid URLs into proper\nURLs. For example: \"github.com/hashicorp/go-getter\" would turn into a\nGit URL. Or \"./foo\" would turn into a file URL. These are extensible.\n\nThis library is used by [Terraform](https://terraform.io) for\ndownloading modules and [Nomad](https://nomadproject.io) for downloading\nbinaries.\n\n## Installation and Usage\n\nPackage documentation can be found on\n[GoDoc](http://godoc.org/github.com/hashicorp/go-getter).\n\nInstallation can be done with a normal `go get`:\n\n```\n$ go get github.com/hashicorp/go-getter\n```\n\ngo-getter also has a command you can use to test URL strings:\n\n```\n$ go install github.com/hashicorp/go-getter/cmd/go-getter\n...\n\n$ go-getter github.com/foo/bar ./foo\n...\n```\n\nThe command is useful for verifying URL structures.\n\n## Security\nFetching resources from user-supplied URLs is an inherently dangerous operation and may\nleave your application vulnerable to [server side request forgery](https://owasp.org/www-community/attacks/Server_Side_Request_Forgery),\n[path traversal](https://owasp.org/www-community/attacks/Path_Traversal), [denial of service](https://owasp.org/www-community/attacks/Denial_of_Service)\nor other security flaws.\n\ngo-getter contains mitigations for some of these security issues, but should still be used with\ncaution in security-critical contexts. See the available [security options](#Security-Options) that\ncan be configured to mitigate some of these risks.\n\ngo-getter may return values that contain caller-provided query parameters that can contain sensitive data.\nContext around what parameters are and are not sensitive is known only by the caller of go-getter, and specific to each use case.\nWe recommend the caller ensure that go-getter's return values (e.g., error messages) are properly handled and sanitized to ensure\nsensitive data is not persisted to logs.\n## URL Format\n\ngo-getter uses a single string URL as input to download from a variety of\nprotocols. go-getter has various \"tricks\" with this URL to do certain things.\nThis section documents the URL format.\n\n### Supported Protocols and Detectors\n\n**Protocols** are used to download files/directories using a specific\nmechanism. Example protocols are Git and HTTP.\n\n**Detectors** are used to transform a valid or invalid URL into another\nURL if it matches a certain pattern. Example: \"github.com/user/repo\" is\nautomatically transformed into a fully valid Git URL. This allows go-getter\nto be very user friendly.\n\ngo-getter out of the box supports the following protocols. Additional protocols\ncan be augmented at runtime by implementing the `Getter` interface.\n\n  * Local files\n  * Git\n  * Mercurial\n  * HTTP\n  * Amazon S3\n  * Google GCP\n\nIn addition to the above protocols, go-getter has what are called \"detectors.\"\nThese take a URL and attempt to automatically choose the best protocol for\nit, which might involve even changing the protocol. The following detection\nis built-in by default:\n\n  * File paths such as \"./foo\" are automatically changed to absolute\n    file URLs.\n  * GitHub URLs, such as \"github.com/mitchellh/vagrant\" are automatically\n    changed to Git protocol over HTTP.\n  * GitLab URLs, such as \"gitlab.com/inkscape/inkscape\" are automatically \n    changed to Git protocol over HTTP.\n  * BitBucket URLs, such as \"bitbucket.org/mitchellh/vagrant\" are automatically\n    changed to a Git or mercurial protocol using the BitBucket API.\n\n### Forced Protocol\n\nIn some cases, the protocol to use is ambiguous depending on the source\nURL. For example, \"http://github.com/mitchellh/vagrant.git\" could reference\nan HTTP URL or a Git URL. Forced protocol syntax is used to disambiguate this\nURL.\n\nForced protocol can be done by prefixing the URL with the protocol followed\nby double colons. For example: `git::http://github.com/mitchellh/vagrant.git`\nwould download the given HTTP URL using the Git protocol.\n\nForced protocols will also override any detectors.\n\nIn the absence of a forced protocol, detectors may be run on the URL, transforming\nthe protocol anyways. The above example would've used the Git protocol either\nway since the Git detector would've detected it was a GitHub URL.\n\n### Protocol-Specific Options\n\nEach protocol can support protocol-specific options to configure that\nprotocol. For example, the `git` protocol supports specifying a `ref`\nquery parameter that tells it what ref to checkout for that Git\nrepository.\n\nThe options are specified as query parameters on the URL (or URL-like string)\ngiven to go-getter. Using the Git example above, the URL below is a valid\ninput to go-getter:\n\n    github.com/hashicorp/go-getter?ref=abcd1234\n\nThe protocol-specific options are documented below the URL format\nsection. But because they are part of the URL, we point it out here so\nyou know they exist.\n\n### Subdirectories\n\nIf you want to download only a specific subdirectory from a downloaded\ndirectory, you can specify a subdirectory after a double-slash `//`.\ngo-getter will first download the URL specified _before_ the double-slash\n(as if you didn't specify a double-slash), but will then copy the\npath after the double slash into the target directory.\n\nFor example, if you're downloading this GitHub repository, but you only\nwant to download the `testdata` directory, you can do the following:\n\n```\nhttps://github.com/hashicorp/go-getter.git//testdata\n```\n\nIf you downloaded this to the `/tmp` directory, then the file\n`/tmp/archive.gz` would exist. Notice that this file is in the `testdata`\ndirectory in this repository, but because we specified a subdirectory,\ngo-getter automatically copied only that directory contents.\n\nSubdirectory paths may also use filesystem glob patterns. The path must\nmatch _exactly one_ entry or go-getter will return an error.\nThis is useful if you're not sure the exact directory name but it follows\na predictable naming structure.\n\nFor example, the following URL would also work:\n\n```\nhttps://github.com/hashicorp/go-getter.git//test-*\n```\n\n### Checksumming\n\nFor file downloads of any protocol, go-getter can automatically verify\na checksum for you. Note that checksumming only works for downloading files,\nnot directories, but checksumming will work for any protocol.\n\nTo checksum a file, append a `checksum` query parameter to the URL. go-getter\nwill parse out this query parameter automatically and use it to verify the\nchecksum. The parameter value can be in the format of `type:value` or just\n`value`, where type is \"md5\", \"sha1\", \"sha256\", \"sha512\" or \"file\" . The\n\"value\" should be the actual checksum value or download URL for \"file\". When\n`type` part is omitted, type will be guessed based on the length of the\nchecksum string. Examples:\n\n```\n./foo.txt?checksum=md5:b7d96c89d09d9e204f5fedc4d5d55b21\n```\n\n```\n./foo.txt?checksum=b7d96c89d09d9e204f5fedc4d5d55b21\n```\n\n```\n./foo.txt?checksum=file:./foo.txt.sha256sum\n```\n \nWhen checksumming from a file - ex: with `checksum=file:url` - go-getter will\nget the file linked in the URL after `file:` using the same configuration. For\nexample, in `file:http://releases.ubuntu.com/cosmic/MD5SUMS` go-getter will\ndownload a checksum file under the aforementioned url using the http protocol.\nAll protocols supported by go-getter can be used. The checksum file will be\ndownloaded in a temporary file then parsed. The destination of the temporary\nfile can be changed by setting system specific environment variables: `TMPDIR`\nfor unix; `TMP`, `TEMP` or `USERPROFILE` on windows. Read godoc of\n[os.TempDir](https://golang.org/pkg/os/#TempDir) for more information on the\ntemporary directory selection. Content of files are expected to be BSD or GNU\nstyle. Once go-getter is done with the checksum file; it is deleted.\n\nThe checksum query parameter is never sent to the backend protocol\nimplementation. It is used at a higher level by go-getter itself.\n\nIf the destination file exists and the checksums match: download\nwill be skipped.\n\n### Unarchiving\n\ngo-getter will automatically unarchive files into a file or directory\nbased on the extension of the file being requested (over any protocol).\nThis works for both file and directory downloads.\n\ngo-getter looks for an `archive` query parameter to specify the format of\nthe archive. If this isn't specified, go-getter will use the extension of\nthe path to see if it appears archived. Unarchiving can be explicitly\ndisabled by setting the `archive` query parameter to `false`.\n\nThe following archive formats are supported:\n\n  * `tar.gz` and `tgz`\n  * `tar.bz2` and `tbz2`\n  * `tar.xz` and `txz`\n  * `zip`\n  * `gz`\n  * `bz2`\n  * `xz`\n\nFor example, an example URL is shown below:\n\n```\n./foo.zip\n```\n\nThis will automatically be inferred to be a ZIP file and will be extracted.\nYou can also be explicit about the archive type:\n\n```\n./some/other/path?archive=zip\n```\n\nAnd finally, you can disable archiving completely:\n\n```\n./some/path?archive=false\n```\n\nYou can combine unarchiving with the other features of go-getter such\nas checksumming. The special `archive` query parameter will be removed\nfrom the URL before going to the final protocol downloader.\n\n## Protocol-Specific Options\n\nThis section documents the protocol-specific options that can be specified for\ngo-getter. These options should be appended to the input as normal query\nparameters ([HTTP headers](#headers) are an exception to this, however).\nDepending on the usage of go-getter, applications may provide alternate ways of\ninputting options. For example, [Nomad](https://www.nomadproject.io) provides a\nnice options block for specifying options rather than in the URL.\n\n## General (All Protocols)\n\nThe options below are available to all protocols:\n\n  * `archive` - The archive format to use to unarchive this file, or \"\" (empty\n    string) to disable unarchiving. For more details, see the complete section\n    on archive support above.\n\n  * `checksum` - Checksum to verify the downloaded file or archive. See\n    the entire section on checksumming above for format and more details.\n\n  * `filename` - When in file download mode, allows specifying the name of the\n    downloaded file on disk. Has no effect in directory mode.\n\n### Local Files (`file`)\n\nNone\n\n### Git (`git`)\n\n  * `ref` - The Git ref to checkout. This is a ref, so it can point to\n    a commit SHA, a branch name, etc. If it is a named ref such as a branch\n    name, go-getter will update it to the latest on each get.\n\n  * `sshkey` - An SSH private key to use during clones. The provided key must\n    be a base64-encoded string. For example, to generate a suitable `sshkey`\n    from a private key file on disk, you would run `base64 -w0 \u003cfile\u003e`.\n\n    **Note**: Git 2.3+ is required to use this feature.\n  \n  * `depth` - The Git clone depth. The provided number specifies the last `n`\n    revisions to clone from the repository.\n\n\nThe `git` getter accepts both URL-style SSH addresses like\n`git::ssh://git@example.com/foo/bar`, and \"scp-style\" addresses like\n`git::git@example.com/foo/bar`. In the latter case, omitting the `git::`\nforce prefix is allowed if the username prefix is exactly `git@`.\n\nThe \"scp-style\" addresses _cannot_ be used in conjunction with the `ssh://`\nscheme prefix, because in that case the colon is used to mark an optional\nport number to connect on, rather than to delimit the path from the host.\n\n### Mercurial (`hg`)\n\n  * `rev` - The Mercurial revision to checkout.\n\n### HTTP (`http`)\n\n#### Basic Authentication\n\nTo use HTTP basic authentication with go-getter, simply prepend `username:password@` to the\nhostname in the URL such as `https://Aladdin:OpenSesame@www.example.com/index.html`. All special\ncharacters, including the username and password, must be URL encoded.\n\n#### Headers\n\nOptional request headers can be added by supplying them in a custom\n[`HttpGetter`](https://godoc.org/github.com/hashicorp/go-getter#HttpGetter)\n(_not_ as query parameters like most other options). These headers will be sent\nout on every request the getter in question makes.\n\n### S3 (`s3`)\n\nS3 takes various access configurations in the URL. Note that it will also\nread these from standard AWS environment variables if they're set. S3 compliant servers like Minio\nare also supported. If the query parameters are present, these take priority.\n\n  * `aws_access_key_id` - AWS access key.\n  * `aws_access_key_secret` - AWS access key secret.\n  * `aws_access_token` - AWS access token if this is being used.\n  * `aws_profile` - Use this profile from local ~/.aws/ config. Takes priority over the other three.\n\n#### Using IAM Instance Profiles with S3\n\nIf you use go-getter and want to use an EC2 IAM Instance Profile to avoid\nusing credentials, then just omit these and the profile, if available will\nbe used automatically.\n\n### Using S3 with Minio\n If you use go-gitter for Minio support, you must consider the following:\n\n  * `aws_access_key_id` (required) - Minio access key.\n  * `aws_access_key_secret` (required) - Minio access key secret.\n  * `region` (optional - defaults to us-east-1) - Region identifier to use.\n  * `version` (optional - defaults to Minio default) - Configuration file format.\n\n#### S3 Bucket Examples\n\nS3 has several addressing schemes used to reference your bucket. These are\nlisted here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html\n\nSome examples for these addressing schemes:\n- s3::https://s3.amazonaws.com/bucket/foo\n- s3::https://s3-eu-west-1.amazonaws.com/bucket/foo\n- bucket.s3.amazonaws.com/foo\n- bucket.s3-eu-west-1.amazonaws.com/foo/bar\n- \"s3::http://127.0.0.1:9000/test-bucket/hello.txt?aws_access_key_id=KEYID\u0026aws_access_key_secret=SECRETKEY\u0026region=us-east-2\"\n\n### GCS (`gcs`)\n\n#### GCS Authentication\n\nIn order to access to GCS, authentication credentials should be provided. More information can be found [here](https://cloud.google.com/docs/authentication/getting-started)\n\n#### GCS Bucket Examples\n\n- gcs::https://www.googleapis.com/storage/v1/bucket\n- gcs::https://www.googleapis.com/storage/v1/bucket/foo.zip\n- www.googleapis.com/storage/v1/bucket/foo\n\n#### GCS Testing\n\nThe tests for `get_gcs.go` require you to have GCP credentials set in your environment.  These credentials can have any level of permissions to any project, they just need to exist.  This means setting `GOOGLE_APPLICATION_CREDENTIALS=\"~/path/to/credentials.json\"` or `GOOGLE_CREDENTIALS=\"{stringified-credentials-json}\"`.  Due to this configuration, `get_gcs_test.go` will fail for external contributors in CircleCI.\n\n\n### Security Options\n\n**Disable Symlinks**\n\nIn your getter client config, we recommend using the `DisableSymlinks` option,\nwhich prevents writing through or copying from symlinks (which may point outside the directory).\n\n```go\nclient := getter.Client{\n    // This will prevent copying or writing files through symlinks\n    DisableSymlinks: true,\n}\n```\n\n**Disable or Limit `X-Terraform-Get`**\n\nGo-Getter supports arbitrary redirects via the `X-Terraform-Get` header. This functionality\nexists to support [Terraform use cases](https://www.terraform.io/language/modules/sources#http-urls),\nbut is likely not needed in most applications.\n\nFor code that uses the `HttpGetter`, add the following configuration options:\n\n```go\nvar httpGetter = \u0026getter.HttpGetter{\n    // Most clients should disable X-Terraform-Get\n    // See the note below\n    XTerraformGetDisabled: true,\n    // Your software probably doesn’t rely on X-Terraform-Get, but\n    // if it does, you should set the above field to false, plus\n    // set XTerraformGet Limit to prevent endless redirects\n    // XTerraformGetLimit: 10,\n}\n```\n\n**Enforce Timeouts**\n\nThe `HttpGetter` supports timeouts and other resource-constraining configuration options. The `GitGetter` and `HgGetter`\nonly support timeouts.\n\nConfiguration for the `HttpGetter`:\n\n```go\nvar httpGetter = \u0026getter.HttpGetter{\n    // Disable pre-fetch HEAD requests\n    DoNotCheckHeadFirst: true,\n    \n    // As an alternative to the above setting, you can\n    // set a reasonable timeout for HEAD requests\n    // HeadFirstTimeout: 10 * time.Second,\n\n    // Read timeout for HTTP operations\n    ReadTimeout: 30 * time.Second,\n\n    // Set the maximum number of bytes\n    // that can be read by the getter\n    MaxBytes: 500000000, // 500 MB\n}\n```\n\nFor code that uses the `GitGetter` or `HgGetter`, set the `Timeout` option:\n```go\nvar gitGetter = \u0026getter.GitGetter{\n    // Set a reasonable timeout for git operations\n    Timeout: 5 * time.Minute,\n}\n```\n\n```go\nvar hgGetter = \u0026getter.HgGetter{\n    // Set a reasonable timeout for hg operations\n    Timeout: 5 * time.Minute,\n}\n```\n\n","funding_links":[],"categories":["Go","开源类库","Programming","Middlewares \u0026 framework add-ons","Open source library","Networking","\u003cspan id=\"网络-networking\"\u003e网络 Networking\u003c/span\u003e","网络相关库","网络","Relational Databases"],"sub_categories":["网络","Golang","The Internet","Advanced Console UIs","Uncategorized","Transliteration","\u003cspan id=\"高级控制台用户界面-advanced-console-uis\"\u003e高级控制台用户界面 Advanced Console UIs\u003c/span\u003e","Strings","暂未分类这些库被放在这里是因为其他类别似乎都不适合。","音译","交流","暂未分类"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhashicorp%2Fgo-getter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhashicorp%2Fgo-getter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhashicorp%2Fgo-getter/lists"}