{"id":24503625,"url":"https://github.com/plus3it/terraform-aws-wrangler","last_synced_at":"2025-06-12T19:32:19.021Z","repository":{"id":32207188,"uuid":"125841175","full_name":"plus3it/terraform-aws-wrangler","owner":"plus3it","description":"Terraform module to collect files from various sources and manage them in an S3 bucket","archived":false,"fork":false,"pushed_at":"2024-12-31T23:50:11.000Z","size":230,"stargazers_count":0,"open_issues_count":1,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-01-21T23:17:46.011Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/plus3it.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-19T10:32:04.000Z","updated_at":"2024-08-19T21:33:20.000Z","dependencies_parsed_at":"2024-08-06T23:45:33.606Z","dependency_job_id":"c8ae1671-694d-420c-8176-5537b3134e12","html_url":"https://github.com/plus3it/terraform-aws-wrangler","commit_stats":null,"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plus3it%2Fterraform-aws-wrangler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plus3it%2Fterraform-aws-wrangler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plus3it%2Fterraform-aws-wrangler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plus3it%2Fterraform-aws-wrangler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/plus3it","download_url":"https://codeload.github.com/plus3it/terraform-aws-wrangler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243703850,"owners_count":20334056,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-21T23:17:52.959Z","updated_at":"2025-06-12T19:32:19.006Z","avatar_url":"https://github.com/plus3it.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![pullreminders](https://pullreminders.com/badge.svg)](https://pullreminders.com?ref=badge)\n\n# terraform-aws-wrangler\n\nTerraform module to collect files from various sources and manage them in an S3\nbucket. In order to support arbitrary file-types, this module uses [`terraform-external-file-cache`](https://registry.terraform.io/modules/plus3it/file-cache/external)\nto create a local cache of the files. The files are then managed in the S3\nbucket using the terraform resource `aws_s3_object`. A sha512 hash of\nevery file is also published to the bucket.\n\n## Use Cases\n\nThis module supports a couple use cases:\n\n1.  Retrieve files from source URIs and store them in an S3 bucket.\n2.  Copy files from one S3 bucket to another.\n\n### Retrieve files and store them in an S3 bucket\n\nThese variables are used to retrieve files and store them in an S3 bucket:\n\n-   `uri_map` - Map of `URI = S3 Path`. Each `URI` will be retrieved and the\n    file will be saved to `S3 Path`. The filename is preserved. If the map is\n    empty (the default), no files are retrieved. Here is an example of the\n    structure:\n\n    ```hcl\n    uri_map = {\n      # salt for windows\n      \"https://packages.broadcom.com/artifactory/saltproject-generic/windows/3006.10/Salt-Minion-3006.10-Py3-AMD64-Setup.exe\" = \"saltstack/salt/windows/\"\n\n      # python for windows\n      \"https://www.python.org/ftp/python/3.6.8/python-3.6.8-amd64.exe\" = \"python/python/\"\n    }\n    ```\n\n-   `bucket_name` - Name of the S3 bucket where files will be stored.\n-   `prefix` - S3 prefix prepended to all S3 key paths when the files are put\n    in the bucket.\n\n### Copy files from one bucket to another\n\nThis is accomplished by getting a list of the s3 objects in the source bucket,\nand constructing the `uri_map`. This list can be provided using the data source\n`aws_s3_objects`, but when doing so it is recommended to generate that\nlist in a separate state and output the value. This is because the output of a\ndata source or resource **cannot** be used in the `for_each` statement of a\nresource (without encountering chicken/egg problems).\n\nSee the [`s3_sync` test](tests/s3_sync) for an example.\n\n## Testing\n\nManual testing:\n\n```\n# Replace \"xxx\" with an actual AWS profile, then execute the integration tests.\nexport AWS_PROFILE=xxx \nmake terraform/pytest PYTEST_ARGS=\"-v --nomock\"\n```\n\nFor automated testing, PYTEST_ARGS is optional and no profile is needed:\n\n```\nmake mockstack/up\nmake terraform/pytest PYTEST_ARGS=\"-v\"\nmake mockstack/clean\n```\n\n\u003c!-- BEGIN TFDOCS --\u003e\n## Requirements\n\n| Name | Version |\n|------|---------|\n| \u003ca name=\"requirement_terraform\"\u003e\u003c/a\u003e [terraform](#requirement\\_terraform) | \u003e= 1.0 |\n| \u003ca name=\"requirement_aws\"\u003e\u003c/a\u003e [aws](#requirement\\_aws) | \u003e= 4.0.0 |\n\n## Providers\n\n| Name | Version |\n|------|---------|\n| \u003ca name=\"provider_aws\"\u003e\u003c/a\u003e [aws](#provider\\_aws) | \u003e= 4.0.0 |\n\n## Resources\n\n| Name | Type |\n|------|------|\n\n## Inputs\n\n| Name | Description | Type | Default | Required |\n|------|-------------|------|---------|:--------:|\n| \u003ca name=\"input_bucket_name\"\u003e\u003c/a\u003e [bucket\\_name](#input\\_bucket\\_name) | Name of the S3 bucket where file artifacts are to be stored | `string` | n/a | yes |\n| \u003ca name=\"input_create_hashes\"\u003e\u003c/a\u003e [create\\_hashes](#input\\_create\\_hashes) | Create and host sha512 hashes of each file in the `uri_map` | `bool` | `true` | no |\n| \u003ca name=\"input_prefix\"\u003e\u003c/a\u003e [prefix](#input\\_prefix) | S3 key prefix to prepend to each object | `string` | `\"\"` | no |\n| \u003ca name=\"input_python_cmd\"\u003e\u003c/a\u003e [python\\_cmd](#input\\_python\\_cmd) | Command to use with the filecache module when executing python external resources | `list(any)` | \u003cpre\u003e[\u003cbr/\u003e  \"python\"\u003cbr/\u003e]\u003c/pre\u003e | no |\n| \u003ca name=\"input_s3_endpoint_url\"\u003e\u003c/a\u003e [s3\\_endpoint\\_url](#input\\_s3\\_endpoint\\_url) | S3 API endpoint for non-AWS hosts; format: https://hostname:port | `string` | `null` | no |\n| \u003ca name=\"input_uri_map\"\u003e\u003c/a\u003e [uri\\_map](#input\\_uri\\_map) | Map of URIs to retrieve and the S3 key path at which to store the file | `map(string)` | `{}` | no |\n\n## Outputs\n\n| Name | Description |\n|------|-------------|\n| \u003ca name=\"output_files\"\u003e\u003c/a\u003e [files](#output\\_files) | Map of file keys =\u003e source\\_hash |\n| \u003ca name=\"output_hashes\"\u003e\u003c/a\u003e [hashes](#output\\_hashes) | Map of hash keys =\u003e source\\_hash |\n\n\u003c!-- END TFDOCS --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplus3it%2Fterraform-aws-wrangler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fplus3it%2Fterraform-aws-wrangler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplus3it%2Fterraform-aws-wrangler/lists"}