{"id":30682571,"url":"https://github.com/cloudandthings/terraform-aws-s3-inventory","last_synced_at":"2025-09-01T18:11:25.042Z","repository":{"id":311914902,"uuid":"1045482614","full_name":"cloudandthings/terraform-aws-s3-inventory","owner":"cloudandthings","description":"Create a locked-down S3 inventory destination bucket, glue table to query via Athena, and LakeFormation permissions. Supports multiple inventoried buckets.","archived":false,"fork":false,"pushed_at":"2025-08-27T14:04:04.000Z","size":25,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-27T20:53:22.687Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudandthings.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-27T08:40:40.000Z","updated_at":"2025-08-27T08:40:47.000Z","dependencies_parsed_at":"2025-08-27T20:53:26.844Z","dependency_job_id":"61935d27-2e70-4085-a69c-8f8ddd1825cf","html_url":"https://github.com/cloudandthings/terraform-aws-s3-inventory","commit_stats":null,"previous_names":["cloudandthings/terraform-aws-s3-inventory"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/cloudandthings/terraform-aws-s3-inventory","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudandthings%2Fterraform-aws-s3-inventory","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudandthings%2Fterraform-aws-s3-inventory/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudandthings%2Fterraform-aws-s3-inventory/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudandthings%2Fterraform-aws-s3-inventory/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudandthings","download_url":"https://codeload.github.com/cloudandthings/terraform-aws-s3-inventory/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudandthings%2Fterraform-aws-s3-inventory/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273168113,"owners_count":25057325,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-01T18:11:23.518Z","updated_at":"2025-09-01T18:11:25.028Z","avatar_url":"https://github.com/cloudandthings.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Terraform AWS S3 Inventory Module\n\nA comprehensive Terraform module for managing AWS S3 inventory configurations, including automated inventory reports, Glue catalog integration, and Athena querying capabilities.\n\n## Features\n\n- **S3 Inventory Destination Bucket**: Creates a dedicated S3 bucket for storing all inventory reports\n- **S3 Inventory Management**: Creates and configures S3 inventory reports for multiple source buckets\n- **Glue Catalog Integration**: Sets up Glue database and tables for querying inventory data\n- **Unified View**: Optional creation of a union view across all inventory tables for cross-bucket analysis\n- **Security \u0026 Compliance**: Configurable encryption, object locking, and IAM and LakeFormation permissions\n- **Lifecycle Management**: Automated lifecycle rules for inventory report retention\n\nMany features are optional and can be enabled/disabled as required.\n\n----\n\n## Quick Start\n\n```hcl\nmodule \"s3_inventory\" {\n  source  = \"cloudandthings/terraform-aws-s3-inventory/aws\"\n  version = \"~\u003e 1.0\"\n\n  # Required variables\n  inventory_bucket_name   = \"my-company-s3-inventory\"\n  inventory_database_name = \"s3_inventory_db\"\n\n  # Source buckets to inventory\n  source_bucket_names = [\n    \"my-app-data-bucket\",\n    \"my-logs-bucket\",\n    \"my-backup-bucket\"\n  ]\n\n  # Optional: Create a union view for cross-bucket queries\n  union_view_name = \"all_inventories_view\"\n\n  # Optional: Add LakeFormation permissions\n  # database_admin_principals = [...]\n  # database_read_principals = [...]\n\n}\n```\n\n\n----\n## Usage\n\nSee examples dropdown on Terraform Cloud, or [browse the GitHub repo](https://github.com/cloudandthings/terraform-aws-s3-inventory/tree/main/examples/).\n\n----\n\n## Querying Your Inventory Data\n\nOnce deployed, you can query your S3 inventory data using Amazon Athena.\n\nQuery a single bucket's inventory:\n\n```sql\nSELECT bucket, key, size, last_modified_date, storage_class\nFROM s3_inventory_db.my_app_data_bucket\nWHERE dt = '2024-08-29-00-00'\nORDER BY size DESC\nLIMIT 100;\n```\n\nQuery across all buckets (using the union view):\n\n```sql\nSELECT bucket, COUNT(*) as object_count, SUM(size) as total_size, AVG(size) as avg_size FROM s3_inventory_db.all_inventories_view\nWHERE dt \u003e= '2024-08-01-00-00'\nGROUP BY bucket\nORDER BY total_size DESC;\n```\n\n----\n\n## Important Considerations\n\n### Athena Partition Date Projection\n\nAs of 2025, Amazon Athena does not properly support dynamic range projection with the S3 inventory partitioning scheme. When using a dynamic range like `\"NOW-3MONTHS,NOW\"` with this module, the Glue tables will return zero rows.\n\nTo work around this limitation, this Terraform module defaults to using the beginning of the previous year as the start date. The year is calculated based on when the Terraform plan runs. For example, if today is `2025-08-25`, the date range will be defaulted to `\"2024-01-01-00-00,NOW\"`.\n\n**Important:** This approach causes Terraform state drift annually when the year changes.\n\n### Workaround\n\nTo avoid state drift, provide a fixed start date for partition projection, such as:\n\n`athena_projection_dt_range = \"2025-08-01-00-00,NOW\"`\n\nChoose your start date based on either:\n- Your specific requirements\n- The date when your S3 inventories were first deployed\n\n\n### Costs\n\n- S3 inventory reports are charged per million objects listed\n- Additional S3 storage costs for inventory files\n- Athena charges apply when querying the data\n- Consider lifecycle rules to manage long-term storage costs\n\n----\n\n## Contributing\n\nDirect contributions are welcome.\n\nSee [`CONTRIBUTING.md`](https://github.com/cloudandthings/terraform-aws-s3-inventory/tree/main/.github/CONTRIBUTING.md) for further information.\n\n\n----\n## License\n\nThis project is currently unlicensed. Please contact the maintaining team to add a license.\n\n----\n\n*This module was created from [terraform-aws-template](https://github.com/cloudandthings/terraform-aws-template)*\n\n\n\u003c!-- BEGIN_TF_DOCS --\u003e\n----\n## Documentation\n\n----\n### Inputs\n\n| Name | Description | Type | Default | Required |\n|------|-------------|------|---------|:--------:|\n| \u003ca name=\"input_apply_default_inventory_lifecyle_rules\"\u003e\u003c/a\u003e [apply\\_default\\_inventory\\_lifecyle\\_rules](#input\\_apply\\_default\\_inventory\\_lifecyle\\_rules) | Whether to attach default lifecycle rules to the S3 inventory bucket | `bool` | `true` | no |\n| \u003ca name=\"input_athena_projection_dt_range\"\u003e\u003c/a\u003e [athena\\_projection\\_dt\\_range](#input\\_athena\\_projection\\_dt\\_range) | Date range for Athena partition projection (format: START\\_DATE,END\\_DATE). If null then a value will be generated, see README for more information. | `string` | `null` | no |\n| \u003ca name=\"input_attach_default_inventory_bucket_policy\"\u003e\u003c/a\u003e [attach\\_default\\_inventory\\_bucket\\_policy](#input\\_attach\\_default\\_inventory\\_bucket\\_policy) | Whether to attach a default bucket policy to the S3 inventory bucket | `bool` | `true` | no |\n| \u003ca name=\"input_create_inventory_bucket\"\u003e\u003c/a\u003e [create\\_inventory\\_bucket](#input\\_create\\_inventory\\_bucket) | Whether to create the S3 inventory bucket | `bool` | `true` | no |\n| \u003ca name=\"input_create_inventory_database\"\u003e\u003c/a\u003e [create\\_inventory\\_database](#input\\_create\\_inventory\\_database) | Whether to create the Glue database for S3 inventory | `bool` | `true` | no |\n| \u003ca name=\"input_database_admin_principals\"\u003e\u003c/a\u003e [database\\_admin\\_principals](#input\\_database\\_admin\\_principals) | List of principal ARNs that will be allowed to manage (create, update, delete) the Glue database and its tables | `list(string)` | `[]` | no |\n| \u003ca name=\"input_database_read_principals\"\u003e\u003c/a\u003e [database\\_read\\_principals](#input\\_database\\_read\\_principals) | List of principal ARNs that will be allowed to read from the Glue database (query tables, describe metadata) | `list(string)` | `[]` | no |\n| \u003ca name=\"input_enable_bucket_inventory_configs\"\u003e\u003c/a\u003e [enable\\_bucket\\_inventory\\_configs](#input\\_enable\\_bucket\\_inventory\\_configs) | Whether to create S3 inventory configurations for the specified buckets | `bool` | `true` | no |\n| \u003ca name=\"input_inventory_bucket_encryption_config\"\u003e\u003c/a\u003e [inventory\\_bucket\\_encryption\\_config](#input\\_inventory\\_bucket\\_encryption\\_config) | Map containing server-side encryption configuration for the S3 inventory bucket. | `any` | `{}` | no |\n| \u003ca name=\"input_inventory_bucket_lifecycle_rules\"\u003e\u003c/a\u003e [inventory\\_bucket\\_lifecycle\\_rules](#input\\_inventory\\_bucket\\_lifecycle\\_rules) | List of lifecycle rules to apply to the S3 inventory bucket | `any` | `[]` | no |\n| \u003ca name=\"input_inventory_bucket_name\"\u003e\u003c/a\u003e [inventory\\_bucket\\_name](#input\\_inventory\\_bucket\\_name) | Name of the S3 inventory bucket | `string` | n/a | yes |\n| \u003ca name=\"input_inventory_bucket_object_lock_mode\"\u003e\u003c/a\u003e [inventory\\_bucket\\_object\\_lock\\_mode](#input\\_inventory\\_bucket\\_object\\_lock\\_mode) | Object Lock mode for the S3 inventory bucket (GOVERNANCE or COMPLIANCE) | `string` | `\"GOVERNANCE\"` | no |\n| \u003ca name=\"input_inventory_bucket_object_lock_retention_days\"\u003e\u003c/a\u003e [inventory\\_bucket\\_object\\_lock\\_retention\\_days](#input\\_inventory\\_bucket\\_object\\_lock\\_retention\\_days) | Number of days to retain objects with Object Lock (null to disable Object Lock) | `number` | `null` | no |\n| \u003ca name=\"input_inventory_config_encryption\"\u003e\u003c/a\u003e [inventory\\_config\\_encryption](#input\\_inventory\\_config\\_encryption) | Map containing encryption settings for the S3 inventory configuration. | `any` | `{}` | no |\n| \u003ca name=\"input_inventory_config_frequency\"\u003e\u003c/a\u003e [inventory\\_config\\_frequency](#input\\_inventory\\_config\\_frequency) | Frequency of the S3 inventory report generation | `string` | `\"Daily\"` | no |\n| \u003ca name=\"input_inventory_config_name\"\u003e\u003c/a\u003e [inventory\\_config\\_name](#input\\_inventory\\_config\\_name) | Name identifier for the S3 inventory configuration | `string` | `\"daily\"` | no |\n| \u003ca name=\"input_inventory_config_object_versions\"\u003e\u003c/a\u003e [inventory\\_config\\_object\\_versions](#input\\_inventory\\_config\\_object\\_versions) | Which object versions to include in the inventory report | `string` | `\"All\"` | no |\n| \u003ca name=\"input_inventory_database_name\"\u003e\u003c/a\u003e [inventory\\_database\\_name](#input\\_inventory\\_database\\_name) | Name of the S3 inventory Glue database | `string` | n/a | yes |\n| \u003ca name=\"input_inventory_optional_fields\"\u003e\u003c/a\u003e [inventory\\_optional\\_fields](#input\\_inventory\\_optional\\_fields) | List of optional fields to include in the S3 inventory report | `list(string)` | \u003cpre\u003e[\u003cbr/\u003e  \"Size\",\u003cbr/\u003e  \"LastModifiedDate\",\u003cbr/\u003e  \"IsMultipartUploaded\",\u003cbr/\u003e  \"ReplicationStatus\",\u003cbr/\u003e  \"EncryptionStatus\",\u003cbr/\u003e  \"BucketKeyStatus\",\u003cbr/\u003e  \"StorageClass\",\u003cbr/\u003e  \"IntelligentTieringAccessTier\",\u003cbr/\u003e  \"ETag\",\u003cbr/\u003e  \"ChecksumAlgorithm\",\u003cbr/\u003e  \"ObjectLockRetainUntilDate\",\u003cbr/\u003e  \"ObjectLockMode\",\u003cbr/\u003e  \"ObjectLockLegalHoldStatus\",\u003cbr/\u003e  \"ObjectAccessControlList\",\u003cbr/\u003e  \"ObjectOwner\"\u003cbr/\u003e]\u003c/pre\u003e | no |\n| \u003ca name=\"input_source_bucket_names\"\u003e\u003c/a\u003e [source\\_bucket\\_names](#input\\_source\\_bucket\\_names) | List of S3 bucket names to create inventory reports for | `list(string)` | `[]` | no |\n| \u003ca name=\"input_union_view_name\"\u003e\u003c/a\u003e [union\\_view\\_name](#input\\_union\\_view\\_name) | Name for the Athena view over S3 inventory data from all the source buckets | `string` | `null` | no |\n\n----\n### Modules\n\n| Name | Source | Version |\n|------|--------|---------|\n| \u003ca name=\"module_inventory_bucket\"\u003e\u003c/a\u003e [inventory\\_bucket](#module\\_inventory\\_bucket) | terraform-aws-modules/s3-bucket/aws | 4.6.0 |\n\n----\n### Outputs\n\n| Name | Description |\n|------|-------------|\n| \u003ca name=\"output_athena_projection_dt_range\"\u003e\u003c/a\u003e [athena\\_projection\\_dt\\_range](#output\\_athena\\_projection\\_dt\\_range) | The value used for projection.dt.range on the Glue table |\n\n----\n### Providers\n\n| Name | Version |\n|------|---------|\n| \u003ca name=\"provider_aws\"\u003e\u003c/a\u003e [aws](#provider\\_aws) | \u003e= 5, \u003c 7 |\n\n----\n### Requirements\n\n| Name | Version |\n|------|---------|\n| \u003ca name=\"requirement_terraform\"\u003e\u003c/a\u003e [terraform](#requirement\\_terraform) | ~\u003e 1.0 |\n| \u003ca name=\"requirement_aws\"\u003e\u003c/a\u003e [aws](#requirement\\_aws) | \u003e= 5, \u003c 7 |\n\n----\n### Resources\n\n| Name | Type |\n|------|------|\n| [aws_glue_catalog_database.s3_inventory](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/glue_catalog_database) | resource |\n| [aws_glue_catalog_table.s3_inventory](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/glue_catalog_table) | resource |\n| [aws_glue_catalog_table.view](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/glue_catalog_table) | resource |\n| [aws_lakeformation_permissions.inventory_database_admin](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lakeformation_permissions) | resource |\n| [aws_lakeformation_permissions.inventory_database_read](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lakeformation_permissions) | resource |\n| [aws_lakeformation_permissions.inventory_tables_admin](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lakeformation_permissions) | resource |\n| [aws_lakeformation_permissions.inventory_tables_read](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lakeformation_permissions) | resource |\n| [aws_s3_bucket_inventory.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_inventory) | resource |\n| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |\n| [aws_iam_policy_document.inventory_bucket_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |\n| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |\n\n----\n\u003c!-- END_TF_DOCS --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudandthings%2Fterraform-aws-s3-inventory","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudandthings%2Fterraform-aws-s3-inventory","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudandthings%2Fterraform-aws-s3-inventory/lists"}