{"id":46824365,"url":"https://github.com/wellcomecollection/terraform-aws-vhs","last_synced_at":"2026-03-10T09:38:53.773Z","repository":{"id":47107994,"uuid":"239989808","full_name":"wellcomecollection/terraform-aws-vhs","owner":"wellcomecollection","description":"A Terraform module for a database that stores objects in S3 with pointers in DynamoDB","archived":false,"fork":false,"pushed_at":"2024-10-24T14:47:10.000Z","size":74,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-10-25T08:34:32.603Z","etag":null,"topics":["amazon-dynamodb","amazon-s3","database","terraform-modules"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wellcomecollection.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null}},"created_at":"2020-02-12T10:51:52.000Z","updated_at":"2021-09-13T08:25:08.000Z","dependencies_parsed_at":"2022-09-03T19:11:29.425Z","dependency_job_id":null,"html_url":"https://github.com/wellcomecollection/terraform-aws-vhs","commit_stats":null,"previous_names":[],"tags_count":24,"template":false,"template_full_name":"wellcomecollection/terraform-aws-template","purl":"pkg:github/wellcomecollection/terraform-aws-vhs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellcomecollection%2Fterraform-aws-vhs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellcomecollection%2Fterraform-aws-vhs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellcomecollection%2Fterraform-aws-vhs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellcomecollection%2Fterraform-aws-vhs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wellcomecollection","download_url":"https://codeload.github.com/wellcomecollection/terraform-aws-vhs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellcomecollection%2Fterraform-aws-vhs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30328864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T05:25:20.737Z","status":"ssl_error","status_checked_at":"2026-03-10T05:25:17.430Z","response_time":106,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-dynamodb","amazon-s3","database","terraform-modules"],"created_at":"2026-03-10T09:38:52.999Z","updated_at":"2026-03-10T09:38:53.758Z","avatar_url":"https://github.com/wellcomecollection.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# terraform-aws-vhs\n\n[![Build status](https://badge.buildkite.com/e003c26e84856dea7d37fc77c9f6e393d3007984bfb6c80f96.svg?main)](https://buildkite.com/wellcomecollection/terraform-module-terraform-aws-vhs)\n\nThis module creates the persistent storage for a *Versioned Hybrid Store (VHS)*.\n\nThe VHS is a combination of a DynamoDB table and an S3 bucket.\nWhen we store large values, we write:\n\n*   The value as an object in S3, and\n*   A pointer to the S3 object as an item in DynamoDB\n\nThis is a hybrid of two data stores, and the DynamoDB pointers record the version of the value -- thus, Versioned Hybrid Store.\n\n\u003cimg src=\"pointer.svg\"\u003e\n\nThis is [Amazon's recommended approach][aws_recommends] for storing large values in DynamoDB:\n\n\u003e If your application needs to store more data in an item than the DynamoDB size limit permits, you can try compressing one or more large attributes, or you can store them as an object in Amazon Simple Storage Service (Amazon S3) and store the Amazon S3 object identifier in your DynamoDB item.\n\nThis module creates the DynamoDB table and S3 bucket for this purpose.\nIt also creates some IAM policies that can be used to read or modify the VHS.\n\n[aws_recommends]: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-use-s3-too.html\n\n\n\n## Possible alternatives\n\n-   **Just DynamoDB?**\n    Items in DynamoDB have a [maximum size of 400KB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items), which is too small for our stores.\n\n-   **Just DynamoDB, but with compressed items?**\n    This only delays the issue -- we'd still hit the 400KB limit, but later.\n\n-   **Just S3?**\n    DynamoDB supports [conditional writes] to ensure updates are ordered correctly; S3 does not.\n    We need correct ordering, so S3 wouldn't work for us.\n\n-   **Another database technology?**\n    Plenty of databases have larger per-item limits and conditional writes (e.g. Cassandra, MySQL, Aurora), but we were already using DynamoDB and S3.\n    It's easier for us to use a few databases that we understand well than use a lot of databases we understand a little.\n\n[conditional writes]: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithItems.html#WorkingWithItems.ConditionalUpdate\n\n\n\n## Variants of VHS\n\nThere are two variants of the VHS:\n\n*   The `single-version-store` tracks a single version of each value.\n    The DynamoDB table only keeps the pointer to the newest object in S3.\n\n    (The table has a hash key `id`.)\n\n*   The `multi-version-store` tracks every version of a value.\n    The DynamoDB table keeps the pointer to every object in S3.\n\n    (The table has a hash key `id` and a range key `version`.)\n\n\n\n## Client libraries\n\nWe have Scala classes for interacting with a VHS in our [shared Scala libraries][scala_libs].\n\n[scala_libs]: https://github.com/wellcomecollection/scala-libs/tree/main/storage\n\n\n\n## Further reading\n\n-   [Creating a data store from S3 and DynamoDB][stacks]\n\n[stacks]: https://stacks.wellcomecollection.org/creating-a-data-store-from-s3-and-dynamodb-8bb9ecce8fc1\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwellcomecollection%2Fterraform-aws-vhs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwellcomecollection%2Fterraform-aws-vhs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwellcomecollection%2Fterraform-aws-vhs/lists"}