{"id":21710863,"url":"https://github.com/jareechang/dynamodb-data-modelling-target-stores","last_synced_at":"2026-04-08T11:32:34.986Z","repository":{"id":87963447,"uuid":"289613736","full_name":"Jareechang/dynamodb-data-modelling-target-stores","owner":"Jareechang","description":"Data modelling with DynamoDB (Target stores)","archived":false,"fork":false,"pushed_at":"2020-08-25T03:38:18.000Z","size":218,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-03-04T04:13:39.870Z","etag":null,"topics":["aws","datamodeling","docker","docker-compose","dynamodb","keyvaluestore","local-dynamodb","nodejs","python","terraform","typescript"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jareechang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-08-23T04:14:14.000Z","updated_at":"2020-08-25T03:38:20.000Z","dependencies_parsed_at":"2023-05-22T03:45:33.530Z","dependency_job_id":null,"html_url":"https://github.com/Jareechang/dynamodb-data-modelling-target-stores","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Jareechang/dynamodb-data-modelling-target-stores","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jareechang%2Fdynamodb-data-modelling-target-stores","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jareechang%2Fdynamodb-data-modelling-target-stores/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jareechang%2Fdynamodb-data-modelling-target-stores/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jareechang%2Fdynamodb-data-modelling-target-stores/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jareechang","download_url":"https://codeload.github.com/Jareechang/dynamodb-data-modelling-target-stores/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jareechang%2Fdynamodb-data-modelling-target-stores/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31554091,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T10:21:54.569Z","status":"ssl_error","status_checked_at":"2026-04-08T10:21:38.171Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","datamodeling","docker","docker-compose","dynamodb","keyvaluestore","local-dynamodb","nodejs","python","terraform","typescript"],"created_at":"2024-11-25T23:18:07.506Z","updated_at":"2026-04-08T11:32:34.976Z","avatar_url":"https://github.com/Jareechang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dynamodb Data modelling: Target stores (WIP)\n\nData modelling demo with DynamoDB using terraform for infrastructure management.\n\nThis demostration goes from the raw data to designing and planning a DyanmoDB store for the given access patterns provided.\n\n## Table Of Contents:\n\n- [Introduction](#Introdiction)\n- [Quick Start](#quick-start)\n- [Analyze Data](#design---analyze-data)\n- [DynamoDB Indexes](#design---dynamodb-indexes)\n- [Data Transformation](#design---data-transformation)\n- [Optimization](#) TODO\n\n## Introduction\n\nDesign an efficient dynamodb with the data and the provided access patterns.\n\nPlease note this is work in process.\n\n**Access Patterns:**\n\n- Given ID, return all attributes for store (ID: 1957).\n- Return all stores that do not observe daylight savings time; return just state and store ID.\n- Given a phone area code, return all stores with formatted addresses and phone numbers (Area Code: 206).\n- Return all attributes for all stores with Starbucks and CVS.\n- Given state, return all attributes for all stores (State: WA).\n- Given state and city, return all attributes for all stores (State: WA; City: Seattle).\n- Given state city and zip, return all attributes for all stores (State: WA; City: Seattle; Zip: 98125).\n\n**Core Concepts:**\n\n- Leverage AWS DynamoDB GSI / LSI\n- Leverage AWS Querying \n- DynamoDB Design with GSI Sparse indexes\n- DynamoDB Design with GSI Composite keys  \n\n## Quick Start\n\n**Technologies:**\n\n- AWS DynamoDB\n- Terraform (\u003e= v0.12.24)\n- Python / Typescript\n\n1. Start local DynamoDB \u0026 Create Table(for local testing)  \n\n```sh\n# Start local dynamodb\ndocker compose up\n\n# Create table using terraform (Use Option 1 or 2)\n\n# Option 1, Env: local (Mock dynamoDB)\nterraform plan -auto-approve -var-file=./env/local.tfvars\nterraform apply -auto-approve -var-file=./env/local.tfvars\n\n# Option 2, Env: Dev (Use a real service from AWS)\nterraform plan -var-file=./env/dev.tfvars\nterraform apply -auto-approve -var-file=./env/dev.tfvars\n\n# IMPORTANT: Remember to destroy resources after finishing with testing it otherwise you will incur charges\nterraform destroy -auto-approve -var-file=./env/dev.tfvars\n```\n\n**Note:** `-var-file=./env/local.tfvars` points the dynamodb to the local endpoint if you want to use local dyamodb (local dev)\n\n2. Create the insert json (Using python)  \n\n```sh\npython ./scripts/generate.py\n```\n\n3. Insert Into Table (Using Node.js)   \n\n```sh\n# Install packages\nyarn\n\n# Insert data into db \nyarn run build \u0026\u0026 yarn run insert-db\n```\n## Design - Analyze Data\n\n1. Identify 'Entities' based on the access patterns  \n\n**Query Entities:**\n- Store\n\n2. Identify 'Attributes' based on the access patterns\n\n**Query Attributes:**\n\n- ID (number) \n- IsDaylightSavingsTimeRecognized (string) \n- Address.FormattedAddress (string)\n- PhoneNumber (string) \n- Address.Subdivision, \"state\" (string) \n- Address.City (string) \n- AllCapability (string[])\n- PostalCode (string) \n\n3. Identify 'Attribute' restrictions\n\n- ID must be unique (Store ID)\n\n## Design - DynamoDB Indexes\n\n1. Given state, return all attributes for all stores (State: WA).  \n\n**This be the PK/SK of the base table**\n\nIndex Type | PK | SK| Projections |\n------------ | -------------| -------------| -------------\n| BASE | Address.Subdivision | ID | None (Base Table) |\n\n**AWS CLI query:**\n```sh\n# Change the --endpoint-url if using a different environment\n# Note: Remove the --select \"Count\" if you like to see all projections from DB\n\naws dynamodb query \\\n--endpoint-url \"http://localhost:8000/\" \\\n--table-name Store \\\n--select \"COUNT\" \\\n--index-name \"Query-By-Store-State\" \\\n--key-condition-express \"#state = :v\"  \\\n--expression-attribute-names \"{\\\"#state\\\":\\\"Address.Subdivision\\\"}\" \\\n--expression-attribute-values \"{\\\":v\\\": {\\\"S\\\": \\\"WA\\\"}}\"\n```\n\n2. Given ID, return all attributes for store (ID: 1957)\n\nIndex Type | PK | SK| Projections |\n------------ | -------------| -------------| -------------\n| GSI | ID | Address.Subdivision | ALL |\n\n**Implementation:** `ID = '\u003cStore-ID\u003e'`\n\n3. Return all stores that do not observe daylight savings time; return just state and store ID.\n\nIndex Type | PK | SK| Projections |\n------------ | -------------| -------------| -------------\n| GSI | IsDaylightSavingsTimeRecognized | None | KEYS (Base PK/SK) |\n\n**Implementation:** `IsDaylightSavingsTimeRecognized = 'TRUE' | 'FALSE'`\n\n4. Given a phone area code, return all stores with formatted addresses and phone numbers (Area Code: 206).\n\nIndex Type | PK | SK| Projections |\n------------ | -------------| -------------| -------------\n| LSI | X.locale | PhoneNumber | All |\n\n**Implementation:** `X.locale = '\u003clocale\u003e' and begins_with('(\u003carea-code\u003e)')`\n\n\n5. Return all attributes for all stores with Starbucks and CVS.\n\nIndex Type | PK | SK| Projections |\n------------ | -------------| -------------| -------------\n| LSI | Starbucks | CVS | KEYS (Base PK/SK) |\n\n- Add new Fields `Starbucks` (\"EXISTS\" or blank)\n- Add new Fields `CVS` (\"EXISTS\" or blank)\n\n**Note:** *For any item in a table, DynamoDB writes a corresponding index entry only if the index sort key value is present in the item.* We will leverage the sparse indexing.\n\n**AWS CLI query:**\n\n```sh\n# Change the --endpoint-url if using a different environment\n# Note: Remove the --select \"Count\" if you like to see all projections from DB\n\naws dynamodb query \\\n--endpoint-url \"http://localhost:8000/\" \\\n--select \"COUNT\" \\\n--table-name Store \\\n--projection-expression \"ID\" \\\n--index-name \"Query-By-Starbucks-And-CSV\" \\\n--key-condition-express \"#sb = :v and #cvs = :v\"  \\\n--expression-attribute-names \"{\\\"#sb\\\":\\\"Starbucks\\\", \\\"#cvs\\\":\\\"CVS\\\"}\" \\\n--expression-attribute-values \"{\\\":v\\\": {\\\"S\\\": \\\"True\\\"}}\"\n```\n\nIndex Type | PK | SK| Projections |\n------------ | -------------| -------------| -------------\n| GSI | IsDaylightSavingsTimeRecognized | None | KEYS (Base PK/SK) |\n\n**Implementation:** `IsDaylightSavingsTimeRecognized = 'FALSE'`\n\n6. Given state and city, return all attributes for all stores (State: WA; City: Seattle).\n\nIndex Type | PK | SK| Projections |\n------------ | -------------| -------------| -------------\n| LSI | Address.Subdivision | CityZip | KEYS (Base PK/SK) |\n\n**Pre-computed key (composite key):** `Address.Subdivision` and `Address.PostalCode` named `CityZip` separated by #\n\n**Implementation:** `state = '\u003cState\u003e' and CityZip begins_with('\u003ccity\u003e')`\n\n7. Given state city and zip, return all attributes for all stores (State: WA; City: Seattle; Zip: 98125).\n\nSame as #7.\n\n**Pre-computed key (composite key):** `Address.Subdivision` and `Address.PostalCode` named `CityZip` separated by #\n\n**Implementation:** `state = '\u003cState\u003e' and CityZip begins_with('\u003ccity\u003e#\u003czip\u003e')`\n\n## Design - Data Transformation \n\nBelow are the final list of data we need to extract from the raw data to insert into dyanmoDB to support the query patterns above. \n\n**Query Attributes:**\n\n- ID (number) \n- IsDaylightSavingsTimeRecognized (string) \n- Address.FormattedAddress (string)\n- PhoneNumber, FaxNumber (string) \n- Address.Subdivision, \"state\" (string) \n- Address.City (string) \n- AllCapability (string[])\n- Address.PostalCode (string) \n- X.locale (string)\n\n**Custom Attributes:**\n\n- `CitZip` - Address.Subdivision#Address.PostalCode(string)\n- `Starbucks` (string)\n- `CVS` (string)\n\n**GSI:**\n\n1. Query-By-Store-State - PK: Address.Subdivision, SK: ID\n2. Query-By-DayLightSavings - PK: IsDaylightSavingsTimeRecognized, SK: None\n3. Query-By-Phone-Area-Code - PK: X.Locale, SK: PhoneNumber\n4. Query-By-Starbucks-And-CSV - PK: Starbucks, SK: CVS (Sparse indexing)\n\n**LSI:**\n\n5. Query-By-State-CityZip - PK: ADdress.Subdivision (by default), SK: CityZip\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjareechang%2Fdynamodb-data-modelling-target-stores","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjareechang%2Fdynamodb-data-modelling-target-stores","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjareechang%2Fdynamodb-data-modelling-target-stores/lists"}