{"id":17444460,"url":"https://github.com/jameskyburz/dynamodb-logs","last_synced_at":"2026-03-10T05:03:58.105Z","repository":{"id":38736527,"uuid":"327881775","full_name":"JamesKyburz/dynamodb-logs","owner":"JamesKyburz","description":"sample repo to use dynamodb with append only logs","archived":false,"fork":false,"pushed_at":"2023-02-03T17:44:39.000Z","size":7907,"stargazers_count":4,"open_issues_count":19,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-10T13:09:23.870Z","etag":null,"topics":["append-only","aws","aws-lambda","aws-lambda-node","aws-lambda-python","dynamodb","dynamodb-streams","event-sourcing","eventbridge","eventlog","lambda","pubsub","serverless"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JamesKyburz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-08T11:18:25.000Z","updated_at":"2025-01-14T06:39:19.000Z","dependencies_parsed_at":"2023-02-18T09:31:31.309Z","dependency_job_id":null,"html_url":"https://github.com/JamesKyburz/dynamodb-logs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/JamesKyburz/dynamodb-logs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JamesKyburz%2Fdynamodb-logs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JamesKyburz%2Fdynamodb-logs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JamesKyburz%2Fdynamodb-logs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JamesKyburz%2Fdynamodb-logs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JamesKyburz","download_url":"https://codeload.github.com/JamesKyburz/dynamodb-logs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JamesKyburz%2Fdynamodb-logs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30325603,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T01:36:58.598Z","status":"online","status_checked_at":"2026-03-10T02:00:06.579Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["append-only","aws","aws-lambda","aws-lambda-node","aws-lambda-python","dynamodb","dynamodb-streams","event-sourcing","eventbridge","eventlog","lambda","pubsub","serverless"],"created_at":"2024-10-17T17:26:33.135Z","updated_at":"2026-03-10T05:03:58.080Z","avatar_url":"https://github.com/JamesKyburz.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dynamodb log\n\nsample repo to use DynamoDB with append only logs.\n\nexamples contain either Node.js or Python.\n\nDynamoDB write triggers\n\n![DynamoDB table(dynamodb-logs) writes][writes]\n\nEventBridge reads\n\n![EventBridge DynamoDB table (dynamodb-logs) reads][reads]\n\n### [Speakeasy JS][speakeasyjs]\n\nHere is the Speakeasy JS video from April 2021.\n\n[![Speakeasy JS](https://img.youtube.com/vi/jyITB37DXFs/1.jpg)](https://www.youtube.com/watch?v=jyITB37DXFs)\n\n### why?\n\nI previously built [level-eventstore] and wanted the same benefits of append only logs, but using serverless.\n\nBy using DynamoDB with [DynamoDB Streams] we can build append only logs.\nAlthough we cannot implement a strict append only log we can order log items by when they are written, and with conditional writes we can perform optimistic locking.\n\nWe can preserve the item order when reading from DynamoDB using a sort key.\n\n- Logs are saved in DynamoDB.\n- Publish / Subscribe changes using [EventBridge].\n\n### license\n\n[Apache License, Version 2.0](LICENSE)\n\n### writing to DynamoDB\n\nexample payload written to DynamoDB\n\n```json\n{\n  \"pk\": \"users#12#stream\",\n  \"sk\": 1,\n  \"type\": \"signup\",\n  \"log\": \"users\",\n  \"payload\": {\n    \"id\": \"12\",\n    \"name\": \"test\",\n    \"email\": \"test@example.com\"\n  }\n}\n```\n\n- pk (partition key) is `log name#id#stream`.\n- sk (sort key).\n- type is the name of the event useful for event handlers.\n- log name of log.\n- payload should contain the id and any optional extra fields.\n\nWhen items are written to DynamoDB they are written to the DynamoDB stream for the shard they belong to in the order they are written.\n\nThe lambda is then triggered which will publish the changed keys to [EventBridge].\n\n### example lambda triggers\n\n- [Node.js lambda trigger](./src/trigger.js)\n- [Python lambda trigger](./src/trigger.py)\n\n### example event handlers triggered by EventBridge\n\n- [Node.js event handler example](./src/handler.js)\n- [Python event handler example](./src/handler.py)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003einstall prerequisites\u003c/summary\u003e\n\n### only required if using python\n\n- [python 3.8.6](https://www.python.org/downloads/release/python-386)\n- [virtualenv](https://virtualenv.pypa.io/en/latest/installation.html)\n\n### required to run locally in offline mode and for linting\n\n- [docker](https://www.docker.io)\n- [docker-compose](https://docs.docker.com/compose)\n\n### required (serverless framework and tools)\n\n- [nodejs 14](https://nodejs.org)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003ecli.sh\u003c/summary\u003e\n\n### interactive cli script for both local and AWS\n\n```\nnpm install\n```\n\n```sh\n./cli.sh\n```\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\u003csummary\u003eSpeakeasy JS\u003c/summary\u003e\n\n### [Speakeasy JS][speakeasyjs]\n\n```sh\n  npm run speakeasyjs\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003ein AWS\u003c/summary\u003e\n\nexport AWS credentials before running `cli.sh`\n\n### Node.js deploy to AWS\n\n```sh\nnpm i\n./cli.sh\nnpm exec sls deploy -- --stage dev -c serverless-node.yml\n```\n\n### Node.js remove stack in AWS\n\n```sh\nnpm i\n./cli.sh\nnpm exec sls remove -- --stage dev -c serverless-node.yml\n```\n\n### Python deploy to AWS\n\n```sh\nnpm i\n./cli.sh\nrm -rf venv\nvirtualenv venv\n. venv/bin/activate\npip3 install -r requirements.txt\nnpm exec sls deploy -- --stage dev -c serverless-python.yml\nrm -rf venv\n```\n\n### Python remove stack in AWS\n\n```sh\nnpm i\n./cli.sh\nnpm exec sls remove -- --stage dev -c serverless-python.yml\n```\n\nQuery DynamoDB\n\n```sh\n./cli.sh\n./dynamodb-local-query.sh\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003egenerate fake user data\u003c/summary\u003e\n\nGenerate fake user data locally or in aws.\n\n```sh\n./create-fake-data.sh\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eoffline\u003c/summary\u003e\n\n### Node.js\n\n```sh\n./cli.sh\nnpm exec sls offline start -- --stage=local -c serverless-node.yml\n```\n\n### Python\n\n```sh\n./cli.sh\nvirtualenv venv\n. venv/bin/activate\npip3 install -r requirements.txt\nnpm exec sls offline start -- --stage=local -c serverless-python.yml\n```\n\nAdd item using aws cli.\n\n```sh\n./cli.sh\nexport AWS_ACCESS_KEY_ID=x\nexport AWS_SECRET_ACCESS_KEY=x\nexport AWS_DEFAULT_REGION=us-east-1\naws dynamodb put-item \\\n  --table-name local-dynamodb-logs \\\n  --item \"\"\"\n  {\n    \\\"pk\\\": { \\\"S\\\": \\\"users#12#stream\\\" },\n    \\\"sk\\\": { \\\"N\\\": \\\"${sk:-1}\\\" },\n    \\\"type\\\": { \\\"S\\\": \\\"signup\\\" },\n    \\\"log\\\": { \\\"S\\\": \\\"users\\\" },\n    \\\"payload\\\": { \\\"M\\\": {\n      \\\"id\\\": { \\\"S\\\": \\\"12\\\"},\n      \\\"name\\\": { \\\"S\\\": \\\"test\\\"}\n      \\\"email\\\": { \\\"S\\\": \\\"test@example.com\\\"}\n    }}\n  }\"\"\" \\\n  --condition-expression \"attribute_not_exists(pk)\" \\\n  --endpoint http://localhost:8000\n```\n\nQuery DynamoDB\n\n```sh\n./cli.sh\n./dynamodb-local-query.sh\n```\n\nCleanup\n\nRemove dynamodb data and docker-compose processes.\n\nAlso needed if you have new AWS credentials.\n\n```sh\n./cli.sh stop\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eretries offline and in AWS\u003c/summary\u003e\n\n### offline retries\n\nper stack retry limit is configurable using the [offline eventbridge plugin].\n\n```yaml\ncustom:\n  serverless-offline-aws-eventbridge:\n    retryDelayMs: 1000\n    maximumRetryAttempts: 5\n```\n\n### destinations\n\nusing onFailure failures can be handled by your application.\n\nfailure payloads look like this\n\n```yaml\neventHandler:\n  handler: src/handler.handler\n  destinations:\n    onFailure: retry\n  iamRoleStatements:\n    - Effect: Allow\n      Action:\n        - \"lambda:InvokeFunction\"\n      Resource:\n        - !Sub \"arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:dynamodb-logs-${opt:stage}-retry\"\n```\n\n```json\n{\n  \"version\": \"1.0\",\n  \"timestamp\": \"\u003ctimestamp\u003e\",\n  \"requestContext\": {\n    \"requestId\": \"\u003cuuid\u003e\",\n    \"functionArn\": \"arn:...\",\n    \"condition\": \"RetriesExhausted\",\n    \"approximateInvokeCount\": 3\n  },\n  \"requestPayload\": {\n    \"version\": \"0\",\n    \"id\": \"\u003cuuid\u003e\",\n    \"detail-type\": \"stream changes\",\n    \"source\": \"dynamodb-log\",\n    \"account\": \"\u003caccount\u003e\",\n    \"time\": \"\u003ctimestamp\u003e\",\n    \"region\": \"\u003cregion\u003e\",\n    \"resources\": [],\n    \"detail\": {\n      \"log\": \"\u003clog\u003e\",\n      \"key\": {\n        \"pk\": \"\u003cpk\u003e\",\n        \"sk\": 0\n      },\n      \"type\": \"\u003csignup\u003e\",\n      \"payload\": {\n        \"id\": \"\u003cid\u003e\"\n      }\n    }\n  },\n  \"responseContext\": {\n    \"statusCode\": 200,\n    \"executedVersion\": \"$LATEST\",\n    \"functionError\": \"Unhandled\"\n  },\n  \"responsePayload\": {\n    \"errorType\": \"Error\",\n    \"errorMessage\": \"\u003cerror message\u003e\",\n    \"trace\": []\n  }\n}\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003ereplay using EventBridge archive\u003c/summary\u003e\n\nWhen run offline events are persisted to sqlite3 file `./db`.\n\nEvents can be replayed to EventBridge or DynamoDB, you can also replay events from AWS to local which will deplay an extra stack using a websocket lambda and a new event target that writes to the connected websockets.\n\n```sh\n./cli.sh\n./run-archive.js\n```\n\n\u003c/details\u003e\n\n[speakeasyjs]: https://speakeasyjs.com\n[writes]: ./diagrams/writes.png\n[reads]: ./diagrams/reads.png\n[eventbridge]: https://aws.amazon.com/eventbridge/\n[offline eventbridge plugin]: https://github.com/rubenkaiser/serverless-offline-eventBridge\n[level-eventstore]: https://github.com/JamesKyburz/level-eventstore\n[dynamodb streams]: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjameskyburz%2Fdynamodb-logs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjameskyburz%2Fdynamodb-logs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjameskyburz%2Fdynamodb-logs/lists"}