{"id":19878635,"url":"https://github.com/ironsource/node-dynamodb-stream","last_synced_at":"2025-07-19T15:32:56.755Z","repository":{"id":36761582,"uuid":"41068258","full_name":"ironSource/node-dynamodb-stream","owner":"ironSource","description":"A wrapper around low level aws sdk that makes it easy to consume a dynamodb-stream","archived":false,"fork":false,"pushed_at":"2023-10-20T23:58:52.000Z","size":2618,"stargazers_count":44,"open_issues_count":15,"forks_count":21,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-05-01T07:03:15.731Z","etag":null,"topics":["dynamodb-stream","javascript","node-js","nodejs"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ironSource.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-08-20T01:20:45.000Z","updated_at":"2022-11-02T19:58:39.000Z","dependencies_parsed_at":"2024-06-18T23:30:43.467Z","dependency_job_id":"7839b8e6-491a-448d-a114-b7af1a2b9476","html_url":"https://github.com/ironSource/node-dynamodb-stream","commit_stats":{"total_commits":65,"total_committers":4,"mean_commits":16.25,"dds":0.09230769230769231,"last_synced_commit":"61c870fed9c91b93dec5978a56aa6e5d9a7f4dd4"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Fnode-dynamodb-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Fnode-dynamodb-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Fnode-dynamodb-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Fnode-dynamodb-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ironSource","download_url":"https://codeload.github.com/ironSource/node-dynamodb-stream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252046038,"owners_count":21685935,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dynamodb-stream","javascript","node-js","nodejs"],"created_at":"2024-11-12T17:06:10.465Z","updated_at":"2025-05-02T13:30:40.330Z","avatar_url":"https://github.com/ironSource.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DynamoDBStream\n\nA wrapper around low level aws sdk that makes it easy to consume a dynamodb-stream, even in a browser.\n\n_update: serious overhaul with this [commit](https://github.com/ironSource/node-dynamodb-stream/commit/a418f5b4fc8e3948f279c6a2e57974051025c13c) and a few smaller ones after. Major version is bumped to 1.x.x_\n\n### Example: Replicating small tables\n\nfetchStreamState() should be invoked whenever the consumer wishes to get the updates.\n\nWhen a consumer needs to maintain a replica of the table data, fetchStreamState() is invoked on regular intervals.\n\nThe current best practice for replication is to manage the state of the stream as it relates to the consumer in a separate dynamodb table (shard iterators/sequence numbers etc), so if a failure occurs, that consumer can get back to the point he was in the stream. However for small or even medium tables this is not necessary. One can simply reread the entire table on startup.\n\nThis different approach make things more \"stateless\" and slightly simpler (in my view):\n\n- call fetchStreamState() first, one may safely disregard any events that are emitted at this stage. Under the hood DynamoDBStream uses ```ShardIteratorType: LATEST``` to get shard iterators for all the current shards of the stream. These iterators act as a \"bookmark\" in the stream.\n- Obtain an initial copy of the table's data (via a dynamodb scan api call for example) and store it locally\n- call fetchStreamState() again, at this point some of the events might already be included in the initial local copy of the data and some won't. Depending on the data structure thats houses the local copy of data, some filtering might be needed.\n- start polling on fetchStreamState() and blindly mutate the local copy according to the updates\n\nWrapping the initial data scan with fetchStreamState() calls insures that no changes will be missed. At worst, the second call might yield some duplicates.\n\n```js\nconst DynamoDBStream = require('dynamodb-stream')\nconst { DynamoDB } = require('@aws-sdk/client-dynamodb')\nconst { DynamoDBStreams } = require('@aws-sdk/client-dynamodb-streams')\nconst { unmarshall } = require('@aws-sdk/util-dynamodb')\n\nconst STREAM_ARN = 'your stream ARN'\nconst TABLE_NAME = 'testDynamoDBStream'\n\nasync function main() {\n\n  // table primary key is \"pk\"\n\n  const ddb = new DynamoDB()\n  const ddbStream = new DynamoDBStream(\n    new DynamoDBStreams(),\n    STREAM_ARN,\n    unmarshall\n  )\n\n  const localState = new Map()\n  await ddbStream.fetchStreamState()\n  const { Items } = await ddb.scan({ TableName: TABLE_NAME })\n  Items.map(unmarshall).forEach(item =\u003e localState.set(item.pk, item))\n  \n  // parse results and store in local state\n  const watchStream = () =\u003e {\n    console.log(localState)\n    setTimeout(() =\u003e ddbStream.fetchStreamState().then(watchStream), 10 * 1000)\n  }\n\n  watchStream()\n\n  ddbStream.on('insert record', (data, keys) =\u003e {\n    localState.set(data.pk, data)\n  })\n\n  ddbStream.on('remove record', (data, keys) =\u003e {\n    localState.remove(data.pk)\n  })\n\n  ddbStream.on('modify record', (newData, oldData, keys) =\u003e {\n    localState.set(newData.pk, newData)\n  })\n\n  ddbStream.on('new shards', (shardIds) =\u003e {})\n  ddbStream.on('remove shards', (shardIds) =\u003e {})\n}\n\nmain()\n```\n\n### Example: shards / iterator persistence\n\nIf your program crash and you want to pick up where you left off then `setShardsState()` and `getShardState()` are here for the rescue (though, I haven't tested them yet but they should work... :) )\n\n```js\nconst DynamoDBStream = require('dynamodb-stream')\nconst { DynamoDBStreams } = require('@aws-sdk/client-dynamodb-streams')\nconst { unmarshall } = require('@aws-sdk/util-dynamodb')\nconst fs = require('fs').promises\n\nconst STREAM_ARN = 'your stream ARN'\nconst FILE = 'shardState.json'\n\nasync function main() {\n  const ddbStream = new DynamoDBStream(\n    new DynamoDBStreams(),\n    STREAM_ARN,\n    unmarshall\n  )\n\n  // update the state so it will pick up from where it left last time\n  // remember this has a limit of 24 hours or something along these lines\n  // https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html\n  ddbStream.setShardState(await loadShardState())\n\n  const fetchStreamState = () =\u003e {\n    setTimeout(async () =\u003e {\n      await ddbStream.fetchStreamState()\n      const shardState = ddbStream.getShardState()\n      await fs.writeFile(FILE, JSON.stringify(shardState))\n      fetchStreamState()\n    }, 1000 * 20)\n  }\n\n  fetchStreamState()\n}\n\nasync function loadShardState() {\n  try {\n    return JSON.parse(await fs.readFile(FILE, 'utf8'))\n  } catch (e) {\n    if (e.code === 'ENOENT') return {}\n    throw e\n  }\n}\n\nmain()\n```\n\n#### TODO\n - make sure the aggregation of records is in order - the metadata from the stream might be helpful (order by sequence number?)\n - what about sequence numbers and other types of iterators? (TRIM_HORIZON | LATEST | AT_SEQUENCE_NUMBER | AFTER_SEQUENCE_NUMBE)\n \n#### Wishlist to DynamoDB team:\n1. expose push interface so one won't need to poll the stream api\n2. obtain a sequence number from a dynamodb api scan operation\n\n[MIT](http://opensource.org/licenses/MIT) © ironSource ltd.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fironsource%2Fnode-dynamodb-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fironsource%2Fnode-dynamodb-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fironsource%2Fnode-dynamodb-stream/lists"}