{"id":20597096,"url":"https://github.com/tradle/aws-stream-sort","last_synced_at":"2026-04-17T23:04:27.186Z","repository":{"id":66294658,"uuid":"93979195","full_name":"tradle/aws-stream-sort","owner":"tradle","description":null,"archived":false,"fork":false,"pushed_at":"2017-06-13T18:39:03.000Z","size":23,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-01-17T01:07:44.890Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tradle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-11T03:23:47.000Z","updated_at":"2017-09-19T22:49:52.000Z","dependencies_parsed_at":"2023-02-25T23:00:14.006Z","dependency_job_id":null,"html_url":"https://github.com/tradle/aws-stream-sort","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tradle%2Faws-stream-sort","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tradle%2Faws-stream-sort/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tradle%2Faws-stream-sort/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tradle%2Faws-stream-sort/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tradle","download_url":"https://codeload.github.com/tradle/aws-stream-sort/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242231441,"owners_count":20093636,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T08:20:18.659Z","updated_at":"2026-04-17T23:04:27.152Z","avatar_url":"https://github.com/tradle.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aws-stream-sort\n\ncursor / priority queue using DynamoDB Streams (optionally) and Lambda (no SQS). To enable the following:\n\nout-of-order messages -\u003e [lambda using this library] -\u003e in-order event stream from DynamoDB\n\n## Usage\n\n```js\nconst co = require('co').wrap\nconst { createAutpilot } = require('aws-cursor')\nconst docClient = new AWS.DynamoDB.DocumentClient()\nconst autopilot = createAutopilot({\n  docClient,\n  cursorTable: 'MyCursorTableName',\n  itemsTable: 'MyItemsTableName',\n  queueProp: 'author',\n  seqProp: 'seq'\n})\n\n// lambda - incoming message processor\n\nexports.enqueue = co(function* (event, context, callback) {\n  try {\n    const result = yield autopilot.put(event)\n    if (result.new \u003e result.old) {\n      // you can safely process items with sequence numbers `result.old \u003c seq \u003c= result.new`\n      // or you can stream from the CursorTable to another lambda and process there (below)\n    }\n\n    callback()\n  } catch (err) {\n    console.error(err)\n    callback(err)\n  }\n})\n\n// lambda - cursor table stream processor\n\nexports.process = co(function* (event, context, callback) {\n  // inspect OldImage and NewImage\n  // to get change in cursor position\n  // process batch indicated by the change in position\n})\n```\n\n## The algorithm\n\nGiven two tables:\n- `items`: table for your events\n- `cursor`: table for storing cursor position for queues Q1, Q2, ...\n\nFor incoming message M, with sequence number S, which falls into queue Q1\n\n1. Store M in `items`\n2. If S is not the next sequence number in Q1, continue to step 3 (else, exit)\n3. increment the cursor value for Q1\n4. scan ahead `batchSize` items in `items` to see if we have future messages that arrived out-of-order, and update the cursor for Q1 again\n5. repeat step 4 as long as there are more in-order items ahead\n\nThe DynamoDB stream from the Cursor table will be in-order per queue, with the old and new cursor positions for that queue in OldImage and NewImage respectively\n\n## Setup\n1. `queueProp` + `seq` should be your partition + sort keys in the `items` table\n2. The cursor table should have `queueProp` as its partition key (no sort key)\n\nSee [./test/inbox-table-schema.json](./test/inbox-table-schema.json) and [./test/cursor-table-schema.json](./test/cursor-table-schema.json) for an example of a compatible schema.\n\n## Testing\n\nTo run the tests, first run [localstack](https://github.com/atlassian/localstack)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftradle%2Faws-stream-sort","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftradle%2Faws-stream-sort","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftradle%2Faws-stream-sort/lists"}