{"id":18908557,"url":"https://github.com/superluminar-io/step-worker","last_synced_at":"2025-08-02T03:09:24.311Z","repository":{"id":103367487,"uuid":"178374101","full_name":"superluminar-io/step-worker","owner":"superluminar-io","description":"Use AWS State Machine to process DynamoDB Objects.","archived":false,"fork":false,"pushed_at":"2019-04-02T12:07:14.000Z","size":853,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-06-17T14:50:10.642Z","etag":null,"topics":["aws","dynamodb","lambda","serverless","statemachine","stepfunctions","worker"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/superluminar-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-29T09:24:39.000Z","updated_at":"2021-03-18T09:38:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"e2696103-8c25-4555-aa00-f378dd228a6a","html_url":"https://github.com/superluminar-io/step-worker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/superluminar-io/step-worker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superluminar-io%2Fstep-worker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superluminar-io%2Fstep-worker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superluminar-io%2Fstep-worker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superluminar-io%2Fstep-worker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/superluminar-io","download_url":"https://codeload.github.com/superluminar-io/step-worker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superluminar-io%2Fstep-worker/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268330911,"owners_count":24233151,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","dynamodb","lambda","serverless","statemachine","stepfunctions","worker"],"created_at":"2024-11-08T09:27:10.554Z","updated_at":"2025-08-02T03:09:24.291Z","avatar_url":"https://github.com/superluminar-io.png","language":"Go","readme":"# DynamoDB Step Worker\n\n\u003e Example project for AWS [Step Functions](https://aws.amazon.com/step-functions/) and [State Machines](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html) to process data objects stored in a DynamoDB table with parallel Scans using Segments.\n\nThe CloudFormation Stack creates two State Machines: **Puppeteer** and **Worker**. You can configure the **Puppeteer** to fan out the desired number of **Worker** executions to process the DynamoDB objects with the following configuration:\n\n```json\n{\n  \"Comment\": \"Run State Machine Workers\",\n  \"Configuration\": {\n    \"TableName\": \"step-worker-url-stable\",\n    \"TableHash\": \"url\",\n    \"Workers\": 4,\n    \"BatchSize\": 250\n  }\n}\n```\n\n\n![State Machine](/machine.png)\n\n## Usage\n\n\n\n### Dependencies\n\n```bash\n# Install Go dependencies\n\n$ \u003e make install\n```\n\n### Deployment\n\n```bash\n# Compile Lambda w/ Go and deploy CloudFormation Stack\n\n$ \u003e make build package deploy\n```\n\n### Execute State Machine\n\n```bash\n# Get ARN of State Machine \n\n$ \u003e make outputs\n\n[\n  {\n    \"OutputKey\": \"Puppeteer\",\n    \"OutputValue\": \"arn:aws:states:eu-west-1:1234567890:stateMachine:step-worker-puppeteer-stable\",\n    \"Description\": \"ARN for State Machine\"\n  }\n]\n\n# Execute State Machine and configure BatchSize\n\n$ \u003e aws stepfunctions start-execution \\\n    --state-machine-arn arn:aws:states:eu-west-1:1234567890:stateMachine:step-worker-puppeteer-stable \\\n    --input '\n    {\n      \"Comment\": \"Run State Machine with BatchSize 25\",\n      \"Configuration\": {\n        \"TableName\": \"step-worker-url-stable\",\n        \"TableHash\": \"url\",\n        \"Workers\": 4,\n        \"BatchSize\": 250\n      }\n    }'\n```\n\n## License\n\nFeel free to use the code, it's released using the [MIT license](LICENSE.md).\n\n## Contribution\n\nYou are welcome to contribute to this project! 😘 \n\nTo make sure you have a pleasant experience, please read the [code of conduct](CODE_OF_CONDUCT.md). It outlines core values and beliefs and will make working together a happier experience.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuperluminar-io%2Fstep-worker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuperluminar-io%2Fstep-worker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuperluminar-io%2Fstep-worker/lists"}