{"id":18964011,"url":"https://github.com/JaneliaSciComp/burst-compute","last_synced_at":"2025-04-16T05:31:36.140Z","repository":{"id":78049239,"uuid":"301732359","full_name":"JaneliaSciComp/burst-compute","owner":"JaneliaSciComp","description":"Burst compute framework for AWS","archived":false,"fork":false,"pushed_at":"2025-02-26T20:22:48.000Z","size":1050,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-29T08:02:45.475Z","etag":null,"topics":["aws-lambda","burst-parallel","compute-engine","hpc"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JaneliaSciComp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-10-06T13:16:08.000Z","updated_at":"2023-04-24T13:50:33.000Z","dependencies_parsed_at":"2023-11-17T03:20:30.303Z","dependency_job_id":null,"html_url":"https://github.com/JaneliaSciComp/burst-compute","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fburst-compute","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fburst-compute/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fburst-compute/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fburst-compute/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JaneliaSciComp","download_url":"https://codeload.github.com/JaneliaSciComp/burst-compute/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249201117,"owners_count":21229004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-lambda","burst-parallel","compute-engine","hpc"],"created_at":"2024-11-08T14:22:37.910Z","updated_at":"2025-04-16T05:31:35.840Z","avatar_url":"https://github.com/JaneliaSciComp.png","language":"JavaScript","readme":"# Burst Compute Framework\n\n[![DOI](https://zenodo.org/badge/301732359.svg)](https://zenodo.org/badge/latestdoi/301732359)\n![CI workflow](https://github.com/JaneliaSciComp/burst-compute/actions/workflows/node.js.yml/badge.svg)\n\nServerless burst-compute implementation for AWS, using only native AWS services.\n\nAs seen in the [AWS Architecture Blog](https://aws.amazon.com/blogs/architecture/scaling-neuroscience-research-on-aws/).\n\nFor [embarassingly parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel) workloads, **N** items may be trivially processed by **T** threads. Given **N** items, and a **batchSize** (the maximum number of items to be processed by a single process in serial), we divide the work into **N/batchSize** batches and invoke that many user-provided **worker** Lambdas. When all the worker functions are done, the results are combined by the user-provided **combiner** Lambda. \n\nIn the diagram below, the code you write is indicated by the blue lambda icons.\n\n![Architecture Diagram](docs/burst-compute-diagram.png)\n\nHere's how it works, step-by-step:\n1) You define a **worker** function and a **combiner** function\n2) Launch your burst compute job by calling the **dispatch** function with a range of items to process\n3) The dispatcher will start copies of itself recursively and efficiently start your worker lambdas\n4) Each **worker** is given a range of inputs and must compute results for those inputs and write results to DynamoDB\n5) The Step Function monitors all the results and calls the combiner function when all workers are done\n6) The **combiner** function reads all output from DynamoDB and aggregates them into the final result\n\n## Build\n\nYou need Node.js 12.x or later in your path, then:\n\n```bash\nnpm install\n```\n\n## Deployment\n\nFollow the build instructions above before attempting to deploy.\n\nDeployment will create all the necessary AWS services, including Lambda functions, DynamoDB tables, and Step Functions. To deploy this framework to your AWS account, you must have the [AWS CLI configured](https://www.serverless.com/framework/docs/providers/aws/guide/credentials#sign-up-for-an-aws-account). \n\nTo deploy to the *dev* stage:\n```bash\nnpm run sls -- deploy\n```\n\nThis will create a application stack named `burst-compute-dev`. \n\nTo deploy to a different stage (e.g. \"prod\"), add a stage argument:\n```bash\nnpm run sls -- deploy -s prod\n```\n\n## Usage\n\n1. Create **worker** and **combiner** functions which follow the input/output specification defined in the [Interfaces](docs/Interfaces.md) document.\n2. Invoke the **dispatch** function to start a burst job. \n","funding_links":[],"categories":["High Performance Computing"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJaneliaSciComp%2Fburst-compute","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJaneliaSciComp%2Fburst-compute","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJaneliaSciComp%2Fburst-compute/lists"}