{"id":13518241,"url":"https://github.com/littlstar/s3-lambda","last_synced_at":"2025-10-09T17:36:42.315Z","repository":{"id":57157562,"uuid":"45482759","full_name":"littlstar/s3-lambda","owner":"littlstar","description":"Lambda functions over S3 objects with concurrency control (each, map, reduce, filter)","archived":false,"fork":false,"pushed_at":"2017-05-17T16:38:53.000Z","size":1362,"stargazers_count":1065,"open_issues_count":3,"forks_count":47,"subscribers_count":43,"default_branch":"master","last_synced_at":"2025-09-30T12:57:53.007Z","etag":null,"topics":["concurrency","lambda-functions","node","s3"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/littlstar.png","metadata":{"files":{"readme":"README.md","changelog":"History.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-11-03T17:20:02.000Z","updated_at":"2025-09-29T13:18:13.000Z","dependencies_parsed_at":"2022-09-05T21:30:27.370Z","dependency_job_id":null,"html_url":"https://github.com/littlstar/s3-lambda","commit_stats":null,"previous_names":["littlstar/s3renity"],"tags_count":68,"template":false,"template_full_name":null,"purl":"pkg:github/littlstar/s3-lambda","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/littlstar%2Fs3-lambda","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/littlstar%2Fs3-lambda/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/littlstar%2Fs3-lambda/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/littlstar%2Fs3-lambda/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/littlstar","download_url":"https://codeload.github.com/littlstar/s3-lambda/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/littlstar%2Fs3-lambda/sbom","scorecard":{"id":593213,"data":{"date":"2025-08-11","repo":{"name":"github.com/littlstar/s3-lambda","commit":"79e04b21db8a489df99850991f1e0d2963845580"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":3,"reason":"Found 5/13 approved changesets -- score normalized to 3","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 22 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-20T22:29:04.373Z","repository_id":57157562,"created_at":"2025-08-20T22:29:04.373Z","updated_at":"2025-08-20T22:29:04.373Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001871,"owners_count":26083197,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","lambda-functions","node","s3"],"created_at":"2024-08-01T05:01:42.549Z","updated_at":"2025-10-09T17:36:42.295Z","avatar_url":"https://github.com/littlstar.png","language":"JavaScript","funding_links":[],"categories":["JavaScript","Open Source Repos"],"sub_categories":["Lambda"],"readme":"## s3-lambda\n\n`s3-lambda` enables you to run lambda functions over a context of [S3](https://aws.amazon.com/s3/) objects. It has a stateless architecture with concurrency control, allowing you to process a large number of files very quickly. This is useful for quickly prototyping complex data jobs without an infrastructure like Hadoop or Spark.\n\nAt Littlstar, we use `s3-lambda` for all sorts of data pipelining and analytics.\n\n**Disclaimer** This module does *not* interact with the AWS Lambda service; the name `s3-lambda` is referring to lambda functions in computer science, and all s3 file processing happens locally.\n\n## Install\n```bash\nnpm install s3-lambda --save\n```\n\n## Quick Example\n```javascript\nconst S3Lambda = require('s3-lambda')\n\n// example options\nconst lambda = new S3Lambda({\n  accessKeyId: 'aws-access-key',       // Optional. (falls back on local AWS credentials)\n  secretAccessKey: 'aws-secret-key',   // Optional. (falls back on local AWS credentials)\n  showProgress: true,                  // Optional. Show progress bar in stdout\n  verbose: true,                       // Optional. Show all S3 operations in stdout (GET, PUT, DELETE)\n  signatureVersion: 'v4',              // Optional. Signature Version used in Authentication. Defaults to \"v4\"\n  maxRetries: 10,                      // Optional. Maximum request retries on an S3 object. Defaults to 10.\n  timeout: 10000                       // Optional. Amount of time for request to timeout. Defaults to 10000 (10s)\n})\n\nconst context = {\n  bucket: 'my-bucket',\n  prefix: 'path/to/files/'\n}\n\nlambda\n  .context(context)\n  .forEach(object =\u003e {\n    // do something with object\n  })\n  .then(_ =\u003e console.log('done!'))\n  .catch(console.error)\n```\n\n## Setting Context\nBefore initiating a lambda expression, you must tell `s3-lambda` what files to operate over by calling `context`. A context is defined with an options object with the following properties: **bucket**, **prefix**, **marker**, **limit**, and **reverse**.\n\n```javascript\nlambda.context({\n  bucket: 'my-bucket',       // The S3 bucket to use\n  prefix: 'prefix/',         // The prefix of the files to use - s3-lambda will operate over every file with this prefix.\n  marker: 'prefix/file1',    // Optional. Start at the first file with this prefix. If it is a full file path, starts with next file. Defaults to null.\n  endPrefix: 'prefix/file3', // Optional. Process files up to (not including) this prefix. Defaults to null.\n  match: /2017/i,            // Optional. Process files matching this regex / string. Defaults to null.\n  limit: 1000,               // Optional. Limit the # of files operated over. Default is Infinity.\n  reverse: false             // Optional. If true, operate over all files in reverse. Defaults to false.\n})\n```\nYou can also provide an array of context options, which will tell `ls-lambda` to operate over all the files in each.\n```javascript\nconst ctx1 = {\n  bucket: 'my-bucket',\n  prefix: 'path/to/files/',\n  marker: 'path/to/logs/2017'\n}\nconst ctx2 = {\n  bucket: 'my-other-bucket',\n  prefix: 'path/to/other/logs/',\n  limit: 100\n}\n\nlambda.context([ctx1, ctx2])\n```\n\n## Modifiers\nAfter setting context, you can chain several other functions that modify the operation. Each returns a `Request` object, so they can be chained. All of these are optional.\n### .concurrency(c)\n{Number} Set the request concurrency level (default is `Infinity`).\n\n### .exclude(e)\n{Function} Sets the exclude function to use before getting objects from S3. This function will be called with the key and should return `true` if the object should be excluded.  \n**Example:** exclude png files\n```javascript\nlambda\n  .context(context)\n  .exclude(key =\u003e /.png$/.test(key))\n  .each(...)\n```\n\n### .transform(f)\n{Function} Sets the transformation function to use when getting objects. This transformer will be called with the raw object that is returned by the [`S3#getObject()`](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property) method in the AWS SDK and the key, and should return the transformed object.  When a transformer function is provided, objects are not automatically converted to strings, and the `encoding` parameter is ignored.\n**Example:** unzipping compressed S3 files before each operation\n```javascript\nconst zlib = require('zlib')\n\nlambda\n  .context(context)\n  .transform((object) =\u003e {\n    return zlib.gunzipSync(object.Body).toString('utf8')\n  })\n  .each(...)\n```\n### .encode(e)\n{String} Sets the string encoding to use when getting objects.  This setting is ignored if a transformer function is used.\n### limit(l)\n{Number} Limit the number of files operated over.\n### reverse(r)\n{Boolean} Reverse the order of files operated over.\n### async()\nLets the resolver know that your function is async (returns a Promise).\n\n## Lambda Functions\nPerform synchronous or asynchronous functions over each file in the set context.\n- each\n- forEach\n- map\n- reduce\n- filter\n\n### each\neach(fn[, isasync])  \n\nPerforms `fn` on each S3 object in parallel. You can set the concurrency level (defaults to `Infinity`).\nIf `isasync` is true, `fn` should return a Promise.\n```javascript\nlambda\n  .context(bucket, prefix)\n  .concurrency(5) // operates on 5 objects at a time\n  .each(object =\u003e console.log(object))\n  .then(_ =\u003e console.log('done!'))\n  .catch(console.error)\n```\n\n### forEach\nforEach(fn[, isasync])  \n\nSame as `each`, but operates sequentially, one file at a time. Setting concurrency for this function is superfluous.\n```javascript\nlambda\n  .context(bucket, prefix)\n  .forEach(object =\u003e { /* do something with object */ })\n  .then(_ =\u003e console.log('done!'))\n  .catch(console.error)\n```\n### map\nmap(fn[, isasync])  \n\nMaps `fn` over each file in an S3 directory, replacing each file with what is returned\nfrom the mapper function.   If `isasync` is true, `fn` should return a Promise.\n\nThis is a **destructive** action, meaning what you return from `fn` will change the S3 object itself. For your protection, you must specify `inplace()` to map over the existing files. Alternatively, you can use `output()` to output the results of the mapper function elsewhere (as demonstrated below). You can pass a third argument (a function) to rename the output key (bucket + prefix).\n```javascript\nconst addSmiley = object =\u003e object + ':)'\n\nlambda\n  .context(bucket, prefix)\n  .inplace()\n  .map(addSmiley)\n  .then(console.log('done!'))\n  .catch(console.error)\n```\nMake this *non-destructive* by specifying an `output` directory.\n```javascript\nconst outputBucket = 'my-bucket'\nconst outputPrefix = 'path/to/output/'\n\nlambda\n  .context(bucket, prefix)\n  .output(outputBucket, outputPrefix, (key) =\u003e key.replace('-', '/'))\n  .map(addSmiley)\n  .then(console.log('done!'))\n  .catch(console.error)\n```\n### reduce\nreduce(func[, isasync])  \n\nReduces the objects in the working context to a single value.\n```javascript\n// concatonates all the files\nconst reducer = (previousValue, currentValue, key) =\u003e {\n  return previousValue + currentValue\n}\n\nlambda\n  .context(bucket, prefix)\n  .reduce(reducer)\n  .then(result =\u003e { /* do something with result */ })\n  .catch(console.error)\n```\n### filter\nfilter(func[, isasync])  \n\n**Destructive**.  Filters (deletes) files in S3. `func` should return `true` to keep the object, and `false` to delete it. If `isasync` is true, `func` returns a Promise.\n\nThis is a **destructive** action, meaning if `fn` is `false`, the object will be deleted from S3. For your protection, you must specify `inplace()` to filter the existing files. Alternatively, you can use `output()` to output the results of the filter function elsewhere (as demonstrated below). As with map, you can pass a function to output to rename the output key.\n\n```javascript\n// filters empty files\nconst fn = object =\u003e object.length \u003e 0\n\nlambda\n  .context(bucket, prefix)\n  .inplace()\n  .filter(fn)\n  .then(_ =\u003e console.log('done!'))\n  .catch(console.error)\n```\nMake this *non-destructive* by specifying an `output` directory.\n```javascript\nlambda\n  .context(bucket, prefix)\n  .output(outputBucket, outputPrefix, (key) =\u003e key.replace('-', '/'))\n  .filter(filter)\n  .then(console.log('done!'))\n  .catch(console.error()\n```\n## S3 Functions\nPromise-based wrapper around common S3 methods.\n- list\n- keys\n- get\n- put\n- copy\n- delete\n\n### list\nlist(bucket, prefix[, marker])  \n\nList all keys in `s3://bucket/prefix`.  If you use a marker, `s3-lambda` will start listing alphabetically from there.\n```javascript\nlambda\n  .list(bucket, prefix)\n  .then(list =\u003e console.log(list))\n  .catch(console.error)\n```\n### keys\nkeys(bucket, prefix[, marker])  \n\nReturns an array of keys for the given `bucket` and `prefix`.\n```javascript\nlambda\n  .keys(bucket, prefix)\n  .then(keys =\u003e console.log(keys))\n  .catch(console.error)\n```\n### get\nget(bucket, key[, encoding[, transformer]])  \n\nGets an object in S3, calling `toString(encoding` on objects.\n```javascript\nlambda\n  .get(bucket, key)\n  .then(object =\u003e { /* do something with object */ })\n  .catch(console.error)\n```\n\nOptionally you can supply your own transformer function to use when retrieving objects.  This transformer will be called with the raw object that is returned by the [`S3#getObject()`](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property) method in the AWS SDK, and should return the transformed object.  When a transformer function is provided, objects are not automatically converted to strings, and the `encoding` parameter is ignored.\n\n```javascript\nconst zlib = require('zlib')\n\nconst transformer = object =\u003e {\n  return zlib.gunzipSync(object.Body).toString('utf8')\n}\n\nlambda\n  .get(bucket, key, null, transformer)\n  .then(object =\u003e { /* do something with object */ })\n  .catch(console.error)\n```\n\n### put\nput(bucket, key, object[, encoding])  \n\nPuts an object in S3.  Default encoding is `utf8`.\n```javascript\nlambda\n  .put(bucket, key, 'hello world!')\n  .then(console.log('done!')).catch(console.error)\n```\n### copy\ncopy(bucket, key, targetBucket, targetKey)  \n\nCopies an object in S3 from `s3://sourceBucket/sourceKey` to `s3://targetBucket/targetKey`.\n```javascript\nlambda\n  .copy(sourceBucket, sourceKey, targetBucket, targetKey)\n  .then(console.log('done!')).catch(console.error)\n```\n### delete\ndelete(bucket, key)  \n\nDeletes an object in S3 (`s3://bucket/key`).\n```javascript\nlambda\n  .delete(bucket, key)\n  .then(console.log('done!')).catch(console.error)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flittlstar%2Fs3-lambda","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flittlstar%2Fs3-lambda","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flittlstar%2Fs3-lambda/lists"}