{"id":17862146,"url":"https://github.com/jtblin/alamo","last_synced_at":"2025-10-06T11:26:25.277Z","repository":{"id":29893398,"uuid":"33438988","full_name":"jtblin/alamo","owner":"jtblin","description":"High level full streaming API s3 client with multipart concurrent uploads, and automatic retries with exponential backoff","archived":false,"fork":false,"pushed_at":"2016-07-04T06:55:35.000Z","size":23,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-12T10:25:37.640Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"justjavac/free-programming-books-zh_CN","license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jtblin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-04-05T11:34:20.000Z","updated_at":"2016-02-01T22:59:07.000Z","dependencies_parsed_at":"2022-09-03T08:51:17.777Z","dependency_job_id":null,"html_url":"https://github.com/jtblin/alamo","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtblin%2Falamo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtblin%2Falamo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtblin%2Falamo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtblin%2Falamo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jtblin","download_url":"https://codeload.github.com/jtblin/alamo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234067147,"owners_count":18774203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-28T08:50:44.401Z","updated_at":"2025-09-24T09:31:45.569Z","avatar_url":"https://github.com/jtblin.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![NPM version](https://badge.fury.io/js/alamo.svg)](http://badge.fury.io/js/alamo)\n[![Build Status](https://travis-ci.org/jtblin/alamo.png)](https://travis-ci.org/jtblin/alamo)\n[![Code Climate](https://codeclimate.com/github/jtblin/alamo/badges/gpa.svg)](https://codeclimate.com/github/jtblin/alamo)\n[![Dependency Status](https://david-dm.org/jtblin/alamo.svg)](https://david-dm.org/jtblin/alamo)\n![Code Coverage](https://rawgit.com/jtblin/alamo/master/test/coverage.svg)\n[![experimental](http://badges.github.io/stability-badges/dist/experimental.svg)](http://github.com/badges/stability-badges)\n\n# Overview\n\n`alamo` is a wrapper around [knox](https://github.com/LearnBoost/knox) that provides an higher level abstraction\nfor s3 with handling of response status codes and automatic parsing of XML error bodies. It also provide a consistent\nfull (writing and reading) streaming interface, including multipart upload for large artifacts. Alamo implements\nautomatic retries on error with exponential back-off.\n\n## Why alamo?\n\n1. **knox** is quite low-level with regards to response and error handling which lead to code duplication to parse\nresponse status codes and errors for each request as pointed out by @domenic himself in https://github.com/Automattic/knox/issues/114\n1. the aws-sdk allow uploading streams (in an awkward way) but does not allow retrieving streams\n1. the aws-sdk allow multipart upload but is low-level and let a lot to implement by the caller\n1. [knox-mpu](https://github.com/nathanoehlman/knox-mpu) allows multipart upload but buffers everything in memory\nwhich is only viable when uploading from your desktop without concurrency, but not viable from a server\n1. neither **knox** or **knox-mpu** implements retries which is problematic when uploading large artifacts to s3\n1. [Fort Knox](http://en.wikipedia.org/wiki/Fort_Knox) - [Fort Alamo](http://en.wikipedia.org/wiki/Battle_of_the_Alamo)\n\n## Usage\n\n    npm install --save alamo\n\n## API\n\n### alamo.createClient(options)\n\n  Returns an s3 client. It accepts the same options as\n  [knox.createClient](https://github.com/Automattic/knox#client-creation-options) plus the following ones\n  that set how retries work (see [retry](https://www.npmjs.com/package/retry)):\n\n  * `retries`: max number of retries (default **10**)\n  * `factor`: factor for retry (default **2**)\n  * `minTimeout`: minimum time for first retry (default **1000**)\n  * `maxTimeout`: maximum time for all retries (default **60000**)\n\n```js\nvar client = require('alamo').createClient({\n\tkey: process.env.AWS_ACCESS_KEY_ID,\n\tsecret: process.env.AWS_SECRET_ACCESS_KEY,\n\tbucket: process.env.AWS_BUCKET,\n\tregion: process.env.AWS_REGION\n});\n```\n\n### Client.prototype.client\n\n  Access to the lower level **knox** client\n\n### Client.prototype.createReadStream(filename, headers)\n\n  Returns a readable stream.\n\n```js\nvar fs = require('fs');\n\nclient.createReadStream('/somekey').pipe(fs.createWriteStream('somefile'));\n```\n\n* `filename`: the s3 file name to retrieve\n* `headers`: optional headers\n* `options`: retry options\n\nAlias: `readStream`\n\n### Client.prototype.createWriteStream(filename, headers, content)\n\n  Returns a writable upload stream. You can optionally pass a buffer to upload instead of piping to it.\n\n```js\nvar fs = require('fs');\nvar ws = client.createWriteStream('/somekey');\n\nfs.createReadStream('somefile')\n\t.pipe(ws)\n\t.on('error', console.error)\n\t.on('finish', console.log.bind(console, 'Upload complete'));\n```\n\n* `filename`: the s3 file name to upload to\n* `headers`: optional headers\n* `content`: optional content to upload. If content is passed, it is passed to the underlying `request.end`\n* `options`: retry options\n\nAlias: `writeStream`\n\n### Client.prototype.stream(method, filename, headers, content)\n\n  Generic stream implementation that accepts the method as 1st argument as 2nd argument\n\n```js\nvar fs = require('fs');\nvar ws = client.stream('PUT', '/somekey');\n\nfs.createReadStream('somefile')\n\t.pipe(ws)\n\t.on('error', console.error)\n\t.on('finish', console.log.bind(console, 'Upload complete'));\n```\n\n* `method`: the http method e.g. `GET`, `PUT`\n* `filename`: the s3 file name to upload to\n* `headers`: optional headers\n* `content`: optional content to upload. If content is passed, it is passed to the underlying `request.end`\n* `options`: retry options\n\n### Client.prototype.get(filename, headers, cb)\n\n  Get an object and retrieve the response with the body\n\n```js\nclient.get('/somekey', function (err, res) {\n  if (err) console.error(err);\n  else console.log(res.statusCode, res.body.toString());\n});\n```\n\n* `filename`: the s3 file name to retrieve\n* `headers`: optional headers\n* `options`: retry options\n* `cb`: callback that returns an error or `null` as 1st argument, and the response\nwith the body if no error as 2nd argument\n\n### Client.prototype.del(filename, headers, cb)\n\n  Delete an object from s3\n\n```js\nclient.del('/somekey', function (err, res) {\n  if (err) console.error(err);\n  else console.log('object deleted %d', res.statusCode);\n});\n```\n\n* `filename`: the s3 file name to delete\n* `headers`: optional headers\n* `options`: retry options\n* `cb`: callback that returns an error or `null` as 1st argument, and the response if no error as 2nd argument\n\n### Client.prototype.put(filename, content, headers, cb)\n\n  Put an object\n\n```js\nclient.put('/somekey', 'somedata', function (err, res) {\n  if (err) console.error(err);\n  else console.log('object uploaded %d', res.statusCode);\n});\n```\n\n* `filename`: the s3 file name to upload to\n* `content`: content to upload\n* `headers`: optional headers\n* `options`: retry options\n* `cb`: callback that returns an error or `null` as 1st argument, and the response\nif no error as 2nd argument\n\n### Client.prototype.post(filename, content, headers, cb)\n\n  Post an object\n\n```js\nclient.post('/somekey', 'somedata', function (err, res) {\n  if (err) console.error(err);\n  else console.log('object uploaded %d with etag %s', res.statusCode, res.headers.etag);\n});\n```\n\n* `filename`: the s3 file name to post to\n* `content`: content to post\n* `headers`: optional headers\n* `options`: retry options\n* `cb`: callback that returns an error or `null` as 1st argument, and the response\nif no error\n\n### Client.prototype.request(method, filename, content, headers, cb)\n\n  Generic non streaming interface\n\n```js\nclient.request('PUT', '/somekey', 'somedata', function (err, res) {\n  if (err) console.error(err);\n  else console.log('object uploaded %d with etag %s', res.statusCode, res.headers.etag);\n});\n```\n\n* `method`: the http method e.g. `GET`, `PUT`, `DELETE`, `POST`\n* `filename`: the s3 file name\n* `content`: content to post\n* `headers`: optional headers\n* `cb`: callback that returns an error or `null` as 1st argument, and the response with the body if no error\n\n### Client.prototype.signedUrl(filename, expiration, options)\n\n  Returns a signed url\n\n```js\nvar url = client.signedUrl('/somekey', 1000 * 60 * 15);\nconsole.log(url);\n```\n\n* `filename`: the s3 file name to retrieve\n* `expiration`: number of milliseconds that the signed url is valid for\n* `options`: signed url options passed to **knox**, take `verb`, `contentType`, and `qs` object\n\n### Client.prototype.multipart(filename, headers)\n\n  Returns a writable stream to upload using the s3 multipart API. The stream is uploaded by chunks of 5mb in parallel\n  with max concurrent uploads and automatic retries\n\n```js\nvar fs = require('fs');\nvar ws = client.multipart('/somekey');\n\nfs.createReadStream('somefile')\n\t.pipe(ws)\n\t.on('error', console.error)\n\t.on('finish', console.log.bind(console, 'Upload complete'));\n```\n\n* `filename`: the s3 file name to upload to\n* `headers`: optional headers\n* `options`: retry options\n\n## Comparison with knox\n\n### Retrieve a stream with **knox** with full error handling\n\n```js\nvar fs = require('fs');\nvar XML = require('simple-xml');\nvar req = client.get('/filename');\nreq.on('response', function (res) {\n  if (res.statusCode !== 200) {\n  \tvar body = '';\n  \tres.on('data', function (chunk) {\n  \t\tbody += chunk.toString();\n  \t});\n\tres.on('end', function () {\n\t\ttry {\n\t\t\tbody = XML.parse(body);\n\t\t\tcb(new Error(body.message);\n\t\t} catch (err) {\n\t\t\tcb(new Error(body);\n\t\t}\n\t\tcb(null, res);\n\t});\n  \tres.on('error', cb);\n  \treturn;\n  }\n  res.pipe(fs.createWriteStream('filename').on('error', cb).on('finish', cb.bind(null, null));\n});\nreq.on('error', cb);\n```\n\n## Roadmap\n\n* Handle redirects and other 30x status codes: http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html\n* Implement global max concurrent uploads\n* Implement basic progress for multipart upload\n* Accept string value for expiration that can be parsed by [ms](https://github.com/rauchg/ms.js)\n* Add higher level functions for \"file\" upload / download\n* Maybe use multipart upload automatically if content-length is unknown?\n* Maybe allow automatic handling (parsing, marshalling) of json?\n\n## Contributions\n\nPlease open issues for bugs and suggestions in [github](https://github.com/jtblin/alamo/issues).\nPull requests with tests are welcome.\n\n## Author\n\nJerome Touffe-Blin, [@jtblin](https://twitter.com/jtlbin), [About me](http://about.me/jtblin)\n\n## License\n\nalamo is copyright 2015 Jerome Touffe-Blin and contributors. It is licensed under the BSD license.\nSee the include LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjtblin%2Falamo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjtblin%2Falamo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjtblin%2Falamo/lists"}