{"id":20643634,"url":"https://github.com/extrabacon/rest-collection-stream","last_synced_at":"2025-04-16T02:05:39.585Z","repository":{"id":16602579,"uuid":"19357172","full_name":"extrabacon/rest-collection-stream","owner":"extrabacon","description":"REST client for streaming collections and lists","archived":false,"fork":false,"pushed_at":"2017-06-30T17:44:00.000Z","size":132,"stargazers_count":25,"open_issues_count":3,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-16T02:05:22.801Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/extrabacon.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-05-01T20:38:39.000Z","updated_at":"2021-02-05T18:58:58.000Z","dependencies_parsed_at":"2022-09-26T20:52:45.145Z","dependency_job_id":null,"html_url":"https://github.com/extrabacon/rest-collection-stream","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extrabacon%2Frest-collection-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extrabacon%2Frest-collection-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extrabacon%2Frest-collection-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extrabacon%2Frest-collection-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/extrabacon","download_url":"https://codeload.github.com/extrabacon/rest-collection-stream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249183103,"owners_count":21226141,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T16:13:28.761Z","updated_at":"2025-04-16T02:05:39.568Z","avatar_url":"https://github.com/extrabacon.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rest-collection-stream\n\nA simple HTTP client for streaming JSON content from REST lists and collections. Includes extensive pagination support for continuous streaming across multiple HTTP requests.\n\nUse this module to pull an entire collection of JSON documents from your favorite API into a Node.js stream! It will parse the response as JSON, extract content as objects, push it down the stream, and continue doing so as long as there is more data.\n\n```js\nvar restCollection = require('rest-collection-stream');\nrestCollection('https://somecloudapp.io/api/v1/resource').pipe(/*somewhere*/);\n```\n\n## Installation\n\n```bash\nnpm install rest-collection-stream\n```\n\n## Documentation\n\nThe module consists of a function returning a stream.\n\nIt's that simple.\n\nPulling data from an API is much more fun with streaming! Especially when paging is handled automatically...\n\n```js\nvar restCollection = require('rest-collection-stream');\nvar fs = require('fs');\nvar es = require('event-stream');\n\n// export all customers into a backup file\nrestCollection('https://somecloudapp.io/api/v1/customers')\n  .pipe(es.map(/* my mapping function */))\n  .pipe(es.stringify())\n  .pipe(fs.createWriteStream(backupFile));\n```\n\nMore precisely, the function has the same signature as [request](https://github.com/mikeal/request), accepting the same [parameters and options](https://github.com/mikeal/request/blob/master/README.md#requestoptions-callback), but returning a stream instead of invoking a callback.\n\n```js\nvar stream = restCollection(url, options);\n```\n\nIn addition to request options, you may also set:\n- `data` - a function for extracting content from the response\n- `next` - a function for handling pagination\n\nThe following options are also enforced:\n- `json` - always **true**, only JSON APIs are supported at the moment and the response body is always parsed as such\n- `headers` - HTTP compression is enabled by default via `Accept-Encoding` - a nicety if you are pulling thousands of documents from the API servers\n\n### Streaming objects from the response\n\nBy default, if the response body consists of an array, all elements are emitted down the stream.\n\nIf the response is an envelope object, commonly used properties will be inspected for an array of data. Those are (in order): `data`, `results`, `items`, `value` and `objects`.\n\n#### Custom parsing\n\nIf the elements you want to stream are not found in those common properties, you can supply your own function to extract the elements. Your function is expected to return an array of items to emit down the stream.\n\nTo use a custom function for parsing the body, specify it as `data` in the options.\n\n```js\nrestCollection(url, {\n  data: function (res, body) {\n    return body.items;\n  }\n}).pipe(...);\n```\n\n#### Custom streaming\n\nYou can also gain full control over what gets emitted by not returning values from your `data` function. Use `this.emit('data', item)` to send whatever you want, including results from an asynchronous function.\n\n```js\nrestCollection(url, {\n  data: function (res, body) {\n    this.emit('data', { something: 'else' });\n  }\n}).pipe(...);\n```\n\n### Pagination\n\nAnother request for the next page of data is automatically sent if there is more data available. The module implements common pagination support from popular APIs, but you can also supply your own.\n\n#### Built-in pagination support\n\n- Standard HTTP header `Link: \u003cURL\u003e; rel='next'`\n- Common envelope properties\n    + Facebook: `paging.next`\n    + Google APIs: `nextLink`\n    + OData 2.0 and 3.0: `__next`\n    + OData 4.0: `@odata.nextLink`\n    + AWS: `NextMarker`\n- A self-incrementing `page` query string parameter\n\nShould I support more APIs? Pull requests are welcome.\n\n#### Custom paging\n\nIf your API works differently, you can provide you own paging function. The function can return the URL of the next page or query string parameters to apply on the next request. To stop pagination, return something falsy.\n\nTo use a custom function for pagination, specify it as `next` in the options.\n\n```js\nrestCollection(url, {\n  next: function (res, body) {\n    return urlOfNextPage;\n  }\n}).pipe(...);\n```\n\nTo specify the URL of the next page to request, return it as a string. The URL is automatically resolved using [url.resolve](http://nodejs.org/api/url.html#url_url_resolve_from_to).\n\nTo specify query string parameters instead, return an object. The key/value pairs will be applied on the next request. The current query string values can be accessed in `this.req.qs`.\n\n```js\nrestCollection(url, {\n  next: function (res, body) {\n    // increment the page manually (we should also have a stop condition...)\n    return { page: this.req.qs.page + 1 };\n  }\n}).pipe(...);\n```\n\nTo use anything else, such as request headers, you may also access the request parameters using `this.req`. Any modification will be used onward for the next request. Make sure to return the URL, otherwise paging will stop.\n\n```js\nrestCollection(url, {\n  next: function (res, body) {\n    // return the same URL - paging is controlled using headers instead\n    this.req.headers['X-Page-Index'] = nextPage;\n    return this.req.uri.href;\n  }\n}).pipe(...);\n```\n\n## Dependencies\n\n+ [request](https://github.com/mikeal/request)\n+ [event-stream](https://github.com/dominictarr/event-stream)\n\n## License\n\nThe MIT License (MIT)\n\nCopyright (c) 2014, Nicolas Mercier\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fextrabacon%2Frest-collection-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fextrabacon%2Frest-collection-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fextrabacon%2Frest-collection-stream/lists"}