{"id":15694005,"url":"https://github.com/jonschlinkert/paged-request","last_synced_at":"2025-05-07T21:45:20.667Z","repository":{"id":45062738,"uuid":"119662637","full_name":"jonschlinkert/paged-request","owner":"jonschlinkert","description":"Simplified requests for paged (paginated) content.","archived":false,"fork":false,"pushed_at":"2022-01-11T17:45:59.000Z","size":21,"stargazers_count":7,"open_issues_count":1,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-04T05:02:24.935Z","etag":null,"topics":["get","http","needle","paged","pages","paginated","request"],"latest_commit_sha":null,"homepage":"https://github.com/jonschlinkert","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonschlinkert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-31T09:08:23.000Z","updated_at":"2023-07-30T17:00:02.000Z","dependencies_parsed_at":"2022-09-09T14:21:30.746Z","dependency_job_id":null,"html_url":"https://github.com/jonschlinkert/paged-request","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonschlinkert%2Fpaged-request","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonschlinkert%2Fpaged-request/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonschlinkert%2Fpaged-request/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonschlinkert%2Fpaged-request/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonschlinkert","download_url":"https://codeload.github.com/jonschlinkert/paged-request/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252961843,"owners_count":21832191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["get","http","needle","paged","pages","paginated","request"],"created_at":"2024-10-03T18:51:00.756Z","updated_at":"2025-05-07T21:45:20.645Z","avatar_url":"https://github.com/jonschlinkert.png","language":"JavaScript","readme":"# paged-request [![NPM version](https://img.shields.io/npm/v/paged-request.svg?style=flat)](https://www.npmjs.com/package/paged-request) [![NPM monthly downloads](https://img.shields.io/npm/dm/paged-request.svg?style=flat)](https://npmjs.org/package/paged-request) [![NPM total downloads](https://img.shields.io/npm/dt/paged-request.svg?style=flat)](https://npmjs.org/package/paged-request) [![Linux Build Status](https://img.shields.io/travis/jonschlinkert/paged-request.svg?style=flat\u0026label=Travis)](https://travis-ci.org/jonschlinkert/paged-request)\n\n\u003e Simplified requests for paged (paginated) content.\n\nPlease consider following this project's author, [Jon Schlinkert](https://github.com/jonschlinkert), and consider starring the project to show your :heart: and support.\n\n## Install\n\nInstall with [npm](https://www.npmjs.com/):\n\n```sh\n$ npm install --save paged-request\n```\n\n## Heads up!\n\nSee the [release notes](#release-notes) for information about changes made in v2.0.\n\n## Usage\n\nThis library recursively calls [needle's](https://github.com/tomas/needle#needlemethod-url-data-options-callback--20x) `.get` method as long as the user-provided `next()` function returns a string (the next url to get). See [an example](#example).\n\n**Example**\n\n```js\nconst request = require('paged-request');\n\nrequest(url, options, next)\n  .then(acc =\u003e console.log(acc.pages.length))\n  .catch(console.error);\n```\n\n### Params\n\n* `url` **{string}** - (required) the initial url to get\n* `options` **{object}** - (optional) options object to pass to [needle](https://github.com/tomas/needle)\n* `next` **{function}** - (required) function that returns the next url to get, a promise or undefined.\n\n### `next` function params\n\n* `url` **{string}** - the original (base) user-provided url\n* `resp` **{object}** - [needle](https://github.com/tomas/needle) response object\n* `acc` **{object}** - accumulator object with the following properties:\n  - `options` **{object}** - user-provided options object\n  - `pages` **{array}** - array of responses\n  - `urls` **{array}** - array of requested urls\n\nThe `next` function should return a string (the next url to get), promise or undefined.\n\n## Example\n\nThe following example shows how to loop over pages of `CSS` posts on [smashingmagazine.com](https://www.smashingmagazine.com/category/css) (an arbitrary example, but they have great content!).\n\n```js\nconst request = require('paged-request');\n\nasync function next(url, resp, acc) {\n  // do stuff to check response first if necessary\n  const regex = /href=\"\\/.*?\\/(\\d+)\\/\"/;\n  const num = (regex.exec(resp.data) || [])[1];\n\n  if (num \u0026\u0026 /^[0-9]+$/.test(num) \u0026\u0026 +num \u003c= n) {\n    // use the \"original\" url to avoid having to reparse\n    // and recreate the url each time\n    return `${acc.orig}/page/${num}/`;\n  }\n}\n\nrequest('https://www.smashingmagazine.com/category/css', {}, next)\n  .then(acc =\u003e console.log(acc.pages.length))\n  .catch(console.error);\n```\n\n## Release notes\n\n### v2.0\n\n* renamed `.hrefs` to `.urls` in response object\n* now using [axios](https://github.com/axios/axios) instead of [needle](https://github.com/tomas/needle). Please see the axios documentation for API information.\n\n## About\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eContributing\u003c/strong\u003e\u003c/summary\u003e\n\nPull requests and stars are always welcome. For bugs and feature requests, [please create an issue](../../issues/new).\n\nPlease read the [contributing guide](.github/contributing.md) for advice on opening issues, pull requests, and coding standards.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eRunning Tests\u003c/strong\u003e\u003c/summary\u003e\n\nRunning and reviewing unit tests is a great way to get familiarized with a library and its API. You can install dependencies and run tests with the following command:\n\n```sh\n$ npm install \u0026\u0026 npm test\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eBuilding docs\u003c/strong\u003e\u003c/summary\u003e\n\n_(This project's readme.md is generated by [verb](https://github.com/verbose/verb-generate-readme), please don't edit the readme directly. Any changes to the readme must be made in the [.verb.md](.verb.md) readme template.)_\n\nTo generate the readme, run the following command:\n\n```sh\n$ npm install -g verbose/verb#dev verb-generate-readme \u0026\u0026 verb\n```\n\n\u003c/details\u003e\n\n### Related projects\n\nYou might also be interested in these projects:\n\n* [gists](https://www.npmjs.com/package/gists): Methods for working with the GitHub Gist API. Node.js/JavaScript | [homepage](https://github.com/jonschlinkert/gists \"Methods for working with the GitHub Gist API. Node.js/JavaScript\")\n* [github-base](https://www.npmjs.com/package/github-base): Low-level methods for working with the GitHub API in node.js/JavaScript. | [homepage](https://github.com/jonschlinkert/github-base \"Low-level methods for working with the GitHub API in node.js/JavaScript.\")\n* [repos](https://www.npmjs.com/package/repos): Tiny wrapper around github-base for getting publicly available information for a repository, or all of… [more](https://github.com/jonschlinkert/repos) | [homepage](https://github.com/jonschlinkert/repos \"Tiny wrapper around github-base for getting publicly available information for a repository, or all of the repositories for one or more users or orgs, from the GitHub API.\")\n\n### Contributors\n\n| **Commits** | **Contributor** |  \n| --- | --- |  \n| 12 | [jonschlinkert](https://github.com/jonschlinkert) |  \n| 9  | [doowb](https://github.com/doowb) |  \n| 2  | [whitneyit](https://github.com/whitneyit) |  \n\n### Author\n\n**Jon Schlinkert**\n\n* [GitHub Profile](https://github.com/jonschlinkert)\n* [Twitter Profile](https://twitter.com/jonschlinkert)\n* [LinkedIn Profile](https://linkedin.com/in/jonschlinkert)\n\n### License\n\nCopyright © 2021, [Jon Schlinkert](https://github.com/jonschlinkert).\nReleased under the [MIT License](LICENSE).\n\n***\n\n_This file was generated by [verb-generate-readme](https://github.com/verbose/verb-generate-readme), v0.8.0, on January 20, 2021._","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonschlinkert%2Fpaged-request","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonschlinkert%2Fpaged-request","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonschlinkert%2Fpaged-request/lists"}