{"id":18901238,"url":"https://github.com/regular/pull-paginated-api-request","last_synced_at":"2025-10-22T10:52:18.124Z","repository":{"id":65371448,"uuid":"42066204","full_name":"regular/pull-paginated-api-request","owner":"regular","description":"Use this pull-stream to lazily traverse a paginated (REST) API!","archived":false,"fork":false,"pushed_at":"2015-10-18T12:21:52.000Z","size":152,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-10-09T22:19:27.943Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/regular.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-09-07T17:32:43.000Z","updated_at":"2017-10-24T23:03:45.000Z","dependencies_parsed_at":"2023-01-19T22:35:14.700Z","dependency_job_id":null,"html_url":"https://github.com/regular/pull-paginated-api-request","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/regular/pull-paginated-api-request","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/regular%2Fpull-paginated-api-request","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/regular%2Fpull-paginated-api-request/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/regular%2Fpull-paginated-api-request/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/regular%2Fpull-paginated-api-request/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/regular","download_url":"https://codeload.github.com/regular/pull-paginated-api-request/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/regular%2Fpull-paginated-api-request/sbom","scorecard":{"id":769081,"data":{"date":"2025-08-11","repo":{"name":"github.com/regular/pull-paginated-api-request","commit":"d34dc38b74c464db4ef581d5a9f9c12dba797178"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.6,"checks":[{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Code-Review","score":0,"reason":"Found 0/11 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":0,"reason":"license file not detected","details":["Warn: project does not have a license file"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-23T01:44:59.877Z","repository_id":65371448,"created_at":"2025-08-23T01:44:59.877Z","updated_at":"2025-08-23T01:44:59.877Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280424214,"owners_count":26328462,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-22T02:00:06.515Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T08:55:33.875Z","updated_at":"2025-10-22T10:52:18.096Z","avatar_url":"https://github.com/regular.png","language":"JavaScript","readme":"# pull-paginated-api-request\nUse this pull-stream to lazily traverse a paginated (REST) API\n![![NPM](https://nodei.co/npm/pull-paginated-api-request.png)](https://nodei.co/npm/pull-paginated-api-request/)\n\n## Simple Usage\n\n```js\n    var pull = require('pull-stream');\n    var paginated = require('pull-paginated-api-request');\n    ver querystring = require('querystring');\n    var extend = require('xtend');\n    var hyperquest = require('hyperquest')\n\n    var request = paginated(function(o, pageToken) {\n        return hyperquest(o.endpoint + '?' + querystring.stringify(\n            extend({o.query, pageToken ? {pageToken: pageToken} : {})\n        ));\n    });\n\n    pull(\n        request({\n            endpoint: 'https://api.whatever.com/playlistItems',\n            query: {\n                playlistId: id,\n                maxResults: 50\n            }\n        },\n        [\n            ['items', true], \n            ['nextPageToken']\n        ]),\n        pull.log()\n    );\n\n    // outputs all playlist items until a REST API response\n    // does not contain a nextPageToken proerty\n```\n\nSee [regular/liked-on-youtube](https://github.com/regular/liked-on-youtube) for a real-life example.\n\n## API\n\n### paginated(makeResponseStream)\n\nReturns a request function that can be used to query the API.\nTakes a factory function that provides the response of an API server as a Node-style ReadableStream. The arguments of the factory-function are:\n\n- makeResponseStream(obj, pageToken)\n    \n    - obj - an ojbect (opaque to pull-paginated-api-request) (see below)\n    - pageToken: the token of the result page to request (see below)\n    - please return a Node-style ReadStream. (Hint: use hyperquest, it works in Node and in the browser)\n\n## request(obj, [itemPathway, nextPageTokenPathway])\n\nThis is the function retruned by `paginated` (see above). The first argument `obj` will simply be forwarded to `makeResponseStream`. The second argument is an array of `pathways`. They describe where to find the items (the stuff that we are interessted in) and the token of the next page of API results. The pathways themselves are arrays. They describe the location of those two properties within the JSON-formatted response of the API server. See [pathway](https://github.com/substack/node-pathway) for details.\n\n`request` returns a pull-stream source. It generates output only when that output is consumed. This means that only those result pages get requested from the server that actually are needed. See [Dominic Tarr's pull-stream](https://github.com/dominictarr/pull-stream) to learn more about pull-streams and the power of laziness.\n\n## Advanced Usage\n\nInstead of specifying an array of pathways, you can provide your own function to extract properties from the server response. This is also how you extract a pageToken from the HTTP headers (used by GitHub for example). See [test.js](https://github.com/regular/pull-paginated-api-request/blob/master/test.js#L89) for details.\n\n## License\nMIT\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fregular%2Fpull-paginated-api-request","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fregular%2Fpull-paginated-api-request","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fregular%2Fpull-paginated-api-request/lists"}