{"id":13625074,"url":"https://github.com/chrisvxd/combine-pagination","last_synced_at":"2025-03-17T12:11:01.196Z","repository":{"id":34297507,"uuid":"175102155","full_name":"chrisvxd/combine-pagination","owner":"chrisvxd","description":"A JavaScript library for paginating data from multiple sources 🦑","archived":false,"fork":false,"pushed_at":"2023-01-03T17:40:32.000Z","size":1211,"stargazers_count":172,"open_issues_count":22,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-20T15:50:10.946Z","etag":null,"topics":["algolia","graphql","javascript","nodejs","pagination","pagination-library","paging"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrisvxd.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-11T23:57:17.000Z","updated_at":"2024-10-21T04:32:13.000Z","dependencies_parsed_at":"2023-01-15T06:02:41.784Z","dependency_job_id":null,"html_url":"https://github.com/chrisvxd/combine-pagination","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisvxd%2Fcombine-pagination","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisvxd%2Fcombine-pagination/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisvxd%2Fcombine-pagination/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisvxd%2Fcombine-pagination/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrisvxd","download_url":"https://codeload.github.com/chrisvxd/combine-pagination/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244031033,"owners_count":20386534,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["algolia","graphql","javascript","nodejs","pagination","pagination-library","paging"],"created_at":"2024-08-01T21:01:50.461Z","updated_at":"2025-03-17T12:11:01.176Z","avatar_url":"https://github.com/chrisvxd.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# combine-pagination 🦑\n\n[![NPM](https://img.shields.io/npm/v/combine-pagination.svg)](https://www.npmjs.com/package/combine-pagination) [![JavaScript Style Guide](https://img.shields.io/badge/code_style-prettier-brightgreen.svg)](https://prettier.io)\n\n`combine-pagination` is a JavaScript library for paginating across multiple data sources at once, whilst retaining the sort order.\n\n- **Great for Infinity Scroll**: easily support multiple data sources in your infinity scroll.\n- **Retain order**: your data is always in order, even when it comes from different sources.\n- **Service agnostic**: work with any data service, whether REST, GraphQL, Algolia or another.\n- **Mix-and-match services**: mix data services as you see fit, making one query from GraphQL and one from Algolia.\n- **Efficient**: only fetch data when needed for that data source.\n\nUsed in production at https://wellpaid.io.\n\n## Installation\n\n```sh\nnpm i combine-pagination --save\n```\n\nor\n\n```sh\nyarn add combine-pagination\n```\n\n## Quick examples\n\nIf you already understand [the problem](#the-problem), here are some quick examples for paginating across multiple data sets in different scenarios.\n\n### Generic\n\nPaginate data from two generic data sets.\n\n```js\nconst combinedGetters = combinePagination({\n  getters: [page =\u003e getDataA(page), page =\u003e getDataB(page)],\n  sortKey: \"popularity\"\n});\n\nconst pageOne = await combinedGetters.getNext();\nconst pageTwo = await combinedGetters.getNext();\n```\n\n### Algolia - custom sorting\n\nPaginate data from two distinct algolia queries where the Algolia Index is sorted by a custom field. Each query is using a different keyword.\n\n```js\nconst index = algoliasearch({\n  //...\n}).initIndex(\"hats\");\n\nconst combinedGetters = combinePagination({\n  getters: [\n    async page =\u003e\n      (await index.query({ page, hitsPerPage: 15, query: \"Baseball cap\" }))\n        .hits,\n    async page =\u003e\n      (await index.query({ page, hitsPerPage: 15, query: \"Top hat\" })).hits\n  ],\n  sortKey: \"popularity\"\n});\n\nconst pageOne = await combinedGetters.getNext();\nconst pageTwo = await combinedGetters.getNext();\n```\n\n### Algolia - default sorting\n\nPaginate data from two distinct algolia queries where the Algolia Index is sorted using Algolia's default criteria.\n\nThis approach uses a custom `sort` method that attempts to match [Algolia's sorting algorithm](https://www.algolia.com/doc/guides/managing-results/must-do/custom-ranking/?language=javascript#the-ranking-criteria).\n\n\u003e The `sortAlgolia` sort method used in this example is experimental. You might need to implement your own if using a custom ranking method.\n\n```js\nimport { sortAlgolia } from \"combine-pagination\";\n\nconst index = algoliasearch({\n  getRankingInfo: true // Ask algolia for ranking info\n  //...\n}).initIndex(\"hats\");\n\nconst combinedGetters = combinePagination({\n  getters: [\n    async page =\u003e\n      (await index.query({ page, hitsPerPage: 15, query: \"Baseball cap\" }))\n        .hits,\n    async page =\u003e\n      (await index.query({ page, hitsPerPage: 15, query: \"Top hat\" })).hits\n  ],\n  sort: sortAlgolia\n});\n```\n\n## The Problem\n\nSuppose you have two data sets, `modernHats` and `oldHats`, and you want to combine them into one data set sorted by `popularity` called `hats`:\n\n```js\nconst modernHats = [\n  {\n    name: \"Baseball Cap\",\n    popularity: 95\n  },\n  {\n    name: \"Beanie\",\n    popularity: 50\n  },\n  {\n    name: \"Flat Cap\",\n    popularity: 20\n  }\n];\n\nconst oldHats = [\n  {\n    name: \"Top Hat\",\n    popularity: 85\n  },\n  {\n    name: \"Beret\",\n    popularity: 15\n  },\n  {\n    name: \"Bowler Hat\",\n    popularity: 9\n  }\n];\n```\n\nIn this example, we’ll be paginating 2 results at a time. Let’s create a getter to paginate our data:\n\n```js\nconst getData = (data, page) =\u003e data.slice((page - 1) * 2, page * 2);\n```\n\n\u003e Note, in reality you probably already have a data getting with pagination support to retrieve the data via a server.\n\nNow let's get our data. Without using `combine-pagination`, we might be tempted to just paginate each, sort and combine them. **This is what NOT to do:**\n\n```js\nconst modernHatsPage = getData(modernHats, 0);\nconst oldHatsPage = getData(oldHats, 0);\n\nconst hats = [...modernHatsPage, ...oldHatsPage].sort(\n  (a, b) =\u003e a.popularity - b.popularity\n);\n```\n\nThis will result in hats that looks like this\n\n```js\n[\n  {\n    name: \"Baseball Cap\",\n    popularity: 95\n  },\n  {\n    name: \"Top Hat\",\n    popularity: 85\n  },\n  {\n    name: \"Beanie\",\n    popularity: 50\n  },\n  {\n    name: \"Beret\",\n    popularity: 15\n  }\n];\n```\n\nThis looks fine, until you query the second page, which will look like this\n\n```js\n[\n  {\n    name: \"Flat Cap\",\n    popularity: 20\n  },\n  {\n    name: \"Bowler Hat\",\n    popularity: 9\n  }\n];\n```\n\nIf we combine these results, you’ll notice that now the **results are out of order**. Sure, we could re-sort our entire data set, but this has some problems:\n\n1. Reordering UI is confusing - if we’re rendering `hats` in a UI, such as an infinity scroll, it will cause the UI to reorder and confuse the user.\n2. Inefficient sort - re-sorting the entire data set on each pagination is highly inefficient.\n3. Unnecessary data request - depending on the order of the data, getting both data sets at once might be unnecessary, especially if a network request is involved.\n\n## The Solution\n\nUsing a technique (currently) called [Framed Range Intersection](#framed-range-intersection), we can conservatively hold back trailing data from the first page that we think might overlap with subsequent pages. In the example above, it would mean holding back \"Beret\" until the next page is retrieved.\n\n`combine-pagination` implements this technique. Let's try again using the above data set:\n\n```js\nimport combinePagination from \"combine-pagination\";\n\nconst combinedGetters = combinePagination({\n  getters: [page =\u003e getData(modernHats, page), page =\u003e getData(oldHats, page)],\n  sortKey: \"popularity\"\n});\n```\n\nAnd query the first page:\n\n```js\nconst page = await combinedGetters.getNext();\n```\n\nResulting in:\n\n```js\n[\n  {\n    name: \"Baseball Cap\",\n    popularity: 95\n  },\n  {\n    name: \"Top Hat\",\n    popularity: 85\n  },\n  {\n    name: \"Beanie\",\n    popularity: 50\n  }\n];\n```\n\nAs expected, we only received three results. `combine-pagination` is only showing intersecting data, holding \"Beret\" back until it receives the next data set. Because of this, you can't define exactly how many results you want to receive. See [Fuzzy Pagination](#fuzzy-pagination).\n\nThe second time we run `getNext()`, we get the next set of data, but this time in the correct order:\n\n```js\n[\n  {\n    name: \"Flat Cap\",\n    popularity: 20\n  },\n  {\n    name: \"Beret\",\n    popularity: 15\n  },\n  {\n    name: \"Bowler Hat\",\n    popularity: 9\n  }\n];\n```\n\n`combine-pagination` noticed that \"Beret\", which was held back from the first set of results, intersects \"Flat Cap\" and \"Bowler Hat\", so has inserted it and sorted the page.\n\nThat's it. Each time you call `getNext()`, you'll retreive the next set of sorted data until the getters are exhausted.\n\n## Use cases\n\n- Using infinity scroll across multiple data sources.\n- Paginating across multiple Algolia queries, such as one geo location query and one not.\n- Paginating across different services.\n\n## Framed Range Intersecting\n\nIntersecting ranges is a technique for finding values that overlap in two sets of data. For example:\n\n- Intersection of [0, 3] \u0026 [2, 4] is [2, 3]\n- Intersection of [-1, 34] \u0026 [0, 4] is [0, 4]\n- Intersection of [0, 3] \u0026 [4, 4] is empty set\n\n`combine-pagination` uses a technique called Framed Range Intersecting (name is WIP), a type of intersecting that determines the leading data set, and intersects the other data sets within that.\n\nUnlike normal range intersecting:\n\n- The first value is the first value of the leading data set.\n- The last value is either the last value of the leading data set, or the the last value of the intersecting data set that finishes first.\n- Values in multiple data sets are duplicated.\n\nFor example:\n\n- Framed Intersection of [0, 3] \u0026 [2, 4] is [0, 2, 3]\n- Framed Intersection of [-1, 34] \u0026 [0, 4] is [-1, 0, 4]\n- Framed Intersection of [0, 3] \u0026 [4, 4] is [0, 3]\n- Framed Intersection of [0, 3] \u0026 [2, 4] \u0026 [1, 2] is [0, 1, 2, 2]\n\n## Fuzzy Pagination\n\nEach time you execute `getNext()`, you can't be sure how many results you're going to receive. We call this **Fuzzy Pagination**, which returns `0 - n` results for any given page with page size `n`. This technique is best suited for infinity scroll type use cases.\n\nIn normal pagination, you would receive `n` results for each page, only receiving `0 - n` results on the final page.\n\n## License\n\nMIT © [Chris Villa](http://www.chrisvilla.co.uk)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisvxd%2Fcombine-pagination","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrisvxd%2Fcombine-pagination","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisvxd%2Fcombine-pagination/lists"}