{"id":16858965,"url":"https://github.com/ross/performant-pagination","last_synced_at":"2025-03-22T06:31:21.411Z","repository":{"id":66454361,"uuid":"13721117","full_name":"ross/performant-pagination","owner":"ross","description":"High Performance Python Django pagination","archived":false,"fork":false,"pushed_at":"2020-02-08T16:16:29.000Z","size":25,"stargazers_count":36,"open_issues_count":4,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-15T19:37:58.101Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ross.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-10-20T15:13:11.000Z","updated_at":"2024-11-28T16:28:07.000Z","dependencies_parsed_at":"2023-02-20T20:00:22.434Z","dependency_job_id":null,"html_url":"https://github.com/ross/performant-pagination","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ross%2Fperformant-pagination","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ross%2Fperformant-pagination/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ross%2Fperformant-pagination/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ross%2Fperformant-pagination/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ross","download_url":"https://codeload.github.com/ross/performant-pagination/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244918500,"owners_count":20531682,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T14:15:45.569Z","updated_at":"2025-03-22T06:31:18.870Z","avatar_url":"https://github.com/ross.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"performant-pagination\n=====================\n\n**High Performance Python Django pagination**\n\n# Overview\n\nPerformant Pagination is a (mostly) drop-in replacement for Django's built-in\nPaginator which avoids problems with performance and semantics on large and/or\nfast changing data sets.\n\n# Why It's Needed\n\nWith large datasets the counting of rows to get to subsequent pages, even when\nindexed, can be expensive. When asking for 10 rows starting at the 1000th many\ndatabase engines will need to scan over the first 1000 to return 1001 and on.\n1000 isn't that many rows, but if it's 1M, 10M, ...\n\nIf you dataset is changing fast, especially faster than your walking it, you\ncan end up backtracking, jumping forward, or standing still. You may see some\nelements multiple times and basically your walking order is undefined and\nnon-repeatable. If you ask for 1000-1009 and while you're working on them 10\nnew rows are inserted prior to 1000, asking for 1010-1019 will return the same\n10 rows again. If 10 are removed, you'll jump over what would have been the\nnext items. If 20 are inserted, you'll backtrack and get what was previously\n990-999.\n\n# How It Works\n\nRather than using offset and limit (count) to paginate, PerformantPaginator\nuses an opaque **token** to keep track of place. Under the hood these tokens\nprovide information about the last item returned on the previous page and\nallows the next page to pick up where the previous left off. \n\nProvided you ensure the **ordering** columns are correctly indexed and will\nidentify unique elements the queries run by PerformantPaginator will be able to\nutilize the index to walk directly to the starting point and return the next N\nitems.\n\nA traversal using PerformantPaginator is guaranteed not to re-visit items\n(provided they haven't changed place) though it will not see items that were\ninserted after it passed a given point in the dataset. Overall the traversal is\ndeterministic and predictable.\n\n# What You Give Up\n\n* PerformantPaginator.count is disabled by default (returns None) it's \n  expensive to take the count of large tables on many RDBMS engines.\n* Similarly the number of pages available and the \"number\" of the current\n  page are not supported.\n* Ordering must be on a single field.\n\n# What You Gain\n\n* With proper indexing, performance and scale.\n\n# Examples\n\n    # order by updated descending (order by field must be unique)\n    qs = LargeDataSetModel.objects.all()\n    paginator = PerformantPaginator(qs, per_page=40, ordering='-updated')\n    # hand paginator off to something that takes a pagintor. it implements the\n    # full api, as does Page, but some of that api may have alternative\n    # behaviors, i.e. return None for count etc.\n\n    # get the first page\n    page = paginator.page()\n    # get the second page (number is our token)\n    page = pagination.page(page.next_page_number())\n    # ...\n\n    # public items, ordered by id\n    qs = LargeDataSetModel.objects.filter(public=True)\n    paginator = PerformantPaginator(qs)\n    # ...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fross%2Fperformant-pagination","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fross%2Fperformant-pagination","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fross%2Fperformant-pagination/lists"}