{"id":13491596,"url":"https://github.com/duffelhq/paginator","last_synced_at":"2025-05-13T21:10:34.163Z","repository":{"id":29336218,"uuid":"92597883","full_name":"duffelhq/paginator","owner":"duffelhq","description":"Cursor-based pagination for Elixir Ecto","archived":false,"fork":false,"pushed_at":"2025-04-04T09:55:25.000Z","size":285,"stargazers_count":788,"open_issues_count":31,"forks_count":97,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-04-28T17:05:11.183Z","etag":null,"topics":["cursor","ecto","elixir","paginator"],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/duffelhq.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-27T12:39:32.000Z","updated_at":"2025-04-25T15:02:34.000Z","dependencies_parsed_at":"2024-01-16T09:34:21.284Z","dependency_job_id":"54cf2b18-54ad-44c5-923b-1c6c1b6da612","html_url":"https://github.com/duffelhq/paginator","commit_stats":{"total_commits":131,"total_committers":31,"mean_commits":4.225806451612903,"dds":0.7633587786259541,"last_synced_commit":"a68e6345cbd47a287ed14d4f18943eefdf7b9c16"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duffelhq%2Fpaginator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duffelhq%2Fpaginator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duffelhq%2Fpaginator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duffelhq%2Fpaginator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/duffelhq","download_url":"https://codeload.github.com/duffelhq/paginator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254029002,"owners_count":22002283,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cursor","ecto","elixir","paginator"],"created_at":"2024-07-31T19:00:58.477Z","updated_at":"2025-05-13T21:10:29.118Z","avatar_url":"https://github.com/duffelhq.png","language":"Elixir","funding_links":[],"categories":["Elixir"],"sub_categories":[],"readme":"# Paginator\n\n[![Build status](https://github.com/duffelhq/paginator/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/duffelhq/paginator/actions?query=branch%3Amain)\n[![Module Version](https://img.shields.io/hexpm/v/paginator.svg)](https://hex.pm/packages/paginator)\n[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/paginator/)\n[![Total Download](https://img.shields.io/hexpm/dt/paginator.svg)](https://hex.pm/packages/paginator)\n[![License](https://img.shields.io/hexpm/l/paginator.svg)](https://github.com/duffelhq/paginator/blob/main/LICENSE.md)\n[![Last Updated](https://img.shields.io/github/last-commit/duffelhq/paginator.svg)](https://github.com/duffelhq/paginator/commits/main)\n\n[Cursor based pagination](http://use-the-index-luke.com/no-offset) for Elixir [Ecto](https://github.com/elixir-ecto/ecto).\n\n## Why?\n\nThere are several ways to implement pagination in a project and they all have pros and cons depending on your situation.\n\n### Limit-offset\n\nThis is the easiest method to use and implement: you just have to set `LIMIT` and `OFFSET` on your queries and the\ndatabase will return records based on this two parameters. Unfortunately, it has two major drawbacks:\n\n* Inconsistent results: if the dataset changes while you are querying, the results in the page will shift and your user\nmight end seeing records they have already seen and missing new ones.\n\n* Inefficiency: `OFFSET N` instructs the database to skip the first N results of a query. However, the database must still\nfetch these rows from disk and order them before it can returns the ones requested. If the dataset you are querying is\nlarge this will result in significant slowdowns.\n\n### Cursor-based (a.k.a keyset pagination)\n\nThis method relies on opaque cursor to figure out where to start selecting records. It is more performant than\n`LIMIT-OFFSET` because it can filter records without traversing all of them.\n\nIt's also consistent, any insertions/deletions before the current page will leave results unaffected.\n\nIt has some limitations though: for instance you can't jump directly to a specific page. This may\nnot be an issue for an API or if you use infinite scrolling on your website.\n\n### Learn more\n\n* http://use-the-index-luke.com/no-offset\n* http://use-the-index-luke.com/sql/partial-results/fetch-next-page\n* https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/\n* https://developer.twitter.com/en/docs/tweets/timelines/guides/working-with-timelines\n\n## Getting started\n\n```elixir\ndefmodule MyApp.Repo do\n  use Ecto.Repo,\n    otp_app: :my_app,\n    adapter: Ecto.Adapters.Postgres\n\n  use Paginator\nend\n\nquery = from(p in Post, order_by: [asc: p.inserted_at, asc: p.id])\n\npage = MyApp.Repo.paginate(query, cursor_fields: [:inserted_at, :id], limit: 50)\n\n# `page.entries` contains all the entries for this page.\n# `page.metadata` contains the metadata associated with this page (cursors, limit, total count)\n```\n\n## Installation\n\nAdd `:paginator` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:paginator, \"~\u003e 1.2.0\"}\n  ]\nend\n```\n\n## Usage\n\nAdd `Paginator` to your repo:\n\n```elixir\ndefmodule MyApp.Repo do\n  use Ecto.Repo,\n    otp_app: :my_app,\n    adapter: Ecto.Adapters.Postgres\n\n  use Paginator\nend\n```\n\nUse the `paginate` function to paginate your queries:\n\n```elixir\nquery = from(p in Post, order_by: [asc: p.inserted_at, asc: p.id])\n\n# return the first 50 posts\n%{entries: entries, metadata: metadata}\n  = Repo.paginate(\n    query,\n    cursor_fields: [:inserted_at, :id],\n    limit: 50\n  )\n\n# assign the `after` cursor to a variable\ncursor_after = metadata.after\n\n# return the next 50 posts\n%{entries: entries, metadata: metadata}\n  = Repo.paginate(\n    query,\n    after: cursor_after,\n    cursor_fields: [{:inserted_at, :asc}, {:id, :asc}],\n    limit: 50\n  )\n\n# assign the `before` cursor to a variable\ncursor_before = metadata.before\n\n# return the previous 50 posts (if no post was created in between it should be\n# the same list as in our first call to `paginate`)\n%{entries: entries, metadata: metadata}\n  = Repo.paginate(\n    query,\n    before: cursor_before,\n    cursor_fields: [:inserted_at, :id],\n    limit: 50\n  )\n\n# return total count\n# NOTE: this will issue a separate `SELECT COUNT(*) FROM table` query to the\n# database.\n%{entries: entries, metadata: metadata}\n  = Repo.paginate(\n    query,\n    include_total_count: true,\n    cursor_fields: [:inserted_at, :id],\n    limit: 50\n  )\n\nIO.puts \"total count: #{metadata.total_count}\"\n```\n\n## Dynamic expressions\n\n```elixir\n  query =\n    from(\n      f in Post,\n      # Alias for fragment must match witch cursor field name in fetch_cursor_value_fun and cursor_fields\n      select_merge: %{\n        rank_value:\n          fragment(\"ts_rank(document, plainto_tsquery('simple', ?)) AS rank_value\", ^q)\n      },\n      where: fragment(\"document @@ plainto_tsquery('simple', ?)\", ^q),\n      order_by: [\n        desc: fragment(\"rank_value\"),\n        desc: f.id\n      ]\n    )\n    query\n    |\u003e Repo.paginate(\n      limit: 30,\n      fetch_cursor_value_fun: fn\n        # Here we build the rank_value for each returned row\n        schema, :rank_value -\u003e\n          {:ok, %{rows: [[rank_value]]}} =\n            Repo.query(\"SELECT ts_rank($1, plainto_tsquery('simple', $2))\", [\n              schema.document,\n              q\n            ])\n          rank_value\n        schema, field -\u003e\n          Paginator.default_fetch_cursor_value(schema, field)\n      end,\n      cursor_fields: [\n        {:rank_value, # Here we build the rank_value that will be used in the where clause\n         fn -\u003e\n           dynamic(\n             [x],\n             fragment(\"ts_rank(document, plainto_tsquery('simple', ?))\", ^q)\n           )\n         end},\n        :id\n      ]\n    )\n```\n\n## Security Considerations\n\n`Repo.paginate/4` will throw an `ArgumentError` should it detect an executable term in the cursor parameters passed to it (`before`, `after`).\nThis is done to protect you from potential side-effects of malicious user input, see [paginator_test.exs](https://github.com/duffelhq/paginator/blob/main/test/paginator_test.exs#L830).\n\n## Indexes\n\nIf you want to reap all the benefits of this method it is better that you create indexes on the columns you are using as\ncursor fields.\n\n### Example\n\n```elixir\n# If your cursor fields are: [:inserted_at, :id]\n# Add the following in a migration\n\ncreate index(\"posts\", [:inserted_at, :id])\n```\n\n## Caveats\n\n* This method requires a deterministic sort order. If the columns you are currently using for sorting don't match that\ndefinition, just add any unique column and extend your index accordingly.\n* You need to add `:order_by` clauses yourself before passing your query to `paginate/2`. In the future we might do that\nfor you automatically based on the fields specified in `:cursor_fields`.\n* There is an outstanding issue where Postgrex fails to properly builds the query if it includes custom PostgreSQL types.\n* This library has only be tested with PostgreSQL.\n\n## Documentation\n\nDocumentation is written into the library, you will find it in the source code, accessible from `iex` and of course, it\nall gets published to [hexdocs](http://hexdocs.pm/paginator).\n\n## Contributing\n\n### Running tests\n\nClone the repo and fetch its dependencies:\n\n```\n$ git clone https://github.com/duffelhq/paginator.git\n$ cd paginator\n$ mix deps.get\n$ mix test\n```\n\n### Building docs\n\n```\n$ mix docs\n```\n\n## Copyright and License\n\nCopyright (c) 2017 Steve Domin.\n\nThis software is licensed under [the MIT license](./LICENSE.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduffelhq%2Fpaginator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fduffelhq%2Fpaginator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduffelhq%2Fpaginator/lists"}