{"id":20710051,"url":"https://github.com/oxylabs/pagination-with-python","last_synced_at":"2025-04-23T05:05:43.272Z","repository":{"id":134336660,"uuid":"373198555","full_name":"oxylabs/Pagination-With-Python","owner":"oxylabs","description":"A guide on how to deal with pagination via Python. ","archived":false,"fork":false,"pushed_at":"2025-02-11T12:27:56.000Z","size":725,"stargazers_count":24,"open_issues_count":0,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-23T05:05:15.926Z","etag":null,"topics":["amazon-scraper-python","github-python","json-database-python","pagination","python","python-ecommerce","python-image-scraper","python-web-crawler","scraper-python","serp-api-python","webscraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-06-02T14:32:03.000Z","updated_at":"2025-04-16T13:08:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"875c2dcc-6a54-43ef-b70b-b49006897292","html_url":"https://github.com/oxylabs/Pagination-With-Python","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2FPagination-With-Python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2FPagination-With-Python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2FPagination-With-Python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2FPagination-With-Python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/Pagination-With-Python/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250372937,"owners_count":21419722,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-scraper-python","github-python","json-database-python","pagination","python","python-ecommerce","python-image-scraper","python-web-crawler","scraper-python","serp-api-python","webscraping"],"created_at":"2024-11-17T02:09:40.399Z","updated_at":"2025-04-23T05:05:43.265Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dealing With Pagination Via Python   \n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.go2cloud.org/aff_c?offer_id=7\u0026aff_id=877\u0026url_id=112)\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/GbxmdGhZjq)\n\n\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=Python\u0026color=blueviolet\" /\u003e  \u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=Pagination\u0026color=blue\" /\u003e \u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=Web%20Scraping\u0026color=yellowgreen\" /\u003e \n\nThis article covers everything you need to know about dealing with pagination using Python. By the end of this article, you will be able to deal with various kinds of pagination in web scraping projects.\n\n## Table of Contents\n\n- [Introduction](#introduction)\n- [Pagination With Next button](#pagination-with-next-button)\n  - [Analyzing the Website](#analyzing-the-website)\n  - [Python Code to Handle Pagination](#python-code-to-handle-pagination)\n- [Pagination Without Next Button](#pagination-without-next-button)\n- [Pagination With Infinite Scroll](#pagination-with-infinite-scroll)\n  - [Handling Sites with JSON Response](#handling-sites-with-json-response)\n  - [Handling Sites with HTML Response](#handling-sites-with-html-response)\n- [Pagination With Load More Button](#pagination-with-load-more-button)\n\n## Introduction\n\nPagination is a common feature for most websites. There is always a lot to display which can be done on one page. This is true for product listings, blogs, photos, videos, directories, etc. \n\nEach website has its way of using pagination. The common types of pagination are as follows:\n\n- With Next Button\n- Page Numbers without Next button\n- Pagination with Infinite Scroll \n- Pagination with Load More\n\nIn this article, we will examine these cases and explore ways to handle these websites. \n\n## Pagination With Next button\n\nLet's start with a simple example. Head over to [this page](http://books.toscrape.com/catalogue/category/books/fantasy_19/index.html) and see the pagination.\n\n### Analyzing the Website\n\n![Next Button Example](images/next_button_example.png)\n\nThis site has a next button. If this button is clicked, the website will go to the next page. \n\n![Page 2](images/next_button_example_page2.png)\n\nNow this site displays a previous button along with a next button. If we keep clicking next until it takes to the last page, this is how this looks:\n\n![Last Page](images/next_button_example_page3.png)\n\nMoreover, with every click, the URL is changing:\n\n- Page 1 - `http://books.toscrape.com/catalogue/category/books/fantasy_19/index.html`\n- Page 2 - `http://books.toscrape.com/catalogue/category/books/fantasy_19/page-2.html`\n- Page 3 - `http://books.toscrape.com/catalogue/category/books/fantasy_19/page-3.html`\n\nThe next step is to press F12 and examine the HTML markup of the next button.\n\n![Next Button Markup](images/next_button_locate.png)\n\nNow that we know that the website is not dynamic and that the next button is not a button, but an anchor element, we can find the URL of the next page by locating this anchor tag.\n\nLet's write some Python code.\n\n### Python Code to Handle Pagination\n\nLet's start with basic code to get the first page using `requests` module. If you do not have it installed, install in a virtual environment. You may also want to install `BeautifuslSoup4`. We will be using `BeautifulSoup4` (or `bs4`) for locating the next button. We need a parser for `bs4`. In this article we are using `lxml`.\n\n```shell\npip install requests beautifulsoup4 lxml\n```\n\nLet's start with writing a simple code that fetches the first page and prints the footer. Note that we are printing the footer so that we can keep track of the current page. In a real-world application, you will write the code here that scrapes data.\n\n```python\nimport requests\nfrom bs4 import BeautifulSoup\n\nurl = 'http://books.toscrape.com/catalogue/category/books/fantasy_19/index.html'\n\nresponse = requests.get(url)\nsoup = BeautifulSoup(response.text, \"lxml\")\nfooter_element = soup.select_one('li.current')\nprint(footer_element.text.strip())\n# Other code to extract data\n```\n\nThe output of this code will be simply the footer of the first page:\n\n```shell\nPage 1 of 3\n```\n\nFew points to note here are as follows:\n\n- Requests library is sending a GET request to the specified URL\n- The `soup` object is being queried using CSS Selector. This CSS selector is website-specific.\n\nLet's modify this code to locate the next button:\n\nFirst we need to locate the next button. \n\n```python\nnext_page_element = soup.select_one('li.next \u003e a')\n```\n\nIf this next element is found, we can get the value of the `href`attribute. One important thing to note here is that often the `href` will be a relative url. In such cases, we can use `urljoin` method from `urllib.parse` module. This whole block of the code now needs to be wrapped in a `while True` loop.\n\n```python\nurl = 'http://books.toscrape.com/catalogue/category/books/fantasy_19/index.html'\nwhile True:\n    response = requests.get(url)\n    soup = BeautifulSoup(response.text, \"lxml\")\n    footer_element = soup.select_one('li.current')\n    print(footer_element.text.strip())\n    # Pagination\n    next_page_element = soup.select_one('li.next \u003e a')\n    if next_page_element:\n        next_page_url = next_page_element.get('href')\n        url = urljoin(url, next_page_url)\n    else:\n        break\n```\n\nThe output of this code will be simply the footer of all three pages:\n\n```shell\nPage 1 of 3\nPage 2 of 3\nPage 3 of 3\n```\n\nYou can find this code in the `next_button_requests.py` file in this repository.\n\n## Pagination Without Next Button\n\nSome websites will not show a next button, but just page numbers. For example, here is an example of the pagination from `https://www.gosc.pl/doc/791526.Zaloz-zbroje`.\n\n![Pager without next](images/pager_without_next.png)\n\nFor example, one such web page is `https://www.gosc.pl/doc/791526.Zaloz-zbroje`.  If we examine the HTML markup for this page, something interesting can be seen:\n\n```html\n\u003cspan class=\"pgr_nrs\"\u003e\n\t\t\u003cspan\u003e1\u003c/span\u003e\n\t\t\u003ca href=\"/doc/791526.Zaloz-zbroje/2\"\u003e2\u003c/a\u003e\n\t\t\u003ca href=\"/doc/791526.Zaloz-zbroje/3\"\u003e3\u003c/a\u003e\n\t\t\u003ca href=\"/doc/791526.Zaloz-zbroje/4\"\u003e4\u003c/a\u003e\n\u003c/span\u003e\n```\n\nThis makes visiting all these pages easy. The first step is to retrieve the first page. Next, we can use BeautifulSoup to extract all these links to other pages. Finally, we can write a `for` loop that runs on all these links:\n\n```python\nurl = 'https://www.gosc.pl/doc/791526.Zaloz-zbroje'\nresponse = requests.get(url)\nsoup = BeautifulSoup(response.text, 'lxml')\npage_link_el = soup.select('.pgr_nrs a')\n# process first page\nfor link_el in page_link_el:\n    link = urljoin(url, link_el.get('href'))\n    response = requests.get(link)\n    soup = BeautifulSoup(response.text, 'lxml')\n    print(response.url)\n    # process remaining pages\n```\n\nYou can find the complete code in the file `no_next_button.py`.\n\n## Pagination With Infinite Scroll\n\nThis kind of pagination does not show page numbers or a next button. \n\nLet's take [this site](https://techinstr.myshopify.com/collections/all) as an example. This site shows 8 products when on the page load. As you scroll down, it will dynamically load more items, 8 at a time. Another important thing to note here is that URL does not change as more pages are loaded. \n\nIn such cases, websites use an asynchronous call to an API to get more content and show this content on the page using JavaScript. The actual data returned by the API can be HTML or JSON. \n\n### Handling Sites With JSON Response\n\nBefore you load the site, press `F12` to open Developer Tools, head over to Network tab, and select XHR.  Now go to `http://quotes.toscrape.com/scroll` and monitor the traffic. Scroll down to load more content. \n\nYou will notices that as you scroll down,  more requests are sent to `quotes?page=x` , where x is the page number.\n\n\n\n![](images/scroll_json_response.png)\n\nRemember that the most important step is **figuring out is when to stop**. This is where `has_next` in the response is going to be useful.\n\nWe can write a while loop as we did in the previous section. This time, there is no need of BeautifulSoup because the response is JSON. \n\n```python\nurl = 'http://quotes.toscrape.com/api/quotes?page={}'\npage_numer = 1\nwhile True:\n    response = requests.get(url.format(page_numer))\n    data = response.json()\n    # Process data here\n    # ...\n    print(response.url)  # only for debug\n    if data.get('has_next'):\n        page_numer += 1\n        else:\n            break\n```\n\nOnce we figure out how the site works, it's quite easy.\n\nNow let's look at one more example.\n\n### Handling Sites With HTML Response\n\nIn the previous section, we looked at that figuring out when to stop is important. The earlier example was easier. All we needed was to examine an attribute in the JSON. \n\nSome websites make it harder. Let's see one such example. \n\nOpen developer tools by pressing F12 in your browser, go to Network tab, and then select XHR. Navigate to `https://techinstr.myshopify.com/collections/all`. You will notice that initially 8 products are loaded.\n\nIf we scroll down, the next 8 products are loaded. Also, notice the following:\n\n- The number of products 132 is on the first page.\n- The URL of the index page is different than the remaining pages.\n- The response is HTML, with no clear way to identify when to stop.\n\n![Infinite Scroll](images/scroll_html_response.png)\n\nTo handle pagination for this site, first of we will load the index page and extract the number of products. We have already observed that 8 products are loaded in one request.  Now we can calculate the number of pages as follows:\n\nNumber of pages = 132/8 =16.5\n\nNow we can use `math.ceil` function to get the last page, which will give us 17. Note that if you use round function, you may end up missing one page in some cases. Using `ceil` function ensures that pages are always rounded *up*.\n\nOne more change that we need to do is to use the session. The complete code is available in the `infinite_scroll_html.py` file. Here is the important part of the code:\n\n```python\nindex_page = 'https://techinstr.myshopify.com/collections/all'\nsession = requests.session()\n\nresponse = session.get(index_page)\nsoup = BeautifulSoup(response.text, \"lxml\")\ncount_element = soup.select_one('.filters-toolbar__product-count')\ncount_str = count_element.text.replace('products', '')\ncount = int(count_str)\n# Process page 1 data here\npage_count = math.ceil(count/8)\nurl = 'https://techinstr.myshopify.com/collections/all?page={}'\nfor page_numer in range(2, page_count+1): \n    response = session.get(url.format(page_numer))\n    soup = BeautifulSoup(response.text, \"lxml\")\n    # process page 2 onwards data here\n\n```\n\n\n\n## Pagination With Load More Button\n\nThe way the pagination using a Load More button works is very similar to how infinite scroll works. The only difference is how loading the next page is triggered on the browser.\n\nAs we are working directly with the web page without a browser, these two scenarios need to be handled the same way. \n\nLet's look at one example. Open `https://smarthistory.org/americas-before-1900/` with Developer Tools and click `Load More`.\n\nYou will see that the response is in JSON format with an attribute `remaining`. The key observations are as follows:\n\n- Each request gets 12 results \n- The value of remaining decreases by 12 with every click of Load More\n- If we set the value page to 1 in the API url, it get's the first page of the results - `https://smarthistory.org/wp-json/smthstapi/v1/objects?tag=938\u0026page=1`\n\n![](images/load_more_button.png)\n\nIn this particular case, the user agent also needs to be set for this to work correctly. The complete code is in `load_more_json.py` file. Here is the important part of the code:\n\n```python\nurl = 'https://smarthistory.org/wp-json/smthstapi/v1/objects?tag=938\u0026page={}'\nheaders = {\n    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36',\n}\npage_numer = 1\nwhile True:\n    response = requests.get(url.format(page_numer), headers=headers)\n    data = response.json()\n    # Process data\n    # ...\n    print(response.url)  # only for debug\n    if data.get('remaining') and int(data.get('remaining')) \u003e 0:\n        page_numer += 1\n        else:\n            break\n\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fpagination-with-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fpagination-with-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fpagination-with-python/lists"}