{"id":20710032,"url":"https://github.com/oxylabs/puppeteer-tutorial","last_synced_at":"2025-06-22T13:08:08.794Z","repository":{"id":54700306,"uuid":"522558870","full_name":"oxylabs/puppeteer-tutorial","owner":"oxylabs","description":"Use this tutorial and learn how to perform web scraping using a headless browser.","archived":false,"fork":false,"pushed_at":"2025-02-11T12:50:15.000Z","size":70,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-22T13:08:02.494Z","etag":null,"topics":["headless-browser","puppeteer","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-08-08T13:22:58.000Z","updated_at":"2025-05-01T07:44:58.000Z","dependencies_parsed_at":"2024-04-19T11:45:56.268Z","dependency_job_id":"dbe97556-bce4-4e44-bfb7-fe869cf55ddc","html_url":"https://github.com/oxylabs/puppeteer-tutorial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oxylabs/puppeteer-tutorial","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpuppeteer-tutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpuppeteer-tutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpuppeteer-tutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpuppeteer-tutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/puppeteer-tutorial/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpuppeteer-tutorial/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261296937,"owners_count":23137220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["headless-browser","puppeteer","web-scraping"],"created_at":"2024-11-17T02:09:36.962Z","updated_at":"2025-06-22T13:08:03.773Z","avatar_url":"https://github.com/oxylabs.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Puppeteer Tutorial: Scraping With a Headless Browser\n## Table of Contents \n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.go2cloud.org/aff_c?offer_id=7\u0026aff_id=877\u0026url_id=112)\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/GbxmdGhZjq)\n\n- [Controlling the browsers programmatically](#controlling-the-browsers-programmatically)\n- [Installation](#installation)\n- [Getting started with Puppeteer](#getting-started-with-puppeteer)\n- [Scraping an element from a page](#scraping-an-element-from-a-page)\n- [Scraping multiple elements](#scraping-multiple-elements)\n- [Scraping a hotel listing page](#scraping-a-hotel-listing-page)\n- [Summary](#summary)\n\n## Controlling the browsers programmatically\n\nThis Puppeteer tutorial will cover web scraping with Puppeteer in much detail. Puppeteer, however, is a Node.js package, making it exclusive for JavaScript developers. Python programmers, therefore, have a similar option – Pyppeteer.  \n\n ### Pyppeteer\n\nPyppeteer is an unofficial port of Puppeteer for Python. This also bundles Chromium and works smoothly with it. Pyppeteer can work with Chrome as well, similar to Puppeteer. \n\nThe syntax is very similar as it uses the asyncio library for Python, except the syntactical differences between Python and JavaScript. Here are two scripts in JavaScript and Python that load a page and then take a screenshot of it.\n\nPuppeteer example:\n\n```javascript\nconst puppeteer = require('puppeteer');\nasync function main() {\n  const browser = await puppeteer.launch();\n  const page = await browser.newPage();\n  await page.goto('https://oxylabs.io/');\n  await page.screenshot({'path': 'oxylabs_js.png'})\n  await browser.close();\n}\nmain();\n```\nPyppeteer Example:\n\n```python\nimport asyncio\nimport pyppeteer\n\nasync def main():\n    browser = await pyppeteer.launch()\n    page = await browser.newPage()\n    await page.goto('https://oxylabs.io/')\n    await page.screenshot({'path': 'oxylabs_python.png'})\n    await browser.close()\nasyncio.get_event_loop().run_until_complete(main())\n```\n\nThe code is very similar. For web scraping dynamic websites, Pyppeteer can be an excellent alternative to Selenium for Python developers. But for the sake of making a Puppeteer tutorial, the following sections, we will cover Puppeteer, starting with the installation.\n\n## Installation\n\nBefore moving on with this Puppeteer tutorial, let’s get the basic tools installed. \n\n**Prerequisite**\n\nThere are only two pieces of software that will be needed:\n\nNode.js (which is bundled with npm—the package manager for Node.js)\n\nAny code editor\n\nThe only thing that you need to know about Node.js is that it is a runtime framework. This means that JavaScript code, which typically runs in a browser, can run without a browser. \nNode.js is available for Windows, Mac OS, and Linux. It can be downloaded at their official [download page](https://nodejs.org/en/download). \n\n### Create node.js project\n\nBefore writing any code to web scrape using node js, create a folder where JavaScript files will be stored. All the code for Puppeteer is written in .js files and is run by Node. \nOnce the folder is created, navigate to this folder and run the initialization command:\n\n```npm init -y```\n\nThis will create a package.json file in the directory. This file will contain information about the packages that are installed in this folder. The next step is to install the Node.js Packages in this folder. \n\n### How do you run Puppeteer\n\nInstalling Puppeteer is very easy. Just run the npm install command from the terminal. Note that the working directory should be the one which contains package.json:\n\n```npm install puppeteer```\n\nNote that Puppeteer is bundled with a full instance of Chromium. When it is installed, it downloads a recent version of Chromium that is guaranteed to work with the version of Puppeteer being installed. \n\n## Getting started with Puppeteer\n\nCreate a new file in your node project directory (the directory that contains package.json and node_modules). Save this file as example1.js and add this code:\n\n```javascript\nconst puppeteer = require('puppeteer');\nasync function main() {\n    // Add code here\n}\nmain();`\n``` \n\nThe code above can be simplified by making the function anonymous and calling it on the same line:\n\n```javascript\nconst puppeteer = require('puppeteer');\n(async () =\u003e {\n    // Add code here\n})();\n```\n\nThe required keyword will ensure that the Puppeteer library is available in the file. The rest of the lines are the placeholder where an anonymous, asynchronous function is being created and executed. For the next step, launch the browser. \n\n```javascript\nconst browser = await puppeteer.launch();\n``` \n\nNote that by default, the browser is launched in the headless mode. If there is an explicit need for a user interface, the above line can be modified to include an object as a parameter. \n\n```javascript\nconst browser = await puppeteer.launch({headless:false}); // default is true\n```\n\nThe next step would be to open a page:\n\n```javascript\nconst page = await browser.newPage();\n```\n\nNow that a page or in other words, a tab, is available, any website can be loaded by simply calling the goto() function:\n\n```javascript\nawait page.goto('https://oxylabs.io/');\n```\n\nOnce the page is loaded, the DOM elements, as well the rendered page is available. This can be verified by taking a quick screenshot:\n\n```javascript\nawait page.screenshot({path: 'oxylabs_1080.png'})\n```\n\nThis, however, will create only an 800×600 pixel image. The reason is that Puppeteer sets an initial page size to 800×600px. This can be changed by setting the viewport, before taking the screenshot.\n\n```javascript\n  await page.setViewport({\n    width: 1920,\n    height: 1080,\n  });\n```\n\nFinally, remember to close the browser:\n\n```javascript\nawait browser.close();\n```\n\nPutting it altogether, here is the complete script. \n\n```javascript\nconst puppeteer = require('puppeteer');\n(async () =\u003e {\n  const browser = await puppeteer.launch();\n  const page = await browser.newPage();\n  await page.setViewport({\n    width: 1920,\n    height: 1080,\n  });\n  await page.goto('https://oxylabs.io/');\n  await page.screenshot({path: 'oxylabs_1080.png'})\n  await browser.close();\n})();\n```\n\nRun this file from the terminal using this command: \n\n```node example1.js```\n\nThis should create a new file oxylabs_1080.png in the same directory.\n\n**Bonus tip:** If you need a PDF, you can use the pdf() function:\n\n```javascript\nawait page.pdf({path: 'oxylabs.pdf', format: 'A4'});\n``` \n\n## Scraping an element from a page\n\nPuppeteer loads the complete page in DOM. This means that we can extract any data from the page. The easiest way to do this is to use the function evaluate(). This allows to execute JavaScript functions like document.querySelector(). Consequently, it lets us extract any Element from the DOM.\n\nTo understand this, open [this link](https://en.wikipedia.org/wiki/Web_scraping) in your preferred browser. \n\nOnce the page is loaded, right-click the heading of the page, and select Inspect. This should open developer tools with the Elements tab activated. Here it is visible that the page’s heading is in h1 element, with id and class both set to firstHeading.\n\nNow, go to the Console tab in the developer toolbox and write in this line:\n\n```javascript\ndocument.querySelector('#firstHeading')\n```\n\nYou will immediately see that our desired tag is extracted.\n\n![ZmJiMWIyNzAtYTg1Zi00ZGE2LTg4MDYtMmFlZmE3YjgyOWFj_puppeteer-tutorial](https://user-images.githubusercontent.com/110830803/183444884-1ac40d44-158d-4967-95bf-b9a2f8f13807.jpg)\n\n\nThis returns one element from the page. For this particular element, all we need is text. Text can be easily extracted with this line of code:\n\n```javascript\ndocument.querySelector('#firstHeading').textContent\n```\n\nThe text can now be returned using the return keyword. The next step is to surround this in the evaluate method. This will ensure that this querySelector can be run. \n\n```javascript\nawait page.evaluate(() =\u003e {\n    return document.querySelector(\"#firstHeading\").textContent;\n});\n```\n\nThe result of the evaluate() function can be stored in a variable to complete the functionality. Finally, do not forget to close the browser. Here is the complete script:\n\n```javascript\nconst puppeteer = require(\"puppeteer\");\n \n(async () =\u003e {\n  const browser = await puppeteer.launch();\n  const page = await browser.newPage();\n  await page.goto(\"https://en.wikipedia.org/wiki/Web_scraping\");\n \n  title = await page.evaluate(() =\u003e {\n    return document.querySelector(\"#firstHeading\").textContent.trim();\n  });\n  console.log(title);\n  await browser.close();\n})();\n```\n\n## Scraping multiple elements\n\nExtracting multiple elements would involve three steps:\n\n1. Use of querySelectorAll to get all elements matching the selector:\n\n```javascript\nheadings_elements = document.querySelectorAll(\"h2 .mw-headline\");\n```\n\n2. create an array, as heading_elements is of type NodeList. \n\n```javascript\nheadings_array = Array.from(headings_elements);\n```\n\n3. Call the map() function can be called to process each element in the array and return it.\n\n```javascript\nreturn headings_array.map(heading =\u003e heading.textContent);\n```\n\nThis of course needs to be surrounded by page.evaluate() function. Putting everything together, this is the complete script. You can save this as wiki_toc.js:\n\n```javascript\nconst puppeteer = require(\"puppeteer\");\n \n(async () =\u003e {\n  const browser = await puppeteer.launch();\n  const page = await browser.newPage();\n  await page.goto(\"https://en.wikipedia.org/wiki/Web_scraping\");\n \n  headings = await page.evaluate(() =\u003e {\n    headings_elements = document.querySelectorAll(\"h2 .mw-headline\");\n    headings_array = Array.from(headings_elements);\n    return headings_array.map(heading =\u003e heading.textContent);\n  });\n  console.log(headings);\n  await browser.close();\n})();\n```\n\nThis file can now be run from your terminal:\n\n```node wiki_toc.js```\n\n**Bonus tip:** Array.from() function can be supplied with a map function directly, without a separate call to map. Depending on the comfort level, the same code can thus be written as:\n\n```javascript\nheadings = await page.evaluate(() =\u003e {\n    return Array.from(document.querySelectorAll(\"h2 .mw-headline\"),\n      heading =\u003e heading.innerText.trim());\n  });\n```\n\n## Scraping a hotel listing page\n\nThis section will explain how a typical listing page can be scraped to get a JSON object with all the required information. The concepts presented in this section will be applicable for any listing, whether it is an online store, a directory, or a hotel listing. \n\nThe example that we will take is an Airbnb. Apply some filters so that you reach a page similar to the one in the screenprint. In this particular example, we will be scraping [this Airbnb page](https://www.airbnb.com/s/homes?refinement_paths%5B%5D=%2Fhomes\u0026amp;search_type=section_navigation\u0026amp;property_type_id%5B%5D=8) that lists 20 hotels. To scrape all 20 hotels, the first step is to identify the selector for each hotel section.\n\n```javascript\nroot = Array.from(document.querySelectorAll(\"#FMP-target [itemprop='itemListElement']\"));\n```\n\nThis returns a NodeList of length 20 and stores in the variable root. Note that so far, text or any attribute has not been extracted. All we have is an array  of elements. This will be done in the map() function.\n\n```javascript\nhotels = root.map(hotel =\u003e ({ \n// code here\n}));\n```\n\nThe URL of the photo of the hotel can be extracted with a code like this:\n\n```javascript\nhotel.querySelector(\"img\").getAttribute(\"src\")\n```\n\nGetting the name of the hotel is a little trickier. The classes used on this page are some random words like _krjbj and _mvzr1f2. These class names appear to be generated dynamically and may change later on. It is better to have selectors which do not rely on these class names. \n\nThe hotel name can be extracted by combining parentElement and nextElementSibling selectors:\n\n```javascript\nhotel.querySelector('ol').parentElement.nextElementSibling.textContent\n```\n\nThe most important concept to understand here is that we are concatenating querySelectors. Effectively, the first hotel name is being extracted with this line of code:\n\n```javascript\ndocument.querySelectorAll(\"#FMP-target [itemprop='itemListElement']\")[0].querySelector('ol').parentElement.nextElementSibling.textContent\n```\n\n![OWJiMjQ4OGItM2E4Yy00ZWFmLWE2NWItOTJlMmY1NDBkNjFm_puppeteer-tutorial2](https://user-images.githubusercontent.com/110830803/183445109-2d30173b-eb31-4f1b-8c60-a313ea7a9b1e.jpg)\n\nFinally, we can create an object containing both of these values. The syntax to create an object is like this:\n\n```javascript\nHotel = {\n    Name: 'x',\n    Photo: 'y'\n}\n```\n\nPutting everything together, here is the final script. Save it as bnb.js.\n\n```javascript\nconst puppeteer = require(\"puppeteer\");\n(async () =\u003e {\n  let url = \"https://www.airbnb.com/s/homes?refinement_paths%5B%5D=%2Fhomes\u0026search_type=section_navigation\u0026property_type_id%5B%5D=8\";\n  const browser = await puppeteer.launch(url);\n  const page = await browser.newPage();\n  await page.goto(url);\n  data = await page.evaluate(() =\u003e {\n    root = Array.from(document.querySelectorAll(\"#FMP-target [itemprop='itemListElement']\"));\n    hotels = root.map(hotel =\u003e ({\n      Name: hotel.querySelector('ol').parentElement.nextElementSibling.textContent,\n      Photo: hotel.querySelector(\"img\").getAttribute(\"src\")\n    }));\n    return hotels;\n  });\n  console.log(data);\n  await browser.close();\n})();\n```\n\nRun this file from the terminal using:\n\n```node bnb.js```\n\nYou should be able to see a JSON object printed on the console. \n\n## Summary\n\nTo get more detailed information, take a look at the official [Puppeteer documentation](https://pptr.dev/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fpuppeteer-tutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fpuppeteer-tutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fpuppeteer-tutorial/lists"}