{"id":13792435,"url":"https://github.com/clemfromspace/scrapy-puppeteer","last_synced_at":"2025-03-16T19:31:49.896Z","repository":{"id":49634171,"uuid":"159150223","full_name":"clemfromspace/scrapy-puppeteer","owner":"clemfromspace","description":"Scrapy + Puppeteer","archived":false,"fork":false,"pushed_at":"2021-06-11T18:03:25.000Z","size":137,"stargazers_count":111,"open_issues_count":7,"forks_count":29,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-02-27T12:27:50.762Z","etag":null,"topics":["crawler","puppeteer","python","scraping","scrapy"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clemfromspace.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-26T10:18:28.000Z","updated_at":"2024-09-23T22:55:06.000Z","dependencies_parsed_at":"2022-08-28T22:10:18.706Z","dependency_job_id":null,"html_url":"https://github.com/clemfromspace/scrapy-puppeteer","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clemfromspace%2Fscrapy-puppeteer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clemfromspace%2Fscrapy-puppeteer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clemfromspace%2Fscrapy-puppeteer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clemfromspace%2Fscrapy-puppeteer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clemfromspace","download_url":"https://codeload.github.com/clemfromspace/scrapy-puppeteer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243826780,"owners_count":20354220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","puppeteer","python","scraping","scrapy"],"created_at":"2024-08-03T22:01:12.190Z","updated_at":"2025-03-16T19:31:49.616Z","avatar_url":"https://github.com/clemfromspace.png","language":"Python","funding_links":[],"categories":["Apps"],"sub_categories":["Process Javascript"],"readme":"# Scrapy with Puppeteer\n[![PyPI](https://img.shields.io/pypi/v/scrapy-puppeteer.svg)](https://pypi.python.org/pypi/scrapy-puppeteer) [![Build Status](https://travis-ci.org/clemfromspace/scrapy-puppeteer.svg?branch=master)](https://travis-ci.org/clemfromspace/scrapy-puppeteer) [![Test Coverage](https://api.codeclimate.com/v1/badges/86603b736e684dd4f8c9/test_coverage)](https://codeclimate.com/github/clemfromspace/scrapy-puppeteer/test_coverage) [![Maintainability](https://api.codeclimate.com/v1/badges/86603b736e684dd4f8c9/maintainability)](https://codeclimate.com/github/clemfromspace/scrapy-puppeteer/maintainability)\n\nScrapy middleware to handle javascript pages using [puppeteer](https://github.com/GoogleChrome/puppeteer).\n\n## ⚠ IN ACTIVE DEVELOPMENT - READ BEFORE USING ⚠\n\nThis is an attempt to make Scrapy and Puppeteer work together to handle Javascript-rendered pages.\nThe design is strongly inspired of the Scrapy [Splash plugin](https://github.com/scrapy-plugins/scrapy-splash).\n\n**Scrapy and Puppeteer**\n\nThe main issue when running Scrapy and Puppeteer together is that Scrapy is using [Twisted](https://twistedmatrix.com/trac/) and that [Pyppeteeer](https://miyakogi.github.io/pyppeteer/) (the python port of puppeteer we are using) is using [asyncio](https://docs.python.org/3/library/asyncio.html) for async stuff. \n\nLuckily, we can use the Twisted's [asyncio reactor](https://twistedmatrix.com/documents/18.4.0/api/twisted.internet.asyncioreactor.html) to make the two talking with each other.\n\nThat's why you **cannot** use the buit-in `scrapy` command line (installing the default reactor), you will have to use the `scrapyp` one, provided by this module.\n\nIf you are running your spiders from a script, you will have to make sure you install the asyncio reactor before importing scrapy or doing anything else:\n\n```python\nimport asyncio\nfrom twisted.internet import asyncioreactor\n\nasyncioreactor.install(asyncio.get_event_loop())\n```\n\n\n## Installation\n```\n$ pip install scrapy-puppeteer\n```\n\n## Configuration\nAdd the `PuppeteerMiddleware` to the downloader middlewares:\n```python\nDOWNLOADER_MIDDLEWARES = {\n    'scrapy_puppeteer.PuppeteerMiddleware': 800\n}\n```\n\n\n## Usage\nUse the `scrapy_puppeteer.PuppeteerRequest` instead of the Scrapy built-in `Request` like below:\n```python\nfrom scrapy_puppeteer import PuppeteerRequest\n\ndef your_parse_method(self, response):\n    # Your code...\n    yield PuppeteerRequest('http://httpbin.org', self.parse_result)\n```\nThe request will be then handled by puppeteer.\n\nThe `selector` response attribute work as usual (but contains the html processed by puppeteer).\n\n```python\ndef parse_result(self, response):\n    print(response.selector.xpath('//title/@text'))\n``` \n\n### Additional arguments\nThe `scrapy_puppeteer.PuppeteerRequest` accept 2 additional arguments:\n\n#### `wait_until`\n\nWill be passed to the [`waitUntil`](https://miyakogi.github.io/pyppeteer/_modules/pyppeteer/page.html#Page.goto) parameter of puppeteer.\nDefault to `domcontentloaded`.\n\n#### `wait_for`\nWill be passed to the [`waitFor`](https://miyakogi.github.io/pyppeteer/reference.html?highlight=image#pyppeteer.page.Page.waitFor) to puppeteer.\n\n#### `screenshot`\nWhen used, puppeteer will take a [screenshot](https://miyakogi.github.io/pyppeteer/reference.html?highlight=headers#pyppeteer.page.Page.screenshot) of the page and the binary data of the .png captured will be added to the response `meta`:\n```python\nyield PuppeteerRequest(\n    url,\n    self.parse_result,\n    screenshot=True\n)\n\ndef parse_result(self, response):\n    with open('image.png', 'wb') as image_file:\n        image_file.write(response.meta['screenshot'])\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclemfromspace%2Fscrapy-puppeteer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclemfromspace%2Fscrapy-puppeteer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclemfromspace%2Fscrapy-puppeteer/lists"}