{"id":20710007,"url":"https://github.com/oxylabs/scrapy-web-scraping","last_synced_at":"2026-06-10T09:31:28.861Z","repository":{"id":178553245,"uuid":"619544273","full_name":"oxylabs/scrapy-web-scraping","owner":"oxylabs","description":"Web Scraping with Scrapy: Python Tutorial","archived":false,"fork":false,"pushed_at":"2025-09-24T12:58:19.000Z","size":840,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-24T14:39:18.991Z","etag":null,"topics":["github-python","python","scrapy","scrapy-crawler"],"latest_commit_sha":null,"homepage":"https://oxylabs.io/blog/scrapy-web-scraping-tutorial","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-03-27T10:57:36.000Z","updated_at":"2025-09-24T12:58:23.000Z","dependencies_parsed_at":"2024-04-19T12:24:10.999Z","dependency_job_id":"431c0bfc-153d-4c61-8db5-8247edf86cb5","html_url":"https://github.com/oxylabs/scrapy-web-scraping","commit_stats":null,"previous_names":["oxylabs/scrapy-web-scraping"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oxylabs/scrapy-web-scraping","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrapy-web-scraping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrapy-web-scraping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrapy-web-scraping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrapy-web-scraping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/scrapy-web-scraping/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrapy-web-scraping/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34146871,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["github-python","python","scrapy","scrapy-crawler"],"created_at":"2024-11-17T02:09:32.360Z","updated_at":"2026-06-10T09:31:28.794Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web Scraping with Scrapy: Python Tutorial\r\n\r\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.io/pages/gitoxy?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=scrapy-web-scraping-github\u0026transaction_id=102f49063ab94276ae8f116d224b67)\r\n\r\n\r\n[![](https://dcbadge.limes.pink/api/server/Pds3gBmKMH?style=for-the-badge\u0026theme=discord)](https://discord.gg/Pds3gBmKMH) [![YouTube](https://img.shields.io/badge/YouTube-Oxylabs-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@oxylabs)\r\n\r\n## How to use Scrapy\r\n\r\nThis section briefly explains setting up a Scrapy project for web scraping use cases. Creating a Scrapy project for web scraping in Python is a simple three-step procedure. \r\n\r\n1. Install Scrapy\r\n2. Create a Scrapy project\r\n3. Generate a new Spider for your web-scraping target \r\n\r\nLet’s start by installing the Srapy. Open your Python command terminal and type the following pip command:\r\n\r\n```Bash\r\npip install scrapy\r\n```\r\n\r\nScrapy would take a few minutes to complete the installation, depending on your internet connection speed.\r\n\r\n![scrapy installation progress](images/scrapy_installation_progress.png)\r\n\r\nOnce the installation for the Scrapy gets complete, you can start creating Scrapy projects using the following command:\r\n\r\n```bash\r\nscrapy startproject \u003cproject_name\u003e\r\n```\r\n\r\nYou need to replace `\u003cproject_name\u003e` with the actual name of your project. For example, the next command will create a new Scrapy project named **scrapyproject**.\r\n\r\n```Bash\r\nscrapy startproject scrapyproject\r\n```\r\n\r\nExecuting the above command creates a new folder named “scrapyproject” in the current directory context and places all the project files in it. \r\n\r\nTo generate your first spider, you need to change your current directory to this “scrapyproject” folder using the `cd scrapyproject` command. Now, you can generate a new spider with the following command:\r\n\r\n```Bash\r\nscrapy genspider \u003cspider_name\u003e \u003curl_domain\u003e\r\n```\r\n\r\nReplace `\u003cspider_namer\u003e` and `\u003curl_domain\u003e` with your actual name for the spider and the web scraping target, respectively. For example, the following command will generate a “books” spider with a target URL [books.toscrape.com](http://books.toscrape.com/).\r\n\r\n```Bash\r\nscrapy genspider books books.toscrape.com\r\n```\r\n\r\nCongratulations! 🎉 We just completed the initial setup for our Scrapy web scraper. Here is what the project folder looks like:\r\n\r\n![Scrapy project structure](images/scrapy_project_structure.png)\r\n\r\n### Scrapy project structure\r\n\r\nEach Scrapy project has the following files and folders: -\r\n\r\n- **spiders folder:** Contains spiders for defining how to extract data from specific websites. Each spider is designed to target a specific website or set of websites. Typically, a spider contains a set of rules that dictate how to navigate the site and extract the desired data.\r\n- **items.py:** Contains objects that represent the data a spider is trying to extract. Items are defined using Python classes and help organize the data in a structured format.\r\n- **middleware.py:** This file contains middleware information for request routing. You can also write the [custom proxy middleware](https://oxylabs.io/resources/integrations/scrapy-proxy) in this file.\r\n- **pipelines.py:** Once the spider completes the data extraction, we need to process and store the data in a structured format. Pipelines define a set of processing steps on data while it is being extracted.\r\n- **settings.py:** It contains various configuration settings that control the behavior of the Scrapy framework. For example, you can set user agent string, configure download delay, limit maximum concurrent requests, configure middleware, etc.\r\n- **Scrapy.cfg:** A plain text file that contains a series of configuration directives. For example, it includes directives for the project name, the location of the spider modules, and the settings to be used when running the spiders.\r\n\r\nYou are encouraged to read [Scrapy’s documentation](https://docs.scrapy.org/en/latest/) page for more information on the basic project structure. Let’s now see how to customize the Scrapy project to serve our web scraping needs.\r\n\r\n## Customizing Scrapy Spider\r\n\r\nThis section discusses customizing a Scrapy project to scrape all the books’ information on the [Books Store](http://books.toscrape.com/). Before jumping into the coding, let’s look at our [initial target](http://books.toscrape.com/) page.\r\n\r\n![BooksToScrape top view](images/Books_to_scrape_top.png)\r\n\r\nNotice the book title is in the `title` attribute of the `\u003ca\u003e` tag inside an \u003ch3\u003e element. This `\u003ch3\u003e` element is enclosed by an `\u003carticle\u003e` tag with the `product_pod` class. Similarly, the book price resides in a `\u003cp\u003e` tag with the `price_color` tag. \r\n\r\nThe target page lists only the first 20 books of 1000 books. Therefore, there would be a total of 50 pages. Let’s see the bottom of the same to know the CSS selector for the next page link. \r\n\r\n![BooksToScrape Footer View](images/books_to_scrape_bottom.png)\r\n\r\nThe URL for the next page is available in the `href` attribute of the `\u003ca\u003e` tag. This `\u003ca\u003e` is enclosed by an `\u003cli\u003e` tag with the `next` class. Mind this information; we will use it in the subsequent section. \r\n\r\n### Scrapy Spider Customization in Action\r\n\r\nOpen the `books.py` spider file in an IDE of your choice. Replace the original template script with the following code:\r\n\r\n```python\r\n class BooksSpider(scrapy.Spider):\r\n    name = 'books'\r\n\r\n    def start_requests(self):\r\n        URL = 'https://books.toscrape.com/'\r\n        yield scrapy.Request(url=URL, callback=self.response_parser)\r\n\r\n    def response_parser(self, response):\r\n        for selector in response.css('article.product_pod'):\r\n            yield {\r\n                'title': selector.css('h3 \u003e a::attr(title)').extract_first(),\r\n                'price': selector.css('.price_color::text').extract_first()\r\n            }\r\n\r\n        next_page_link = response.css('li.next a::attr(href)').extract_first()\r\n        if next_page_link:\r\n            yield response.follow(next_page_link, callback=self.response_parser)\r\n\r\n```\r\n\r\nThe above script contains two generators: the `start_requests()` and the `response_parser()`. The `start_requests()` generator automatically executes whenever a crawl command is issued to this spider. Here, it retrieves the contents from the URL and issues a call back to the `response_parser()`. \r\n\r\nThe `response_parser()` generator, when executed, extracts the desired product information from the iterable `response`. When yields for all the 20 products in the current response are completed, it uses the `response.follow()` method to retrieve the contents of the next page. The `follow()` method calls back to the `reponse_parser()` again to extract and yield products from the new page. The cycle continues until the `next_page_link` becomes NULL or empty. \r\n\r\nYou can execute the Scrapy project by issuing a crawl command from the command terminal:\r\n\r\n```Bash\r\nscrapy crawl books\r\n```\r\n\r\nWe can also give a file name with the `-o` option to write the output to a file.\r\n\r\n```bash\r\nscrapy crawl -o out.csv books\r\n```\r\n\r\n### Running a Scrapy project from within the Python script\r\n\r\nYou might find it inconvenient to issue a crawl command from a command terminal to execute the project. Luckily, you can execute a Scrapy project from within the Python script.\r\n\r\nThe following spider script, when executed, automatically issues a crawl command and saves the output in the **books_data.csv** file. \r\n\r\n```python\r\nimport csv\r\nimport scrapy\r\nfrom scrapy import signals\r\nfrom scrapy.crawler import CrawlerProcess\r\nfrom scrapy.signalmanager import dispatcher\r\n\r\nclass BooksSpider(scrapy.Spider):\r\n    name = 'books'\r\n\r\n    def start_requests(self):\r\n        URL = 'https://books.toscrape.com/'\r\n        yield scrapy.Request(url=URL, callback=self.response_parser)\r\n\r\n    def response_parser(self, response):\r\n        for selector in response.css('article.product_pod'):\r\n            yield {\r\n                'title': selector.css('h3 \u003e a::attr(title)').extract_first(),\r\n                'price': selector.css('.price_color::text').extract_first()\r\n            }\r\n\r\n        next_page_link = response.css('li.next a::attr(href)').extract_first()\r\n        if next_page_link:\r\n            yield response.follow(next_page_link, callback=self.response_parser)\r\n\r\n\r\n\r\ndef book_spider_result():\r\n    books_results = []\r\n\r\n    def crawler_results(item):\r\n        books_results.append(item)\r\n\r\n    dispatcher.connect(crawler_results, signal=signals.item_scraped)\r\n    crawler_process = CrawlerProcess()\r\n    crawler_process.crawl(BooksSpider)\r\n    crawler_process.start()\r\n    return books_results\r\n\r\n\r\nif __name__ == '__main__':\r\n    books_data=book_spider_result()\r\n\r\n    keys = books_data[0].keys()\r\n    with open('books_data.csv', 'w', newline='') as output_file_name:\r\n        writer = csv.DictWriter(output_file_name, keys)\r\n        writer.writeheader()\r\n        writer.writerows(books_data)\r\n\r\n```\r\n\r\nThe `start_requests()` and `response_parser()` are the same as our previous code. The `__main__` serves as a starting point for direct execution. It calls the `book_spider_result()` function and waits for it to return a value.\r\n\r\nThe `book_spider_result()` works as follows: -\r\n\r\n1. Sets control dispatcher to execute the `crawler_results()` function on the encounter of an `item_scraped` signal. The `item_scraped` is generated whenever the spider scrapes an item from the target.\r\n2. Creates a crawler process for the `BooksSpider`and starts it.\r\n3. Whenever the `BookSpider` completes scraping an item, it emits the `item_scraped` signal. This causes the `crawler_results()` function to execute and append this scraped item to the `books_results` list.\r\n4. Once the crawler process finishes scraping the items, the `book_spider_result()` returns the `books_results` list.\r\n\r\nThe `__main__` function on return writes the returned `books_data` to the “books_data.csv” file.\r\n\r\nHere is what the output file looks like:\r\n\r\n![books_data.csv file's partial snippet](images/output_books_data_csv.png)\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fscrapy-web-scraping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fscrapy-web-scraping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fscrapy-web-scraping/lists"}