https://github.com/scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.
https://github.com/scrapy/scrapy

crawler crawling framework hacktoberfest python scraping web-scraping web-scraping-python

Last synced: 2 days ago
JSON representation

Scrapy, a fast high-level web crawling & scraping framework for Python.

• Host: GitHub
• URL: https://github.com/scrapy/scrapy
• Owner: scrapy
• License: bsd-3-clause
• Created: 2010-02-22T02:01:14.000Z (almost 16 years ago)
• Default Branch: master
• Last Pushed: 2025-05-05T10:41:02.000Z (8 months ago)
• Last Synced: 2025-05-05T11:13:25.232Z (8 months ago)
• Topics: crawler, crawling, framework, hacktoberfest, python, scraping, web-scraping, web-scraping-python
• Language: Python
• Homepage: https://scrapy.org
• Size: 26.6 MB
• Stars: 55,082
• Watchers: 1,771
• Forks: 10,776
• Open Issues: 639
• Metadata Files:
  • Readme: README.rst
  • Changelog: NEWS
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md
  • Security: SECURITY.md
  • Authors: AUTHORS

Awesome Lists containing this project

• awesome-python - scrapy - Fast high-level web crawling framework (Web Crawling / Test Runners)
• Starred-Repo - Scrapy: Python Web Crawler
• awesome-list - Scrapy - A fast high-level web crawling & scraping framework for Python. (Data Processing / Data Pre-processing & Loading)
• awesome-python-zh - scrapy - 一个快速的高级屏幕抓取和web爬行框架。 (Web爬行)
• awesome-python-again -
• StarryDivineSky - scrapy/scrapy
• awesome - scrapy/scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python. (Python)
• awesome-asyncio-cn - Scrapy - 一个为了爬取网站数据，提取结构性数据而编写的应用框架。 (爬虫)
• awesome-list-for-developers - scrapy - first-issue)_ <br> A fast high-level web crawling & scraping framework for Python. (Python / Misc)
• awesome-starred - scrapy/scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python. (framework)
• awesome-web-scraping - Scrapy - Comprehensive web scraping framework (Core Libraries / Python)
• awesome-docker - scrapy/scrapy
• my-awesome - scrapy/scrapy - scraping,web-scraping-python pushed_at:2025-12 star:59.4k fork:11.2k Scrapy, a fast high-level web crawling & scraping framework for Python. (Python)
• best-of-web-python - GitHub - 19% open · ⏱️ 27.10.2025): (Web Scraping & Crawling)
• awesome-repositories - scrapy/scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python. (Python)
• fucking-awesome-python - scrapy - A fast high-level screen scraping and web crawling framework. (Web Crawling)
• awesome - scrapy/scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python. (Python)
• awesome-crawler - Scrapy - A fast high-level screen scraping and web crawling framework. (Python)
• awesome-python - scrapy - A fast high-level screen scraping and web crawling framework. (Web Crawling)
• awesome-starts - scrapy/scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python. (Python)
• awesome-starred - scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python. (Python)
• awesome-web-scraper - scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python. (Python)
• awesome-crawler-cn - Scrapy - 一种高效的屏幕,网页数据采集框架。 (Python)

README

          
|logo|

.. |logo| image:: https://raw.githubusercontent.com/scrapy/scrapy/master/docs/_static/logo.svg

   :target: https://scrapy.org

   :alt: Scrapy

   :width: 480px

|version| |python_version| |ubuntu| |macos| |windows| |coverage| |conda| |deepwiki|

.. |version| image:: https://img.shields.io/pypi/v/Scrapy.svg

   :target: https://pypi.org/pypi/Scrapy

   :alt: PyPI Version

.. |python_version| image:: https://img.shields.io/pypi/pyversions/Scrapy.svg

   :target: https://pypi.org/pypi/Scrapy

   :alt: Supported Python Versions

.. |ubuntu| image:: https://github.com/scrapy/scrapy/workflows/Ubuntu/badge.svg

   :target: https://github.com/scrapy/scrapy/actions?query=workflow%3AUbuntu

   :alt: Ubuntu

.. |macos| image:: https://github.com/scrapy/scrapy/workflows/macOS/badge.svg

   :target: https://github.com/scrapy/scrapy/actions?query=workflow%3AmacOS

   :alt: macOS

.. |windows| image:: https://github.com/scrapy/scrapy/workflows/Windows/badge.svg

   :target: https://github.com/scrapy/scrapy/actions?query=workflow%3AWindows

   :alt: Windows

.. |coverage| image:: https://img.shields.io/codecov/c/github/scrapy/scrapy/master.svg

   :target: https://codecov.io/github/scrapy/scrapy?branch=master

   :alt: Coverage report

.. |conda| image:: https://anaconda.org/conda-forge/scrapy/badges/version.svg

   :target: https://anaconda.org/conda-forge/scrapy

   :alt: Conda Version

.. |deepwiki| image:: https://deepwiki.com/badge.svg

   :target: https://deepwiki.com/scrapy/scrapy

   :alt: Ask DeepWiki

Scrapy_ is a web scraping framework to extract structured data from websites.

It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_

(formerly Scrapinghub) and `many other contributors`_.

.. _many other contributors: https://github.com/scrapy/scrapy/graphs/contributors

.. _Scrapy: https://scrapy.org/

.. _Zyte: https://www.zyte.com/

Install with:

.. code:: bash

    pip install scrapy

And follow the documentation_ to learn how to use it.

.. _documentation: https://docs.scrapy.org/en/latest/

If you wish to contribute, see Contributing_.

.. _Contributing: https://docs.scrapy.org/en/master/contributing.html