{"id":13586131,"url":"https://github.com/scrapinghub/scrapyrt","last_synced_at":"2025-05-15T08:07:23.981Z","repository":{"id":25438714,"uuid":"28868495","full_name":"scrapinghub/scrapyrt","owner":"scrapinghub","description":"HTTP API for Scrapy spiders ","archived":false,"fork":false,"pushed_at":"2024-06-28T14:25:11.000Z","size":239,"stargazers_count":852,"open_issues_count":30,"forks_count":160,"subscribers_count":44,"default_branch":"master","last_synced_at":"2025-04-14T22:16:10.271Z","etag":null,"topics":["crawler","crawling","hacktoberfest","hacktoberfest2021","python","scraper","scrapy","twisted","webcrawler","webcrawling"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scrapinghub.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-01-06T15:07:16.000Z","updated_at":"2025-03-30T23:00:12.000Z","dependencies_parsed_at":"2024-11-10T16:25:40.700Z","dependency_job_id":"e09b6cd4-8cf8-4717-a44a-bd9ec9396eed","html_url":"https://github.com/scrapinghub/scrapyrt","commit_stats":{"total_commits":190,"total_committers":17,"mean_commits":"11.176470588235293","dds":0.3631578947368421,"last_synced_commit":"84ce691cef64fe9fb39a1fe263ac86cad1f6b611"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapinghub%2Fscrapyrt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapinghub%2Fscrapyrt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapinghub%2Fscrapyrt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapinghub%2Fscrapyrt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scrapinghub","download_url":"https://codeload.github.com/scrapinghub/scrapyrt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248968918,"owners_count":21191162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","crawling","hacktoberfest","hacktoberfest2021","python","scraper","scrapy","twisted","webcrawler","webcrawling"],"created_at":"2024-08-01T15:05:20.660Z","updated_at":"2025-04-14T22:16:23.051Z","avatar_url":"https://github.com/scrapinghub.png","language":"Python","funding_links":[],"categories":["Python","Apps"],"sub_categories":["Scrapy Service"],"readme":".. image:: https://raw.githubusercontent.com/scrapinghub/scrapyrt/master/artwork/logo.gif\n   :width: 400px\n   :align: center\n\n==========================\nScrapyRT (Scrapy realtime)\n==========================\n\n.. image:: https://github.com/scrapinghub/scrapyrt/workflows/CI/badge.svg\n   :target: https://github.com/scrapinghub/scrapyrt/actions\n\n.. image:: https://img.shields.io/pypi/pyversions/scrapyrt.svg\n    :target: https://pypi.python.org/pypi/scrapyrt\n\n.. image:: https://img.shields.io/pypi/v/scrapyrt.svg\n    :target: https://pypi.python.org/pypi/scrapyrt\n\n.. image:: https://img.shields.io/pypi/l/scrapyrt.svg\n    :target: https://pypi.python.org/pypi/scrapyrt\n\n.. image:: https://img.shields.io/pypi/dm/scrapyrt.svg\n   :target: https://pypistats.org/packages/scrapyrt\n   :alt: Downloads count\n\n.. image:: https://readthedocs.org/projects/scrapyrt/badge/?version=latest\n   :target: https://scrapyrt.readthedocs.io/en/latest/api.html\n\nAdd HTTP API for your `Scrapy \u003chttps://scrapy.org/\u003e`_ project in minutes.\n\nYou send a request to ScrapyRT with spider name and URL, and in response, you get items collected by a spider\nvisiting this URL. \n\n* All Scrapy project components (e.g. middleware, pipelines, extensions) are supported\n* You run Scrapyrt in Scrapy project directory. It starts HTTP server allowing you to schedule spiders and get spider output in JSON.\n\n\nQuickstart\n===============\n\n**1. install**\n\n.. code-block:: shell\n\n    \u003e pip install scrapyrt\n\n**2. switch to Scrapy project (e.g. quotesbot project)**\n\n.. code-block:: shell\n\n    \u003e cd my/project_path/is/quotesbot\n\n**3. launch ScrapyRT**\n\n.. code-block:: shell\n\n    \u003e scrapyrt\n\n**4. run your spiders**\n\n.. code-block:: shell\n\n    \u003e curl \"localhost:9080/crawl.json?spider_name=toscrape-css\u0026url=http://quotes.toscrape.com/\"\n\n**5. run more complex query, e.g. specify callback for Scrapy request and zipcode argument for spider**\n\n.. code-block:: shell\n\n    \u003e  curl --data '{\"request\": {\"url\": \"http://quotes.toscrape.com/page/2/\", \"callback\":\"some_callback\"}, \"spider_name\": \"toscrape-css\", \"crawl_args\": {\"zipcode\":\"14000\"}}' http://localhost:9080/crawl.json -v\n\nScrapyrt will look for ``scrapy.cfg`` file to determine your project settings,\nand will raise error if it won't find one.  Note that you need to have all\nyour project requirements installed.\n\nNote\n====\n* Project is not a replacement for `Scrapyd \u003chttps://scrapyd.readthedocs.io/en/stable/\u003e`_ or `Scrapy Cloud \u003chttps://www.zyte.com/scrapy-cloud/\u003e`_ or other infrastructure to run long running crawls\n* Not suitable for long running spiders, good for spiders that will fetch one response from some website and return items quickly\n\n\nDocumentation\n=============\n\n`Documentation is available on readthedocs \u003chttp://scrapyrt.readthedocs.org/en/latest/index.html\u003e`_.\n\nSupport\n=======\n\nOpen source support is provided here in Github. Please `create a question\nissue`_ (ie. issue with \"question\" label).\n\nCommercial support is also available by `Zyte`_.\n\n.. _create a question issue: https://github.com/scrapinghub/scrapyrt/issues/new?labels=question\n.. _Zyte: http://zyte.com\n\nLicense\n=======\nScrapyRT is offered under `BSD 3-Clause license \u003chttps://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_(%22BSD_License_2.0%22,_%22Revised_BSD_License%22,_%22New_BSD_License%22,_or_%22Modified_BSD_License%22)\u003e`_.\n\n\nDevelopment\n===========\nDevelopment taking place on `Github \u003chttps://github.com/scrapinghub/scrapyrt\u003e`_.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrapinghub%2Fscrapyrt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscrapinghub%2Fscrapyrt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrapinghub%2Fscrapyrt/lists"}