{"id":13502201,"url":"https://github.com/djm/python-scrapyd-api","last_synced_at":"2025-04-13T05:06:00.363Z","repository":{"id":20654936,"uuid":"23937101","full_name":"djm/python-scrapyd-api","owner":"djm","description":"A Python wrapper for working with Scrapyd's API.","archived":false,"fork":false,"pushed_at":"2024-07-31T07:47:49.000Z","size":86,"stargazers_count":271,"open_issues_count":4,"forks_count":33,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-04-13T05:05:43.282Z","etag":null,"topics":["api-wrapper","python","scrapy-spider"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/djm.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-09-11T21:52:59.000Z","updated_at":"2025-03-27T07:39:25.000Z","dependencies_parsed_at":"2024-11-29T19:02:53.059Z","dependency_job_id":"02ade221-0e2d-4e1a-baec-89dbb0f9073d","html_url":"https://github.com/djm/python-scrapyd-api","commit_stats":{"total_commits":79,"total_committers":4,"mean_commits":19.75,"dds":0.03797468354430378,"last_synced_commit":"48eae5606363d796b4abff232e1cc9a9f366a462"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/djm%2Fpython-scrapyd-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/djm%2Fpython-scrapyd-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/djm%2Fpython-scrapyd-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/djm%2Fpython-scrapyd-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/djm","download_url":"https://codeload.github.com/djm/python-scrapyd-api/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248665749,"owners_count":21142123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-wrapper","python","scrapy-spider"],"created_at":"2024-07-31T22:02:05.825Z","updated_at":"2025-04-13T05:06:00.340Z","avatar_url":"https://github.com/djm.png","language":"Python","readme":"# python-scrapyd-api\n\n[![The PyPI version](https://badge.fury.io/py/python-scrapyd-api.png)][pypi] [![Build status on Travis-CI](https://travis-ci.org/djm/python-scrapyd-api.png?branch=master)](https://travis-ci.org/djm/python-scrapyd-api) [![Coverage status on Coveralls](https://coveralls.io/repos/djm/python-scrapyd-api/badge.png)](https://coveralls.io/r/djm/python-scrapyd-api) [![Documentation status on ReadTheDocs](https://readthedocs.org/projects/python-scrapyd-api/badge/?version=latest)][docs]\n\nA Python wrapper for working with [Scrapyd][scrapyd]'s [API][scrapyd-api-docs].\n\nCurrent released version: 2.1.2 (see [history][history]).\n\nAllows a Python application to talk to, and therefore control, the\n[Scrapy][scrapy] daemon: [Scrapyd][scrapyd].\n\n* Supports Python 2.6, 2.7, 3.3 \u0026 3.4\n* Free software: BSD license\n* [Full documentation][docs]\n* On the [Python Package Index (PyPI)][pypi]\n* Scrapyd's [API Documentation][scrapyd-api-docs]\n\n[scrapy]: http://scrapy.org/\n[scrapyd]: https://github.com/scrapy/scrapyd\n[scrapyd-api-docs]: http://scrapyd.readthedocs.org/en/latest/api.html\n[history]: https://github.com/djm/python-scrapyd-api/blob/master/HISTORY.md\n[pypi]: https://pypi.python.org/pypi/python-scrapyd-api/\n[docs]: http://python-scrapyd-api.readthedocs.org/en/latest/\n\n## Install\n\nEasiest installation is via `pip`:\n\n```bash\npip install python-scrapyd-api\n```\n\n## Quick Usage\n\nPlease refer to the [full documentation][docs] for more detailed usage but to get you started:\n\n```python\n\u003e\u003e\u003e from scrapyd_api import ScrapydAPI\n\u003e\u003e\u003e scrapyd = ScrapydAPI('http://localhost:6800')\n```\n\n**Add a project** egg as a new version:\n\n```python\n\u003e\u003e\u003e egg = open('some_egg.egg', 'rb')\n\u003e\u003e\u003e scrapyd.add_version('project_name', 'version_name', egg)\n# Returns the number of spiders in the project.\n3\n\u003e\u003e\u003e egg.close()\n```\n\n**Cancel a scheduled job**:\n\n```python\n\u003e\u003e\u003e scrapyd.cancel('project_name', '14a6599ef67111e38a0e080027880ca6')\n# Returns the \"previous state\" of the job before it was cancelled: 'running' or 'pending'.\n'running'\n```\n\n**Delete a project** and all sibling versions:\n\n```python\n\u003e\u003e\u003e scrapyd.delete_project('project_name')\n# Returns True if the request was met with an OK response.\nTrue\n```\n\n**Delete a version** of a project:\n\n```python\n\u003e\u003e\u003e scrapyd.delete_version('project_name', 'version_name')\n# Returns True if the request was met with an OK response.\nTrue\n```\n\n**Request status** of a job:\n\n```python\n\u003e\u003e\u003e scrapyd.job_status('project_name', '14a6599ef67111e38a0e080027880ca6')\n# Returns 'running', 'pending', 'finished' or '' for unknown state.\n'running'\n```\n\n**List all jobs** registered:\n\n```python\n\u003e\u003e\u003e scrapyd.list_jobs('project_name')\n# Returns a dictionary of running, finished and pending job lists.\n{\n    'pending': [\n        {\n            u'id': u'24c35...f12ae',\n            u'spider': u'spider_name'\n        },\n    ],\n    'running': [\n        {\n            u'id': u'14a65...b27ce',\n            u'spider': u'spider_name',\n            u'start_time': u'2014-06-17 22:45:31.975358'\n        },\n    ],\n    'finished': [\n        {\n            u'id': u'34c23...b21ba',\n            u'spider': u'spider_name',\n            u'start_time': u'2014-06-17 22:45:31.975358',\n            u'end_time': u'2014-06-23 14:01:18.209680'\n        }\n    ]\n}\n```\n\n**List all projects** registered:\n\n```python\n\u003e\u003e\u003e scrapyd.list_projects()\n[u'ecom_project', u'estate_agent_project', u'car_project']\n```\n\n**Displays the load status of a service** registered:\n\n```python\n\u003e\u003e\u003e scrapyd.daemon_status()\n{u'finished': 0, u'running': 0, u'pending': 0, u'node_name': u'ScrapyMachine'}\n```\n\n**List all spiders** available to a given project:\n\n```python\n\u003e\u003e\u003e scrapyd.list_spiders('project_name')\n[u'raw_spider', u'js_enhanced_spider', u'selenium_spider']\n```\n\n**List all versions** registered to a given project:\n\n```python\n\u003e\u003e\u003e scrapyd.list_versions('project_name'):\n[u'345', u'346', u'347', u'348']\n```\n\n**Schedule a job** to run with a specific spider:\n\n```python\n# Schedule a job to run with a specific spider.\n\u003e\u003e\u003e scrapyd.schedule('project_name', 'spider_name')\n# Returns the Scrapyd job id.\nu'14a6599ef67111e38a0e080027880ca6'\n```\n\n**Schedule a job** to run while passing override settings:\n\n```python\n\u003e\u003e\u003e settings = {'DOWNLOAD_DELAY': 2}\n\u003e\u003e\u003e scrapyd.schedule('project_name', 'spider_name', settings=settings)\nu'25b6588ef67333e38a0e080027880de7'\n```\n\n**Schedule a job** to run while passing extra attributes to spider initialisation:\n\n```python\n\u003e\u003e\u003e scrapyd.schedule('project_name', 'spider_name', extra_attribute='value')\n# NB: 'project', 'spider' and 'settings' are reserved kwargs for this\n# method and therefore these names should be avoided when trying to pass\n# extra attributes to the spider init.\nu'25b6588ef67333e38a0e080027880de7'\n```\n\n\n## Setting up the project to contribute code\n\nPlease see [CONTRIBUTING.md][contributing].  This will guide you through our pull request\nguidelines, project setup and testing requirements.\n\n[contributing]: https://github.com/djm/python-scrapyd-api/blob/master/CONTRIBUTING.md\n\n## License\n\n2-clause BSD. See the full [LICENSE][license].\n\n[license]: https://github.com/djm/python-scrapyd-api/blob/master/LICENSE\n","funding_links":[],"categories":["Python","Apps"],"sub_categories":["Scrapy Service"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdjm%2Fpython-scrapyd-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdjm%2Fpython-scrapyd-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdjm%2Fpython-scrapyd-api/lists"}