{"id":13414795,"url":"https://github.com/my8100/scrapydweb","last_synced_at":"2025-04-23T20:56:30.343Z","repository":{"id":39617657,"uuid":"150997854","full_name":"my8100/scrapydweb","owner":"my8100","description":"Web app for Scrapyd cluster management, Scrapy log analysis \u0026 visualization, Auto packaging, Timer tasks, Monitor \u0026 Alert, and Mobile UI. Docs 文档 :point_right:","archived":false,"fork":false,"pushed_at":"2025-02-19T14:41:45.000Z","size":3194,"stargazers_count":3267,"open_issues_count":60,"forks_count":578,"subscribers_count":72,"default_branch":"master","last_synced_at":"2025-04-23T20:55:59.701Z","etag":null,"topics":["dashboard","log-analysis","log-parsing","scrapy","scrapy-log-analysis","scrapy-visualization","scrapyd","scrapyd-admin","scrapyd-api","scrapyd-cluster-management","scrapyd-control","scrapyd-keeper","scrapyd-log-analysis","scrapyd-manage","scrapyd-monitor","scrapyd-ui","scrapyd-visualization","spider"],"latest_commit_sha":null,"homepage":"https://github.com/my8100/files","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/my8100.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-30T19:11:44.000Z","updated_at":"2025-04-19T08:59:10.000Z","dependencies_parsed_at":"2022-07-14T09:22:35.913Z","dependency_job_id":"34d2b455-0bac-4048-a844-56bcb47a0fbb","html_url":"https://github.com/my8100/scrapydweb","commit_stats":{"total_commits":64,"total_committers":2,"mean_commits":32.0,"dds":0.015625,"last_synced_commit":"7a3b81dba2cba4279c9465064a693bb277ac20e9"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/my8100%2Fscrapydweb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/my8100%2Fscrapydweb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/my8100%2Fscrapydweb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/my8100%2Fscrapydweb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/my8100","download_url":"https://codeload.github.com/my8100/scrapydweb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250514767,"owners_count":21443208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dashboard","log-analysis","log-parsing","scrapy","scrapy-log-analysis","scrapy-visualization","scrapyd","scrapyd-admin","scrapyd-api","scrapyd-cluster-management","scrapyd-control","scrapyd-keeper","scrapyd-log-analysis","scrapyd-manage","scrapyd-monitor","scrapyd-ui","scrapyd-visualization","spider"],"created_at":"2024-07-30T21:00:36.892Z","updated_at":"2025-04-23T20:56:30.316Z","avatar_url":"https://github.com/my8100.png","language":"Python","funding_links":[],"categories":["Python","资源列表","Apps","Scrapy Distributed"],"sub_categories":["HTML 处理","Front-End Scrapy Managers"],"readme":":abc: English | [:mahjong: 简体中文](https://github.com/my8100/scrapydweb/blob/master/README_CN.md)\n\n# ScrapydWeb: Web app for Scrapyd cluster management, with support for Scrapy log analysis \u0026 visualization.\n\n[![PyPI - scrapydweb Version](https://img.shields.io/pypi/v/scrapydweb.svg)](https://pypi.org/project/scrapydweb/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/scrapydweb.svg)](https://pypi.org/project/scrapydweb/)\n[![CircleCI](https://circleci.com/gh/my8100/scrapydweb/tree/master.svg?style=shield)](https://circleci.com/gh/my8100/scrapydweb/tree/master)\n[![codecov](https://codecov.io/gh/my8100/scrapydweb/branch/master/graph/badge.svg)](https://codecov.io/gh/my8100/scrapydweb)\n[![Coverage Status](https://coveralls.io/repos/github/my8100/scrapydweb/badge.svg?branch=master)](https://coveralls.io/github/my8100/scrapydweb?branch=master)\n[![Downloads - total](https://static.pepy.tech/badge/scrapydweb)](https://pepy.tech/project/scrapydweb)\n[![GitHub license](https://img.shields.io/github/license/my8100/scrapydweb.svg)](https://github.com/my8100/scrapydweb/blob/master/LICENSE)\n[![Twitter](https://img.shields.io/twitter/url/https/github.com/my8100/scrapydweb.svg?style=social)](https://twitter.com/intent/tweet?text=@my8100_%20ScrapydWeb:%20Web%20app%20for%20Scrapyd%20cluster%20management,%20with%20support%20for%20Scrapy%20log%20analysis%20%26%20visualization.%20%23python%20%23scrapy%20%23scrapyd%20%23webscraping%20%23scrapydweb%20\u0026url=https%3A%2F%2Fgithub.com%2Fmy8100%2Fscrapydweb)\n\n\n##\n![servers](https://raw.githubusercontent.com/my8100/scrapydweb/master/screenshots/servers.png)\n\n## Scrapyd :x: ScrapydWeb :x: LogParser\n### :book: Recommended Reading\n[:link: How to efficiently manage your distributed web scraping projects](https://github.com/my8100/files/blob/master/scrapydweb/README.md)\n\n[:link: How to set up Scrapyd cluster on Heroku](https://github.com/my8100/scrapyd-cluster-on-heroku)\n\n\n## :eyes: Demo\n[:link: scrapydweb.herokuapp.com](https://scrapydweb.herokuapp.com)\n\n\n## :rocket: Sponsor\nThis project is supported by [IPRoyal](https://iproyal.com/?r=802099). You can get premium quality proxies at unbeatable prices\nwith a discount using [this referral link](https://iproyal.com/?r=802099)!\n\n\n## :stars: Features\n\u003cdetails\u003e\n\u003csummary\u003eView contents\u003c/summary\u003e\n\n- :diamond_shape_with_a_dot_inside: Scrapyd Cluster Management\n  - :100: All Scrapyd JSON API Supported\n  - :ballot_box_with_check: Group, filter and select any number of nodes\n  - :computer_mouse: **Execute command on multinodes with just a few clicks**\n\n- :mag: Scrapy Log Analysis\n  - :bar_chart: Stats collection\n  - :chart_with_upwards_trend: **Progress visualization**\n  - :bookmark_tabs: Logs categorization\n\n- :battery: Enhancements\n  - :package: **Auto packaging**\n  - :male_detective: **Integrated with [:link: *LogParser*](https://github.com/my8100/logparser)**\n  - :alarm_clock: **Timer tasks**\n  - :e-mail: **Monitor \u0026 Alert**\n  - :iphone: Mobile UI\n  - :closed_lock_with_key: Basic auth for web UI\n\n\u003c/details\u003e\n\n\n## :computer: Getting Started\n\u003cdetails\u003e\n\u003csummary\u003eView contents\u003c/summary\u003e\n\n### :warning: Prerequisites\n:heavy_exclamation_mark: **Make sure that [:link: Scrapyd](https://github.com/scrapy/scrapyd) has been installed and started on all of your hosts.**\n\n:bangbang: Note that for remote access, you have to manually set 'bind_address = 0.0.0.0' in [:link: the configuration file of Scrapyd](https://scrapyd.readthedocs.io/en/latest/config.html#example-configuration-file)\nand restart Scrapyd to make it visible externally.\n\n### :arrow_down: Install\n- Use pip:\n```bash\npip install scrapydweb\n```\n:heavy_exclamation_mark: Note that you may need to execute `python -m pip install --upgrade pip` first in order to get the latest version of scrapydweb, or download the tar.gz file from https://pypi.org/project/scrapydweb/#files and get it installed via `pip install scrapydweb-x.x.x.tar.gz`\n\n- Use git:\n```bash\npip install --upgrade git+https://github.com/my8100/scrapydweb.git\n```\nOr:\n```bash\ngit clone https://github.com/my8100/scrapydweb.git\ncd scrapydweb\npython setup.py install\n```\n\n### :arrow_forward: Start\n1. Start ScrapydWeb via command `scrapydweb`. (a config file would be generated for customizing settings at the first startup.)\n2. Visit http://127.0.0.1:5000 **(It's recommended to use Google Chrome for a better experience.)**\n\n### :globe_with_meridians: Browser Support\nThe latest version of Google Chrome, Firefox, and Safari.\n\n\u003c/details\u003e\n\n\n## :heavy_check_mark: Running the tests\n\u003cdetails\u003e\n\u003csummary\u003eView contents\u003c/summary\u003e\n\n\u003cbr\u003e\n\n```bash\n$ git clone https://github.com/my8100/scrapydweb.git\n$ cd scrapydweb\n\n# To create isolated Python environments\n$ pip install virtualenv\n$ virtualenv venv/scrapydweb\n# Or specify your Python interpreter: $ virtualenv -p /usr/local/bin/python3.7 venv/scrapydweb\n$ source venv/scrapydweb/bin/activate\n\n# Install dependent libraries\n(scrapydweb) $ python setup.py install\n(scrapydweb) $ pip install pytest\n(scrapydweb) $ pip install coverage\n\n# Make sure Scrapyd has been installed and started, then update the custom_settings item in tests/conftest.py\n(scrapydweb) $ vi tests/conftest.py\n(scrapydweb) $ curl http://127.0.0.1:6800\n\n# '-x': stop on first failure\n(scrapydweb) $ coverage run --source=scrapydweb -m pytest tests/test_a_factory.py -s -vv -x\n(scrapydweb) $ coverage run --source=scrapydweb -m pytest tests -s -vv --disable-warnings\n(scrapydweb) $ coverage report\n# To create an HTML report, check out htmlcov/index.html\n(scrapydweb) $ coverage html\n```\n\n\u003c/details\u003e\n\n\n## :building_construction: Built With\n\u003cdetails\u003e\n\u003csummary\u003eView contents\u003c/summary\u003e\n\n\u003cbr\u003e\n\n- Front End\n  - [:link: Element](https://github.com/ElemeFE/element)\n  - [:link: ECharts](https://github.com/apache/incubator-echarts)\n\n- Back End\n  - [:link: Flask](https://github.com/pallets/flask)\n\n\u003c/details\u003e\n\n\n## :clipboard: Changelog\nDetailed changes for each release are documented in the [:link: HISTORY.md](https://github.com/my8100/scrapydweb/blob/master/HISTORY.md).\n\n\n## :man_technologist: Author\n| [\u003cimg src=\"https://github.com/my8100.png\" width=\"100px;\"/\u003e](https://github.com/my8100)\u003cbr/\u003e [\u003csub\u003emy8100\u003c/sub\u003e](https://github.com/my8100) |\n| --- |\n\n\n## :busts_in_silhouette: Contributors\n| [\u003cimg src=\"https://github.com/simplety.png\" width=\"100px;\"/\u003e](https://github.com/simplety)\u003cbr/\u003e [\u003csub\u003eKaisla\u003c/sub\u003e](https://github.com/simplety) |\n| --- |\n\n\n## :copyright: License\nThis project is licensed under the GNU General Public License v3.0 - see the [:link: LICENSE](https://github.com/my8100/scrapydweb/blob/master/LICENSE) file for details.\n\n\n## :star: Stargazers over time\n[![Stargazers over time](https://starchart.cc/my8100/scrapydweb.svg?variant=adaptive)](https://starchart.cc/my8100/scrapydweb)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmy8100%2Fscrapydweb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmy8100%2Fscrapydweb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmy8100%2Fscrapydweb/lists"}