{"id":13400033,"url":"https://github.com/binux/pyspider","last_synced_at":"2026-10-18T22:02:15.548Z","repository":{"id":14357182,"uuid":"17066884","full_name":"binux/pyspider","owner":"binux","description":"A Powerful Spider(Web Crawler) System in Python.","archived":true,"fork":false,"pushed_at":"2024-04-30T19:43:29.000Z","size":4171,"stargazers_count":16756,"open_issues_count":300,"forks_count":3618,"subscribers_count":5,"default_branch":"master","last_synced_at":"2026-10-17T23:21:09.544Z","etag":null,"topics":["crawler","python"],"latest_commit_sha":null,"homepage":"http://docs.pyspider.org/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/binux.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-02-21T19:18:47.000Z","updated_at":"2026-10-17T17:15:25.000Z","dependencies_parsed_at":"2024-06-18T14:08:30.450Z","dependency_job_id":"ec96d1f0-49d0-4f75-ad36-23b31abc3335","html_url":"https://github.com/binux/pyspider","commit_stats":{"total_commits":1071,"total_committers":65,"mean_commits":"16.476923076923075","dds":0.3613445378151261,"last_synced_commit":"897891cafb21ea5b4ac08e728ad2ea212879f7fa"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/binux/pyspider","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/binux%2Fpyspider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/binux%2Fpyspider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/binux%2Fpyspider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/binux%2Fpyspider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/binux","download_url":"https://codeload.github.com/binux/pyspider/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/binux%2Fpyspider/sbom","scorecard":{"id":239004,"data":{"date":"2025-08-11","repo":{"name":"github.com/binux/pyspider","commit":"897891cafb21ea5b4ac08e728ad2ea212879f7fa"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.8,"checks":[{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"project is archived","details":["Warn: Repository is archived."],"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":2,"reason":"Found 1/4 approved changesets -- score normalized to 2","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating python:3.6 to python:3.6@sha256:f8652afaf88c25f0d22354d547d892591067aa4026a7fa9a6819df9f300af6fc","Warn: npmCommand not pinned by hash: Dockerfile:21","Warn: pipCommand not pinned by hash: Dockerfile:25","Warn: pipCommand not pinned by hash: Dockerfile:32","Info:   0 out of   1 containerImage dependencies pinned","Info:   0 out of   1 npmCommand dependencies pinned","Info:   0 out of   2 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"28 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2018-66 / GHSA-562c-5r94-xh97","Warn: Project is vulnerable to: PYSEC-2019-179 / GHSA-5wv5-4vpf-pj6m","Warn: Project is vulnerable to: PYSEC-2023-62 / GHSA-m2qf-hxjv-5gpq","Warn: Project is vulnerable to: PYSEC-2019-217 / GHSA-462w-v97r-4m45","Warn: Project is vulnerable to: PYSEC-2014-8 / GHSA-8r7q-cvjq-x353","Warn: Project is vulnerable to: GHSA-cpwx-vrp4-4pq7","Warn: Project is vulnerable to: PYSEC-2014-82 / GHSA-fqh9-2qgg-h84h","Warn: Project is vulnerable to: PYSEC-2021-66 / GHSA-g3rq-g295-4j3m","Warn: Project is vulnerable to: GHSA-h5c8-rqwp-cp95","Warn: Project is vulnerable to: GHSA-h75v-3vvj-5mfj","Warn: Project is vulnerable to: PYSEC-2019-220 / GHSA-hj2j-77xm-mc5v","Warn: Project is vulnerable to: GHSA-q2x7-8rv6-6q7h","Warn: Project is vulnerable to: GHSA-55x5-fj6c-h6m8","Warn: Project is vulnerable to: PYSEC-2021-19 / GHSA-jq4v-f5q6-mjqq","Warn: Project is vulnerable to: PYSEC-2020-62 / GHSA-pgww-xf46-h92r","Warn: Project is vulnerable to: PYSEC-2022-230 / GHSA-wrxv-2j5q-m38w","Warn: Project is vulnerable to: GHSA-hgjp-83m4-h4fj","Warn: Project is vulnerable to: GHSA-v5rq-w2xm-7g5f","Warn: Project is vulnerable to: GHSA-m87m-mmvp-v9qm","Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: PYSEC-2023-74 / GHSA-j8r2-6x86-q33q","Warn: Project is vulnerable to: GHSA-753j-mpmx-qq6g","Warn: Project is vulnerable to: GHSA-7cx3-6m66-7c5m","Warn: Project is vulnerable to: GHSA-8w49-h785-mj3c","Warn: Project is vulnerable to: PYSEC-2023-75 / GHSA-hj3f-6gcp-jg8j","Warn: Project is vulnerable to: GHSA-qppv-j76h-2rpx","Warn: Project is vulnerable to: GHSA-w235-7p84-xx57"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 28 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-17T06:18:05.572Z","repository_id":14357182,"created_at":"2025-08-17T06:18:05.572Z","updated_at":"2025-08-17T06:18:05.572Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":344872684,"owners_count":38336094,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-10-03T21:59:58.778Z","status":"online","status_checked_at":"2026-10-17T02:00:06.547Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","python"],"created_at":"2024-07-30T19:00:46.975Z","updated_at":"2026-10-18T22:02:15.148Z","avatar_url":"https://github.com/binux.png","language":"Python","funding_links":[],"categories":["Python","Web Crawling \u0026 Web Scraping","All","Web","资源列表","Web Crawling","python","Web 后端","spider","Data Processing","Uncategorized","HTML 处理","Application Recommendation","Web Crawling [🔝](#readme)","Awesome Python","Data","DevOps Utilities"],"sub_categories":["Scraping and Crawling","HTML 处理","Data Pre-processing \u0026 Loading","Uncategorized","🤖 Automation Tools","Web Crawling \u0026 Web Scraping","Aggregators"],"readme":"pyspider [![Build Status]][Travis CI] [![Coverage Status]][Coverage]\n========\n\nA Powerful Spider(Web Crawler) System in Python.\n\n- Write script in Python\n- Powerful WebUI with script editor, task monitor, project manager and result viewer\n- [MySQL](https://www.mysql.com/), [MongoDB](https://www.mongodb.org/), [Redis](http://redis.io/), [SQLite](https://www.sqlite.org/), [Elasticsearch](https://www.elastic.co/products/elasticsearch); [PostgreSQL](http://www.postgresql.org/) with [SQLAlchemy](http://www.sqlalchemy.org/) as database backend\n- [RabbitMQ](http://www.rabbitmq.com/), [Redis](http://redis.io/) and [Kombu](http://kombu.readthedocs.org/) as message queue\n- Task priority, retry, periodical, recrawl by age, etc...\n- Distributed architecture, Crawl Javascript pages, Python 2.{6,7}, 3.{3,4,5,6} support, etc...\n\nTutorial: [http://docs.pyspider.org/en/latest/tutorial/](http://docs.pyspider.org/en/latest/tutorial/)  \nDocumentation: [http://docs.pyspider.org/](http://docs.pyspider.org/)  \nRelease notes: [https://github.com/binux/pyspider/releases](https://github.com/binux/pyspider/releases)  \n\nSample Code \n-----------\n\n```python\nfrom pyspider.libs.base_handler import *\n\n\nclass Handler(BaseHandler):\n    crawl_config = {\n    }\n\n    @every(minutes=24 * 60)\n    def on_start(self):\n        self.crawl('http://scrapy.org/', callback=self.index_page)\n\n    @config(age=10 * 24 * 60 * 60)\n    def index_page(self, response):\n        for each in response.doc('a[href^=\"http\"]').items():\n            self.crawl(each.attr.href, callback=self.detail_page)\n\n    def detail_page(self, response):\n        return {\n            \"url\": response.url,\n            \"title\": response.doc('title').text(),\n        }\n```\n\n\nInstallation\n------------\n\n* `pip install pyspider`\n* run command `pyspider`, visit [http://localhost:5000/](http://localhost:5000/)\n\n**WARNING:** WebUI is open to the public by default, it can be used to execute any command which may harm your system. Please use it in an internal network or [enable `need-auth` for webui](http://docs.pyspider.org/en/latest/Command-Line/#-config).\n\nQuickstart: [http://docs.pyspider.org/en/latest/Quickstart/](http://docs.pyspider.org/en/latest/Quickstart/)\n\nContribute\n----------\n\n* Use It\n* Open [Issue], send PR\n* [User Group]\n* [中文问答](http://segmentfault.com/t/pyspider)\n\n\nTODO\n----\n\n### v0.4.0\n\n- [ ] a visual scraping interface like [portia](https://github.com/scrapinghub/portia)\n\n\nLicense\n-------\nLicensed under the Apache License, Version 2.0\n\n\n[Build Status]:         https://img.shields.io/travis/binux/pyspider/master.svg?style=flat\n[Travis CI]:            https://travis-ci.org/binux/pyspider\n[Coverage Status]:      https://img.shields.io/coveralls/binux/pyspider.svg?branch=master\u0026style=flat\n[Coverage]:             https://coveralls.io/r/binux/pyspider\n[Try]:                  https://img.shields.io/badge/try-pyspider-blue.svg?style=flat\n[Issue]:                https://github.com/binux/pyspider/issues\n[User Group]:           https://groups.google.com/group/pyspider-users\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbinux%2Fpyspider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbinux%2Fpyspider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbinux%2Fpyspider/lists"}