{"id":31061955,"url":"https://github.com/pymarcus/spiderbot","last_synced_at":"2026-02-14T16:07:04.558Z","repository":{"id":132212202,"uuid":"516511938","full_name":"PyMarcus/SpiderBot","owner":"PyMarcus","description":null,"archived":false,"fork":false,"pushed_at":"2022-07-22T13:14:35.000Z","size":16,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-15T11:49:45.095Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PyMarcus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-07-21T20:24:46.000Z","updated_at":"2022-07-21T20:27:11.000Z","dependencies_parsed_at":null,"dependency_job_id":"188f13d4-4181-411c-ba1b-2d36158d407b","html_url":"https://github.com/PyMarcus/SpiderBot","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/PyMarcus/SpiderBot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PyMarcus%2FSpiderBot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PyMarcus%2FSpiderBot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PyMarcus%2FSpiderBot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PyMarcus%2FSpiderBot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PyMarcus","download_url":"https://codeload.github.com/PyMarcus/SpiderBot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PyMarcus%2FSpiderBot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29449108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-14T15:52:44.973Z","status":"ssl_error","status_checked_at":"2026-02-14T15:52:11.208Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-15T11:46:02.117Z","updated_at":"2026-02-14T16:07:04.553Z","avatar_url":"https://github.com/PyMarcus.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"SCRAPY A FRAMEWORK TO BUILD WEBSCRAPING AND WEBCRAWLER\r\n\r\nhttps://docs.scrapy.org/en/latest/intro/overview.html\r\n\r\nScrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.\r\nEven though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler.\r\n\r\n### start a project\r\n\r\n    scrapy startproject tutorial\r\n\r\n### run the project (first time)\r\n\r\n    python -m venv venv\r\n\r\n    .\\venv\\Scripts\\activate\r\n\r\n    pip install -r requirements\r\n\r\n    scrapy crawl quotes\r\n\r\n\r\n### run if u already install the requirements:\r\n\r\n    scrapy crawl quotes\r\n\r\n\r\n### save data how a json file\r\n\r\n    scrapy crawl quotes -O quotes.json\r\n\r\n\r\n\r\n### run with cache:\r\n\r\n    scrapy crawl quotes -s HTTPCACHE_ENABLE=1\r\n\r\n### create a new spider:\r\n    \r\n    scrapy genspider name site.com\r\n\r\n### list spiders\r\n\r\n    scrapy list","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpymarcus%2Fspiderbot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpymarcus%2Fspiderbot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpymarcus%2Fspiderbot/lists"}