{"id":18301275,"url":"https://github.com/spider-rs/web-crawling-guides","last_synced_at":"2025-06-30T23:04:13.396Z","repository":{"id":247840322,"uuid":"816024957","full_name":"spider-rs/web-crawling-guides","owner":"spider-rs","description":"How to guides on web-crawling or scraping","archived":false,"fork":false,"pushed_at":"2025-04-26T13:29:55.000Z","size":8228,"stargazers_count":20,"open_issues_count":0,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-25T01:54:16.369Z","etag":null,"topics":["agents","ai-agents","ai-scraping","clean-markdown","crawler","fast-webcrawler","html-to-markdown","llm-webcrawler","scraper","web-scraping"],"latest_commit_sha":null,"homepage":"https://spider.cloud/guides","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spider-rs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-16T21:13:38.000Z","updated_at":"2025-06-10T12:57:28.000Z","dependencies_parsed_at":"2024-07-10T23:28:27.533Z","dependency_job_id":"6dade443-babe-4a62-afd0-2a6efc905380","html_url":"https://github.com/spider-rs/web-crawling-guides","commit_stats":null,"previous_names":["spider-rs/web-crawling-guides"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/spider-rs/web-crawling-guides","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spider-rs%2Fweb-crawling-guides","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spider-rs%2Fweb-crawling-guides/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spider-rs%2Fweb-crawling-guides/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spider-rs%2Fweb-crawling-guides/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spider-rs","download_url":"https://codeload.github.com/spider-rs/web-crawling-guides/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spider-rs%2Fweb-crawling-guides/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262864119,"owners_count":23376455,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","ai-agents","ai-scraping","clean-markdown","crawler","fast-webcrawler","html-to-markdown","llm-webcrawler","scraper","web-scraping"],"created_at":"2024-11-05T15:15:00.857Z","updated_at":"2025-06-30T23:04:13.387Z","avatar_url":"https://github.com/spider-rs.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"https://spider.cloud\" target=\"_blank\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"images/spider-logo-github-dark.png\" style=\"max-width: 100%; width: 300px; margin-bottom: 20px\"\u003e\n    \u003cimg alt=\"Spider Logo\" src=\"/images/spider-logo-github-light.png\" width=\"300px\"\u003e\n  \u003c/picture\u003e\n\u003c/a\u003e\n\n# Spider Web Crawling and Scraping Guides\n\nThis repo contains a collection of guides on how to effectively use the Spider service to crawl or scrape. Contributors are welcome! 😁\n\n## Collection\n\n- [Using the Spider API](spider-api.md)\n- [How to Use Proxy Mode](proxy-mode.md)\n- [LangChain + Groq + Spider = 🚀 (Integration Guide)](langchain-groq.md)\n- [CrewAI Spider Stock Research](crewai-spider-research-agent.md)\n- [Extracting Contacts](extracting-contacts.md)\n- [Automated Cold Email Outreach Using Spider](auto-email-response-outreach.md)\n- [How to Archive Full Website](website-archiving.md)\n- Building A Speedy Resilient Web Scraper for RAG AI ([Part 1](building-a-speedy-resilient-web-scraper-for-rag-ai-part1-preparing.md), [Part 2](building-a-speedy-resilient-web-scraper-for-rag-ai-part2-scaling-up.md))\n- [Agents from Scratch](ai-agent-from-scratch.md)\n\n## Anti-Bot Detection\n\nSpider, combined with the [`headless-browser`](https://github.com/spider-rs/headless-browser) repo, achieves **full stealth** against leading bot detection services — even when running fully headless.\n\nOur techniques make Spider the most powerful crawling stack available today, providing an invisible footprint while scraping at scale.\n\nBelow are some screenshots proving Spider's stealth against major bot detectors:\n\n| Detector                                 | Screenshot                                                                                               |\n| :--------------------------------------- | :------------------------------------------------------------------------------------------------------- |\n| BrowserScan.net Bot Detection            | ✅ [View Screenshot](images/anti_bot/www_browserscan_net_bot_detection.png)        |\n| Bot Detector Rebrowser                   | ✅ [View Screenshot](images/anti_bot/bot_detector_rebrowser_net.png)                 |\n| SammySoft Bot Ecom                       | ✅ [View Screenshot](images/anti_bot/bot_sannysoft_com.png)                         |\n| Device and Browser Info (Are You a Bot?) | ✅ [View Screenshot](images/anti_bot/deviceandbrowserinfo_com_are_you_a_bot.png) |\n| Fingerprint Ecom Playground              | ✅ [View Screenshot](images/anti_bot/demo_fingerprint_com_playground.png)            |\n| Device and Browser Info - Device Test    | ✅ [View Screenshot](images/anti_bot/deviceandbrowserinfo_com_info_device.png)       |\n| Creepjs - Device Test                    | ✅ [View Screenshot](images/anti_bot/abrahamjuliot_github_io_creepjs.png)         |\n\nSpider is designed for **extreme evasion**, **high concurrency**, and **human-like behavior**, allowing you to dominate even the most protected websites.\n\n## Contribute\n\nWe're happy to accept requests in the issue tracker, improvements to the content, and additional guides.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspider-rs%2Fweb-crawling-guides","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspider-rs%2Fweb-crawling-guides","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspider-rs%2Fweb-crawling-guides/lists"}