Projects in Awesome Lists tagged with webcrawler

https://github.com/crawlab-team/crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

crawlab crawler crawling-tasks docker go platform scrapy scrapyd-ui spider spiders-management web-crawler webcrawler webspider

Last synced: 25 Sep 2024

https://github.com/ssssssss-team/spider-flow

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

crawler jsoup spider spider-flow web-crawler web-spider webcrawler webspider xpath

Last synced: 26 Sep 2024

https://github.com/GeneralNewsExtractor/GeneralNewsExtractor

新闻网页正文通用抽取器 Beta 版.

python3 webcrawler webspider

Last synced: 31 Jul 2024

https://github.com/generalnewsextractor/generalnewsextractor

新闻网页正文通用抽取器 Beta 版.

python3 webcrawler webspider

Last synced: 02 Oct 2024

https://github.com/zorlan/skycaiji

蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

crawler crawling php spider webcrawler

Last synced: 30 Sep 2024

https://github.com/amirgamil/apollo

A Unix-style personal search engine and web crawler for your digital footprint.

personal-search poseidon search unix-like webcrawler

Last synced: 25 Sep 2024

https://github.com/scrapinghub/scrapyrt

HTTP API for Scrapy spiders

crawler crawling hacktoberfest hacktoberfest2021 python scraper scrapy twisted webcrawler webcrawling

Last synced: 01 Aug 2024

https://github.com/3nock/spidersuite

Advance web security spider/crawler

bugbounty cplusplus crawler gui information-gathering osint-tool pentest qt5 recon security-tools spider web-spider webcrawler

Last synced: 28 Sep 2024

https://github.com/jaeksoft/opensearchserver

Open-source Enterprise Grade Search Engine Software

crawler custom-search enterprise indexing java lucene ocr opensearchserver search search-engine synonyms webcrawler webcrawling

Last synced: 31 Jul 2024

https://github.com/z0m31en7/uscrapper

Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.

darkweb darkweb-crawler information-extraction information-gathering osint osint-python osint-tool python reconnaissance selenium selenium-webscraper tor web-scraping webcra webcrawler webscraping website-scraper websites

Last synced: 28 Sep 2024

https://github.com/z0m31en7/Uscrapper

Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.

darkweb darkweb-crawler information-extraction information-gathering osint osint-python osint-tool python reconnaissance selenium selenium-webscraper tor web-scraping webcra webcrawler webscraping website-scraper websites

Last synced: 02 Aug 2024

https://github.com/voliveirajr/seleniumcrawler

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

asp-net python scraper scraping scraping-websites scrapper scrapy selenium selenium-webdriver webcrawler webcrawling

Last synced: 28 Sep 2024

https://github.com/pavlovtech/WebReaper

Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.

crawler datamining parser parsing scraper scraping scraping-api scraping-data scraping-tool scraping-web scraping-websites webcrawler webscraping

Last synced: 01 Aug 2024

https://github.com/shenxiangzhuang/PythonDataAnalysis

The data and code that used in my book.

data-science python3 webcrawler

Last synced: 31 Jul 2024

https://github.com/hfreire/browser-as-a-service

A web browser :earth_americas: hosted as a service, to render your JavaScript web pages as HTML

browser browser-as-a-service crawler docker github-actions javascript puppeteer rest-api scraper server webcrawler

Last synced: 02 Aug 2024

https://github.com/DeuxHuitHuit/algolia-webcrawler

Simple node worker that crawls sitemaps in order to keep an algolia index up-to-date

algolia algolia-webcrawler indexing javascript search-engine webcrawler

Last synced: 30 Jul 2024

https://github.com/Conso1eCowb0y/Deepminer

Deep web crawler and search engine

crawler crawling dark-web data-mining deepminer deepweb github hacking onion osint python-web-scraper python3 search-engine security security-tools spider the-onion-router tor tor-network webcrawler

Last synced: 02 Aug 2024

https://github.com/opencharles/charles

Java web crawling library

dynamic selenium webcrawler webdriver

Last synced: 01 Aug 2024

https://github.com/marcel0024/cococrawler

An declarative and easy to use web crawler and scraper in C#

cococrawler crawler crawling-tool csharp dotnet dotnetcore scraper scraping-tool webcrawler webcrawler-csharp webcrawling webscraper

Last synced: 28 Sep 2024

https://github.com/gdgd009xcd/AutoMacroBuilderForZAP

A ZAPROXY Add-on that allows testing of web application vulnerabilities by recording complex multi-step sequences. You can test applications that need to access pages in a specific order, such as shopping carts or registration of member information.

activescan addon authentication csrf multistep multistep-form security security-testing security-tools vulnerability-scanners web-security webcrawler websecurity zap-extension zaproxy