Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/crwlrsoft/crawler
Library for Rapid (Web) Crawler and Scraper Development
crawler crawling hacktoberfest php scraper scraping scraping-websites web-crawler web-crawling web-scraper web-scraping
Last synced: 04 Jul 2024
![](https://github.com/crwlrsoft.png)
https://github.com/scrapehero-code/amazon-scraper
A simple web scraper to extract Product Data and Pricing from Amazon
amazon-scraper page-scraper scrape-products web-crawling web-scraping web-scraping-tutorials
Last synced: 14 Jun 2024
![](https://github.com/scrapehero-code.png)
https://github.com/ayakashi-io/ayakashi
:zap: Ayakashi.io - The next generation web scraping framework
automation data-mining headless-chrome web-crawling web-scraping
Last synced: 08 Jun 2024
![](https://github.com/ayakashi-io.png)
https://github.com/jgujerry/python-frameworks
Another curated list of Python frameworks
api artificial-intelligence cms data-workflow deep-learning devops distributed-computing frameworks machine-learning messaging parallel-computing pipeline python task-queue web-crawling webapp
Last synced: 07 Jun 2024
![](https://github.com/jgujerry.png)
https://github.com/wangjksjtu/Data-Mining-51Job
Data-mining on 51Job website
51job data-mining machine-learning scikit-learn seaborn web-crawling
Last synced: 22 May 2024
![](https://github.com/wangjksjtu.png)
https://github.com/my8100/scrapyd-cluster-on-heroku
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO :point_right:
cluster heroku logparser python scrapy scrapyd scrapydweb web-crawling web-scraping
Last synced: 19 May 2024
![](https://github.com/my8100.png)
https://github.com/alyakhtar/Katastrophe
Command Line Tool to download torrents
bittorrent command-line deluge kickass-torrents python screenshot torrent web-crawling
Last synced: 14 May 2024
![](https://github.com/alyakhtar.png)
https://github.com/spyboy-productions/PhantomCrawler
Boost website hits by generating requests from multiple proxy IPs.
ddos-attack-tools proxy proxy-configuration proxy-rotation web-crawling web-scrapping website-analytics website-hits
Last synced: 10 May 2024
![](https://github.com/spyboy-productions.png)
https://github.com/spyboy-productions/omnisci3nt
Unveiling the Hidden Layers of the Web – A Comprehensive Web Reconnaissance Tool
admin-login-finder admin-panel-finder admin-panel-finder-of-any-website directory-enumeration dmarc-record-examination dns-enumeration ip-lookup osint pentesting-tools port-scanning reconnaissance-tool social-media-and-email-discovery ssl-certificate subdomain-enumeration technology-analysis wayback-machine-access web-crawling web-reconnaissance website-hacking whois
Last synced: 10 May 2024
![](https://github.com/spyboy-productions.png)
https://github.com/apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
apify automation crawler crawling headless headless-chrome javascript nodejs npm playwright puppeteer scraper scraping typescript web-crawler web-crawling web-scraping
Last synced: 10 May 2024
![](https://github.com/apify.png)
https://github.com/TurnerSoftware/InfinityCrawler
A simple but powerful web crawler library for .NET
crawler robots-txt spider web-crawler web-crawling
Last synced: 05 May 2024
![](https://github.com/TurnerSoftware.png)
https://github.com/mike-gee/webtranspose
Web scraping API for building AI applications.
chatbots crawling crawling-python python scraping scraping-python web-crawling web-scraping web-scraping-python
Last synced: 13 Apr 2024
![](https://github.com/mike-gee.png)
https://github.com/ScrapingAnt/amazon_scraper
Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt
amazon amazon-scraper amazon-scraping-library data-mining js node-js price-scraper price-scraping scrape-products scraper scraping scraping-api scraping-data scraping-python scraping-web scraping-websites web-crawler web-crawlers web-crawling
Last synced: 08 Apr 2024
![](https://github.com/ScrapingAnt.png)
https://github.com/brianmadden/krawler
A web crawling framework written in Kotlin
crawler4j framework kotlin link-checker web-crawler web-crawling webcrawler
Last synced: 06 Apr 2024
![](https://github.com/brianmadden.png)
https://github.com/omkarcloud/botasaurus
The All in One Framework to build Awesome Scrapers.
anti-bot anti-detect anti-detect-browser anti-detection antidetect-browser bot-detection bypass-cloudflare cloudflare-bypass cloudflare-scrape python-scraper python-web-scraper python-web-scraping scraping-framework scraping-python scraping-tool undetectable undetected undetected-chromedriver web-crawling web-scraping-python
Last synced: 27 Mar 2024
![](https://github.com/omkarcloud.png)