Projects in Awesome Lists tagged with webcrawling

https://github.com/internetarchive/heritrix3

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

heritrix java warc webcrawling

Last synced: 15 May 2025

https://github.com/scrapinghub/scrapyrt

HTTP API for Scrapy spiders

crawler crawling hacktoberfest hacktoberfest2021 python scraper scrapy twisted webcrawler webcrawling

Last synced: 15 May 2025

https://github.com/jaeksoft/opensearchserver

Open-source Enterprise Grade Search Engine Software

crawler custom-search enterprise indexing java lucene ocr opensearchserver search search-engine synonyms webcrawler webcrawling

Last synced: 04 Apr 2025

https://github.com/mehmetozkaya/dotnetcrawler

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

crawler crawling csharp ddd-architecture dotnetcore entity-framework-core htmlagilitypack scraping scrapy scrapy-crawler webcrawler webcrawler-htmlagilitypack webcrawling webscraper webscraping

Last synced: 11 May 2025

https://github.com/mehmetozkaya/DotnetCrawler

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

crawler crawling csharp ddd-architecture dotnetcore entity-framework-core htmlagilitypack scraping scrapy scrapy-crawler webcrawler webcrawler-htmlagilitypack webcrawling webscraper webscraping

Last synced: 18 Apr 2025

https://github.com/dedsecinside/gotor

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

cli command-line command-line-tool docker go golang golang-server hacktoberfest http-server information-extraction osint osint-tools rest-api service tor torbot webcrawler webcrawling webscraping

Last synced: 09 Apr 2025

https://github.com/feddelegrand7/ralger

ralger makes it easy to scrape a website. Built on the shoulders of titans: rvest, xml2.

dataextraction r rstats webcrawling webscraper-website webscraping

Last synced: 06 Apr 2025

https://github.com/voliveirajr/seleniumcrawler

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

asp-net python scraper scraping scraping-websites scrapper scrapy selenium selenium-webdriver webcrawler webcrawling

Last synced: 13 Feb 2025

https://github.com/andersonkrs/malheatmap

An extension for tracking your activities on myanimelist.net

myanimelist rails ruby webcrawling

Last synced: 18 Jan 2025

https://github.com/aavache/llmwebcrawler

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

api distributed-computing fastapi huggingface large-language-models llm machine-learning milvus nlp pydantic python rag ray raylib transformer vector-database webcrawler webcrawling

Last synced: 14 Apr 2025

https://github.com/datawizard1337/ARGUS

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

crawling python scraping scrapy scrapyd webcrawling webscraping

Last synced: 20 Mar 2025

https://github.com/galarzaa90/tibia.py

API to parse tibia.com content into python objects.

beautifulsoup crawling-python python python3 tibia webcrawling

Last synced: 06 Apr 2025

https://github.com/marcel0024/cococrawler

An declarative and easy to use web crawler and scraper in C#

cococrawler crawler crawling-tool csharp dotnet dotnetcore scraper scraping-tool webcrawler webcrawler-csharp webcrawling webscraper

Last synced: 10 Apr 2025

https://github.com/michaelradu/web-crawler

A Web Crawler developed in Python.

crawler crawler-python crawlers python python-3 python-script python3 script scripting scripting-language scripts web web-crawler web-crawler-python web-crawlers web-crawling webcrawl webcrawler webcrawling

Last synced: 01 Dec 2024

https://github.com/moehmeni/ezweb

Easy to use web page analyzer

analyzer crawler scraper text-analysis text-classification text-mining webcrawler webcrawling webpage webscraper webscraping www

Last synced: 06 Apr 2025

https://github.com/robmch/mindfactory_crawling

A Python 3 Crawler for Mindfactory.de

crawler crawling data webcrawler webcrawling

Last synced: 07 May 2025

https://github.com/elektrostudios/fhm-crawler-freehardmusic.com

Crawls download urls of albums from freehardmusic.com website

albums crawl crawler crawling desktop-app desktop-application dotnet music web-crawler web-crawling web-scraper web-scraping webcrawler webcrawling webscraper webscraping windows windows-app windowsapp winforms

Last synced: 29 Jan 2025

https://github.com/congcoi123/crawler-sheis

A small crawler for getting data from the website: https://sheis.vn

crawler webcrawler webcrawling webscraper webscraping

Last synced: 20 Feb 2025

https://github.com/th3-c0der/web-crawler

A simple WebCrawler for exploring and downloading content from web pages within a given domain/url.

th3-c0der th3-coder th3c0der th3coder tool tools web-tool webcrawl webcrawler webcrawlers webcrawling

Last synced: 18 Feb 2025

https://github.com/theghostyced/dictionary-json

👻 A generated json dictionary 📚 using Python

dictionary json pipenv python3 requests webcrawling

Last synced: 11 Apr 2025

https://github.com/oussemabenhassena5/crawl4deepseek

Crawl4DeepSeek = Crawl4AI + DeepSeek 🚀 Smart, efficient, and built for deep web exploration! 🌐🤖

crawl4ai deepseek python webcrawling webscraping

Last synced: 09 Apr 2025

https://github.com/iamfarrokhnejad/murkmaw

A web crawler using Rust.

functional functional-programming rust rust-lang web-crawler web-crawling webcrawler webcrawling

Last synced: 28 Mar 2025

https://github.com/make-school-labs/makescraper

🕷Create your very own web scraper and crawler using Golang!

bew2-5 go golang makeschool webcrawling webscraping

Last synced: 21 Mar 2025

https://github.com/soyeon207/imax_crawling

🎥 파이썬으로 영화 예매 오픈 알리미 만들기

python telegram-bot webcrawling

Last synced: 24 Mar 2025

https://github.com/prosenjitjoy/web-crawling-with-goquery

Simple project to learn web crawling with Goquery using channels, goroutines and semaphore.

goquery goroutine theguardian webcrawling

Last synced: 05 Apr 2025

https://github.com/jaewonson37/python_programming

algorithms data-structures encapsulation exception-handling functions loop matplotlib monte-carlo numpy object-oriented-programming pandas prototyping pseudo-random-number-generation python recursion top-down unit-testing visualization webcrawling

Last synced: 29 Mar 2025

https://github.com/amirespahbodi/google-maps-scraper

google map scraper. extract title, phone, address, latitude and longitude, category, website URL, rating, reviews number, email, active_hours, reviews and first picture of listing

dynamic-website google-map-scraper google-map-scraping google-maps-scraper google-maps-scraper-python google-maps-scraping playwright playwright-python python3 web-crawling web-scraping webcrawling webscraping

Last synced: 14 Apr 2025

https://github.com/sxoxgxi/webcrawler

A multi threaded web crawler

crawler python webcrawling

Last synced: 20 Mar 2025

https://github.com/kalana99/url_listing

[Test] Web Crawling tool to list down URLs for a given domain and build a tree structure

python3 selenium-webdriver tree-structure webcrawling

Last synced: 06 Apr 2025

https://github.com/ajaythorve/data-structures-and-algorithms

compilation of all data structures and algorithms I implement in Java

algorithm-challenges ctci datastructures graph-algorithms java webcrawling

Last synced: 30 Mar 2025

https://github.com/splorg/sage

A scraper to get every quote from a book off of Goodreads.

books crawler datamining goodreads goodreads-data python scraper scrapy webcrawling webscraping

Last synced: 14 Mar 2025

https://github.com/glasswalk3r/app-spamcupng

Perl web crawler for finishing SpamCop.net reports automatically

perl spam spamcop-reports webcrawling

Last synced: 16 Feb 2025

https://github.com/kardbord/web-crawler

A very simple web crawler written in Go

go golang webcrawler webcrawling

Last synced: 18 Mar 2025

https://github.com/mominurr/cars.com

Cars.com Scraper – Extracts car listings (make, model, year, price, seller details) from cars.com using Selenium and BeautifulSoup, saving data in CSV format.

datascraping pandas python scraper scraping webcrawler webcrawling webscraping

Last synced: 25 Mar 2025

https://github.com/mominurr/social-media-scraping

Social Media Scraping – Scrapes data from TikTok, LinkedIn, Facebook, and Twitter (X.com), including user profiles, posts, engagement metrics, and comments.

datascraping pandas python scraper scraping selenium webcrawler webcrawling webscraping