Crawler
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
- GitHub: https://github.com/topics/crawler
- Wikipedia: https://en.wikipedia.org/wiki/Web_crawler
- Last updated: 2026-07-01 00:06:39 UTC
- JSON Representation
https://github.com/sammwyy/craw
a website-crawler library for nodejs
crawler crawlers html javascript library node nodejs nodejs-module npm npm-module parser spider website
Last synced: 29 Apr 2026
https://github.com/karantyagi/web-crawler
BFS and DFS implementations for a wikipedia crawler
Last synced: 05 Jun 2026
https://github.com/priyakdey/github-api-crawler
A crawler to crawl and save the APIs found in the Public APIs github repo - https://github.com/public-apis/public-apis. Visit README for details.
Last synced: 04 May 2026
https://github.com/redco/goose-phantom-environment
Environment for Goose parser which allows to run it in PhantomJS
crawler environment goose goose-parser nodejs parse parser phantomjs scraper
Last synced: 30 Apr 2026
https://github.com/manku27/webscrapping
Crawls and scraped a website to get rental listings as per my custom needs which the website wasnt providing, and to directly scrape necessary information like Property owner's phone number for quick use.
beautifulsoup crawler python scraper
Last synced: 30 Apr 2026
https://github.com/poodle64/supacrawl
Zero-infrastructure web scraping for the terminal
cli crawler llm markdown playwright python scraper terminal web-scraping
Last synced: 04 Mar 2026
https://github.com/amirsorouri00/dsl-se
This is a MVP provided based on the "Search Engine And Data Mining" Course. The idea behind this project is the forked project which its link provided is
container crawler distributed-systems docker docker-compose elasticsearch pagerank search-engine
Last synced: 01 May 2026
https://github.com/emarifer/search-engine
A mini Google. Custom web crawler & indexer written in Golang.
crawler dashboard deep-first-search fiber-framework full-text-search golang gorm-orm htmx htmx-go hyperscript indexer inverted-index response-caching search-engine templ worker-pool
Last synced: 16 Apr 2026
https://github.com/christopher-besch/therapy_search
Compute Call Times from arztsuche-bw into a Calendar.
appointments calendar crawler gatsby therapy time-management typescript
Last synced: 01 May 2026
https://github.com/lukasherz/22fs-sc-twitter-crawler
used for a research project in social computing @ uzh (fs22)
crawler crawling database twitter twitter-api-v2
Last synced: 02 May 2026
https://github.com/sarvarbekup/searchin_v1
Seach system
backend crawler crawling crawling-go frontend go golang nextjs search-engine searching
Last synced: 16 Apr 2026
https://github.com/muhfalihr/pycrawlconnect
Project to connect crawled data to Kafka and monitor using elasticsearch. Still under development, PLEASE UNDERSTAND. Haha:)
apache-kafka beginners books crawl crawler crawling crawling-python elasticsearch indonesian instagram movie news python-script python3 social-media twitter x
Last synced: 04 May 2026
https://github.com/alquipo/dragoongames-api
Dragoon Games Store - API (E-commerce)
crawler e-commerce-project graphql jsdom nodejs postgresql strapi-cms stripe
Last synced: 16 Apr 2026
https://github.com/par7133/splash-bot-crawler
Splash Bot creates splash on the fly of your websites - GPL License 🔥
bot crawler gallery open-source opensource php splash
Last synced: 10 May 2026
https://github.com/octcarp/sustech_cs209a-java2_f24_proj
(Spring Boot + Vue3) Stack Overflow data crawling and visualization: Our project of CS209A 2024 Fall: Computer System Design and Applications A (a.k.a. Java 2), SUSTech. Taught by Dr. Yida Tao @yidatao .
crawler spring-boot stackexchange sustech visualization
Last synced: 10 May 2026
https://github.com/enansari/guess-price-car
Car price estimation based on the information of a car sales site | final project of Maktabkhooneh | حدس قیمت خودرو با ماشین لرنینگ | پروژه نهایی مکتبخونه
crawler jadi machine-learning maktabkhoone maktabkhooneh python
Last synced: 10 May 2026
https://github.com/nagilum/focus
Simple CLI tool, written in C#, to crawl a site and log the responses.
cli crawl crawler csharp playwright
Last synced: 24 Apr 2026
https://github.com/ryanchao2012/okbot
A conversation retrieval engine based on PTT corpus
Last synced: 24 Apr 2026
https://github.com/ssv445/js-rendering-proxy-docker
JS Rendering Proxy API to Handle JS Website in Your Crawler.
Last synced: 25 Apr 2026
https://github.com/alizdavoodi/mcpdocsearch
This project provides a toolset to crawl websites, generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.
Last synced: 13 May 2026
https://github.com/tsoliangwu0130/ex-dividend-date-notification
crawler email-notification python3 stock-market vanguard
Last synced: 11 Jun 2026
https://github.com/kfess/cf_dashboard_crawler
Crawl Codeforces API and web site.
api codeforces competitive-programming crawler github-actions python
Last synced: 25 Apr 2026
https://github.com/leomaurodesenv/smm-maker-profile
A package to fetching the maker profile - Super Mario Maker
crawler javascript json mario-maker nodejs
Last synced: 08 May 2026
https://github.com/scrape-do/python-sample
Best Rotating Proxy & Scraping API Alternative. Python Example.
capcha-solver captcha crawler crawlers data-mining data-science data-scraping free freeproxy freeproxylist proxy proxy-list rotating-proxy scraper scraping scraping-api scraping-tool web-scrapers web-scrapping
Last synced: 12 Jun 2026
https://github.com/dean9703111/ithelp_total_count
計算 IT邦幫忙文章的瀏覽/Like/留言總數
crawler ithelp total-likes total-responses total-views
Last synced: 07 May 2026
https://github.com/m98/email-extractor-crawler
A minimal Node crawler to find emails used inside a website content, this crawler follows links in the website and tries to find an email in the content of the page
crawler email javascript lowdb node-crawler nodejs scraper
Last synced: 25 Apr 2026
https://github.com/siddhantsharma24/web-scraping-application-jsoup
A web crawler application made using Jsoup Library for scraping information from a webpage.
crawler java jsoup jsoup-crawler scraping
Last synced: 13 Jun 2026
https://github.com/siddhantsharma24/stock-market-scraper-jsoup
A web crawler application made using Jsoup Library for scraping Stock Market data from a webpage.
crawler java jsoup jsoup-html jsoup-library web-scraping
Last synced: 13 Jun 2026
https://github.com/suddi/fundscraper
Collection of web crawlers to scrape fund data using Scrapy
Last synced: 06 Jun 2026
https://github.com/ndoolan360/go-crawler
A simple web crawling program written in Go in an afternoon. 🕷️🕸️
afternoon-project crawler scraper
Last synced: 21 Apr 2026
https://github.com/openpj/manifoldcf-sdk
Apache ManifoldCF SDK is a Maven project focused on helping developers to extend ManifoldCF with new connectors and extensions
apache crawler docker ecm extensions integrations manifoldcf migration sdk search
Last synced: 07 May 2026
https://github.com/danielemoraschi/go-sitemap-common
Simple GO sitemap generator and crawler.
crawler golang sitemap sitemap-generator
Last synced: 17 Jun 2026
https://github.com/basemax/buskool.com-crawler
This repository contains a PHP-based crawler and scraper designed to fetch and download all product data from the Buskool website (باسکول). The crawler is designed to handle large-scale data scraping efficiently and stores the collected data in JSON format.
buskool buskoolcom crawler crawler-php php php-crawler
Last synced: 03 May 2026
https://github.com/sinkaroid/webnovelcrawler
Simple PHPcurl and getRequest to grab Light Novel and WebNovel, then create parser with DOMpdf.
Last synced: 18 Jun 2026
https://github.com/bl4ck0w1/swmap
Service Worker security scanner that maps scope, caching, routes & Workbox behavior into actionable risk static-first with optional AST/headless.
app-sec bug-bounty crawler dynamic-analysis penetration-testing playwright pwa recon security-tools service-worker static-analysis web-security work-box
Last synced: 21 Apr 2026
https://github.com/alatiera/ellinofreneia-crawler
Crawler of ellinofreneianet.gr for offline content consumption
Last synced: 19 Jun 2026
https://github.com/chen0040/ios-stock-tracker
Stock tracker implemented using Objective-C for iOS
crawler ios-app objective-c stock-prices
Last synced: 20 Jun 2026
https://github.com/tsaohucn/crawler_fb_group
This is crawler use selenium for facebook groups
crawler facebook-groups rails ruby
Last synced: 27 Apr 2026
https://github.com/dan3002/imdb-crawler
A powerful Python-based web crawler that collects comprehensive movie information from IMDb using both GraphQL API and web scraping techniques. This tool can gather detailed movie data including basic information, reviews, and ratings for any type of movies based on customizable filters.
crawler imdb imdb-dataset selenium
Last synced: 27 Apr 2026
https://github.com/luthfan98/screenshoot-crawl-web-automation
Automated full-website screenshot capture and internal link crawler using Puppeteer. Organized output with full-page screenshots, link discovery, retries, and AEST timestamp logs.
automation crawler nodejs puppeteer puppeteer-screenshot web-scraping
Last synced: 27 Apr 2026
https://github.com/rebrowser/seatgeek-dataset
SeatGeek ticket marketplace data: events with taxonomy and schedule status, listings with section/row and deal bucket, 15K+ performers, 12K+ venues with capacity and coordinates. Updated daily.
concerts crawler data-collection data-science dataset deal-score events open-data scraper seatgeek sports ticket-prices tickets web-scraping
Last synced: 03 May 2026
https://github.com/saketh7382/smartcrawler
Package for crawling items from webpages and store them as json file
crawler crawler-python open-source pip python3 scraper selenium selenium-webdriver webdriver-manager
Last synced: 28 Apr 2026
https://github.com/juangesino/gazette
A personal news aggregator application using Meteor.
crawler meteor meteorjs news news-aggregator news-feed scraper
Last synced: 17 Apr 2026
https://github.com/nb3n/sitemap-indexer
CLI tool for generating and managing sitemap index files for large websites
crawler indexing python search-engine-optimization seo sitemap sitemap-generator
Last synced: 26 Jun 2026
https://github.com/sysadmindoc/stock-video-collector
Headless browser crawler with a PyQt6 GUI for discovering, cataloging, and downloading stock video clips from Artlist, Pexels, Pixabay, Storyblocks, and more.
crawler gui pyqt6 python stock-video video
Last synced: 28 Jun 2026
https://github.com/efishery/wpi-kkp-crawler
This is crawler for fisheries price on wpi.kkp.go.id
Last synced: 29 Jun 2026
https://github.com/tctien342/simple-doc-crawler
Craw all sub page from given URL to markdown
Last synced: 03 Mar 2026
https://github.com/ma-pony/deepspider
智能爬虫工程平台 - 基于 DeepAgents + Patchright 的 AI 爬虫 Agent | Intelligent Web Scraping Platform - AI-powered Crawler Agent built on DeepAgents + Patchright
ai-agent anti-detect automation captcha crawler javascript reverse-engineering web-scraping
Last synced: 03 Apr 2026
https://github.com/949886/pixiv-crawler
Pixiv illustration info crawler to local MySQL database.
Last synced: 17 Apr 2026
https://github.com/puureya2/llm-powered-web-scraper
Big Data Web Scraper Framework, Internship Project
asyncio crawl4ai crawler crush-cli csv deepseek-r1 gemini-2-5-flash gemini-api gzip json llm pandas pydantic python selenium-webdriver seleniumwire web-scraper
Last synced: 29 Apr 2026
https://github.com/pxlrbt/website-diff
Utility tool that bundles a crawler and BackstopJS for visual regression testing.
backstopjs crawler visual-regression-testing
Last synced: 29 Apr 2026
https://github.com/andrew-ld/wowroms-downloader
download all roms from wowroms
aiohttp asyncio crawler python3
Last synced: 17 Jan 2026
https://github.com/shimech/pokemon-db-maker
Webクローリングでポケモン図鑑を生成
beautifulsoup crawler docker pokemon scraper
Last synced: 25 Jan 2026
https://github.com/lucasbotang/project_financial_markets_text_mining
Predict stock market movement based on news
crawler data-science natural-language-processing python
Last synced: 21 May 2026
https://github.com/tasooshi/digslash
A site mapping and enumeration tool for Web applications analysis
crawler mapping sitemap spider
Last synced: 08 Apr 2026
https://github.com/pymarcus/webscrapingiii
Um crawler que pega produtos em uma lista e percorre as páginas do mercado livre selecionando preços, o nome e o link para acessá-los.
crawler mercadolivre python webscraping
Last synced: 15 Sep 2025
https://github.com/vietdoo/sg-property-hub
SG Property Hub is a comprehensive platform for managing and analyzing property data.
airflow celery-redis crawler etl etl-pipeline fastapi minio mongodb nextjs postgresql s3 spark webscraping
Last synced: 08 Apr 2026
https://github.com/fa7ad/aiub-notes-dl
Download all notes from AIUB's portal
Last synced: 12 Mar 2025
https://github.com/ryanking13/bellorin
Multi-threaded Social Media Crawler 🔍
crawler instagram social-media
Last synced: 29 Jun 2025
https://github.com/vlad1kudelko/2023.08.15-scraping
Crawler of cooking sites
cloudflare cloudflare-bypass crawler docker parsing python scraping selenium undetected-chromedriver
Last synced: 08 Apr 2026
https://github.com/krishealty/whoknows
All in One Advanced and Detailed Web Scanner with over 1000 plug-ins.
bug-bounty bypass crawler enumeration ethical-hacking footprinting hacking hacking-tool intelligence-gathering javascript offensive-security osint pentesting pentesting-tools security-tools subdomain-enumeration vulnerability-analysis vulnerability-detection web-application-security web-reconnaissance
Last synced: 11 Apr 2026
https://github.com/nelcifranmagalhaes/web_crawler
A web crawler for all Naruto characters
anime beautifulsoup characters crawler naruto python
Last synced: 14 Jul 2025
https://github.com/khdxsohee/email-miner-pro
EMail Miner Pro is designed specifically for professionals scraping data from search engines like Google, ensuring that generic emails (e.g., Gmail, Yahoo) are correctly linked to their business websites found on the page.
chrome crawler crawling email email-extractor extension-chrome lead-generation miner scraper
Last synced: 03 Feb 2026
https://github.com/muhfalihr/pyxdtelebot
PyXDTeleBot is a Telegram bot created using the Python programming language, specifically designed to facilitate the seamless sharing of media such as photos and videos from Twitter user posts.
crawler crawling crawling-python crontab python3 telegram-bot telegram-bot-api twitter twitter-api x
Last synced: 06 Apr 2025
https://github.com/ambersun1234/lotto_crawler
web crawler for fetching Taiwan lottery history data
Last synced: 15 Jun 2025
https://github.com/sonhm3029/crawl-data-bot
This project making a base crawl data from web bot, include text data and images data
crawler google medical vietnamese
Last synced: 08 Mar 2026
https://github.com/jorgeparavicini/medalytik-python
Python crawlers for a job mediation firm
Last synced: 07 Jul 2025
https://github.com/moontai0724/auto-notify-pu-courses-quota
A small crawler to fetch remains quota of a list of courses in Providence University every 2 to 10 minutes, then send webhook when change.
Last synced: 15 May 2026
https://github.com/somehowchris/swisslos-cralwer
(WIP) Crawler to access the current and history numbers of swisslos
crawler euromillions lotto rust swisslos
Last synced: 22 Mar 2025
https://github.com/deventerprisesoftware/scrapi-sdk-dotnet
The only web scraping service you'll ever need that offers advanced features that are simple to use for efficient data extraction.
browser-automation crawler scraper-api web-scraping webscraper
Last synced: 22 May 2026
https://github.com/thomashirtz/douban-crawler
A simple crawler for retrieving information about movies or TV shows from the famous www.douban.com website.
Last synced: 14 May 2025
https://github.com/dimitar0528/crawlitics
An AI-powered Next.js and Python-based ecommerce web crawler, scraper and data-analyst platform that transforms scattered product data into clear market insights.
crawler nextjs product-analysis python scraper
Last synced: 08 Sep 2025
https://github.com/orafaelfragoso/itunes-crawler
Retrieves information about an artist by crawling the iTunes API and iTunes Page
Last synced: 31 Jul 2025
https://github.com/ccrashzer0/web_crawler
A python based web crawler
crawler internet python python3 webcrawler
Last synced: 22 Mar 2025
https://github.com/im-perativa/public_crawler
A collection of crawler project for Indonesia dataset
crawler indonesia indonesia-api scrapy
Last synced: 20 Mar 2025
https://github.com/eea/eea-crawler
EEA Crawler contains the tasks (DAGs) used by Apache Airflow to index content from various EEA-Eionet websites into a central Elasticsearch (aka content hub).
airflow-dags crawler elasticsearch etl-pipeline indexing
Last synced: 20 May 2026
https://github.com/supratikchatterjee16/serp_bot
A generic SERP bot, that can be used with just about any search engine.
bot crawler python requests scraping search serp user-agent-spoofer
Last synced: 14 Dec 2025
