Crawler | Ecosyste.ms: Awesome

https://github.com/juangesino/gazette

A personal news aggregator application using Meteor.

crawler meteor meteorjs news news-aggregator news-feed scraper

Last synced: 17 Apr 2026

https://github.com/ma-pony/deepspider

智能爬虫工程平台 - 基于 DeepAgents + Patchright 的 AI 爬虫 Agent | Intelligent Web Scraping Platform - AI-powered Crawler Agent built on DeepAgents + Patchright

ai-agent anti-detect automation captcha crawler javascript reverse-engineering web-scraping

Last synced: 03 Apr 2026

https://github.com/949886/pixiv-crawler

Pixiv illustration info crawler to local MySQL database.

crawler mysql pixiv

Last synced: 17 Apr 2026

https://github.com/yassilah/nuxt-crawler

Automatic crawler & search for Nuxt SSG.

algolia crawler nuxt search ssg

Last synced: 17 Apr 2026

https://github.com/davideferre/covid19-data-crawler-ita

Covid 19 italian data crawler

coronavirus covid19 crawler hacktoberfest hacktoberfest2021 python

Last synced: 03 Jun 2026

https://github.com/dizys/weibo-crawler

A nodejs weibo crawler

crawler nodejs typescript weibo-spider

Last synced: 19 Apr 2026

https://github.com/yidinghan/bond-crawler

crawler sse

Last synced: 04 Jun 2026

https://github.com/karantyagi/web-crawler

BFS and DFS implementations for a wikipedia crawler

beautifulsoup crawler

Last synced: 05 Jun 2026

https://github.com/chenty2333/tiktok-youtube_commentscraper

This tool allows you to collect public comments from TikTok and YouTube videos, either via direct video URLs or keyword-based search. It's useful for data analysis, opinion mining, and building datasets for machine learning tasks.一个轻量级的 TikTok 与 YouTube 评论爬虫工具，支持通过视频链接或关键词批量获取评论数据，适用于情感分析、文本挖掘、机器学习等数据收集任务。

comment crawler nlp scraper sentiment-analysis tiktok youtube

Last synced: 20 Apr 2026

https://github.com/bl4ck0w1/swmap

Service Worker security scanner that maps scope, caching, routes & Workbox behavior into actionable risk static-first with optional AST/headless.

app-sec bug-bounty crawler dynamic-analysis penetration-testing playwright pwa recon security-tools service-worker static-analysis web-security work-box

Last synced: 21 Apr 2026

https://github.com/ndoolan360/go-crawler

A simple web crawling program written in Go in an afternoon. 🕷️🕸️

afternoon-project crawler scraper

Last synced: 21 Apr 2026

https://github.com/krishpranav/gocralwer

A awsome crawler made in go

crawler

Last synced: 24 Apr 2026

https://github.com/nagilum/focus

Simple CLI tool, written in C#, to crawl a site and log the responses.

cli crawl crawler csharp playwright

Last synced: 24 Apr 2026

https://github.com/ryanchao2012/okbot

A conversation retrieval engine based on PTT corpus

chatbot crawler django ptt

Last synced: 24 Apr 2026

https://github.com/ssv445/js-rendering-proxy-docker

JS Rendering Proxy API to Handle JS Website in Your Crawler.

crawler proxy puppeteer

Last synced: 25 Apr 2026

https://github.com/kfess/cf_dashboard_crawler

Crawl Codeforces API and web site.

api codeforces competitive-programming crawler github-actions python

Last synced: 25 Apr 2026

https://github.com/m98/email-extractor-crawler

A minimal Node crawler to find emails used inside a website content, this crawler follows links in the website and tries to find an email in the content of the page

crawler email javascript lowdb node-crawler nodejs scraper

Last synced: 25 Apr 2026

https://github.com/suddi/fundscraper

Collection of web crawlers to scrape fund data using Scrapy

crawler funds scraper scrapy

Last synced: 06 Jun 2026

https://github.com/rbkgh/dailytext-crawler

Crawl jw.org to retrieve daily text

crawler dailytext java jsoup jw

Last synced: 06 Jun 2026

https://github.com/cothema/nlp-workers

crawler nlp

Last synced: 26 Apr 2026

https://github.com/tsaohucn/crawler_fb_group

This is crawler use selenium for facebook groups

crawler facebook-groups rails ruby

Last synced: 27 Apr 2026

https://github.com/dan3002/imdb-crawler

A powerful Python-based web crawler that collects comprehensive movie information from IMDb using both GraphQL API and web scraping techniques. This tool can gather detailed movie data including basic information, reviews, and ratings for any type of movies based on customizable filters.

crawler imdb imdb-dataset selenium

Last synced: 27 Apr 2026

https://github.com/saketh7382/smartcrawler

Package for crawling items from webpages and store them as json file

crawler crawler-python open-source pip python3 scraper selenium selenium-webdriver webdriver-manager

Last synced: 28 Apr 2026

https://github.com/puureya2/llm-powered-web-scraper

Big Data Web Scraper Framework, Internship Project

asyncio crawl4ai crawler crush-cli csv deepseek-r1 gemini-2-5-flash gemini-api gzip json llm pandas pydantic python selenium-webdriver seleniumwire web-scraper

Last synced: 29 Apr 2026

https://github.com/pxlrbt/website-diff

Utility tool that bundles a crawler and BackstopJS for visual regression testing.

backstopjs crawler visual-regression-testing

Last synced: 29 Apr 2026

https://github.com/fi1a/crawler

PHP crawler

crawler php

Last synced: 29 Apr 2026

https://github.com/luukalindgren/jobposts-utu

Web site for a database that holds job post data of IT jobs.

crawler docker fastapi mariadb react virtual-machine

Last synced: 29 Apr 2026

https://github.com/zukahai/formosa-views

View Formosa employee profile, salary, bonus year

bonus-year crawler css formosa html javascript nodejs python salary views

Last synced: 29 Apr 2026

https://github.com/sammwyy/craw

a website-crawler library for nodejs

crawler crawlers html javascript library node nodejs nodejs-module npm npm-module parser spider website

Last synced: 29 Apr 2026

https://github.com/redco/goose-phantom-environment

Environment for Goose parser which allows to run it in PhantomJS

crawler environment goose goose-parser nodejs parse parser phantomjs scraper

Last synced: 30 Apr 2026

https://github.com/manku27/webscrapping

Crawls and scraped a website to get rental listings as per my custom needs which the website wasnt providing, and to directly scrape necessary information like Property owner's phone number for quick use.

beautifulsoup crawler python scraper

Last synced: 30 Apr 2026

https://github.com/amirsorouri00/dsl-se

This is a MVP provided based on the "Search Engine And Data Mining" Course. The idea behind this project is the forked project which its link provided is

container crawler distributed-systems docker docker-compose elasticsearch pagerank search-engine

Last synced: 01 May 2026

https://github.com/christopher-besch/therapy_search

Compute Call Times from arztsuche-bw into a Calendar.

appointments calendar crawler gatsby therapy time-management typescript

Last synced: 01 May 2026

https://github.com/lukasherz/22fs-sc-twitter-crawler

used for a research project in social computing @ uzh (fs22)

crawler crawling database twitter twitter-api-v2

Last synced: 02 May 2026

https://github.com/linux0hat/cpp-web-crawler

Explore the web.

cpp crawler sqlite3

Last synced: 09 Jun 2026

https://github.com/ammirsm/data-grabber-cnn-twitter

Basic setup to get data from twitter and CNN with a keyword.

cnn crawler django scrapyd twitter

Last synced: 02 May 2026

https://github.com/comigor/balances

Your checking and savings accounts balances on banks and brokers.

balance bank broker crawler node

Last synced: 02 May 2026

https://github.com/basemax/buskool.com-crawler

This repository contains a PHP-based crawler and scraper designed to fetch and download all product data from the Buskool website (باسکول). The crawler is designed to handle large-scale data scraping efficiently and stores the collected data in JSON format.

buskool buskoolcom crawler crawler-php php php-crawler

Last synced: 03 May 2026

https://github.com/rebrowser/seatgeek-dataset

SeatGeek ticket marketplace data: events with taxonomy and schedule status, listings with section/row and deal bucket, 15K+ performers, 12K+ venues with capacity and coordinates. Updated daily.

concerts crawler data-collection data-science dataset deal-score events open-data scraper seatgeek sports ticket-prices tickets web-scraping

Last synced: 03 May 2026

https://github.com/wshwbluebird/trider

小型go语言爬虫框架

crawler golang

Last synced: 09 Jun 2026

https://github.com/priyakdey/github-api-crawler

A crawler to crawl and save the APIs found in the Public APIs github repo - https://github.com/public-apis/public-apis. Visit README for details.

api crawler mongo python3

Last synced: 04 May 2026

https://github.com/muhfalihr/pycrawlconnect

Project to connect crawled data to Kafka and monitor using elasticsearch. Still under development, PLEASE UNDERSTAND. Haha:)

apache-kafka beginners books crawl crawler crawling crawling-python elasticsearch indonesian instagram movie news python-script python3 social-media twitter x

Last synced: 04 May 2026

https://github.com/kevinchiu2k/kevinchiu2k.github.io

My blog is based on Hexo's Icarus theme

crawler markdown python ssh-eky youtube

Last synced: 04 May 2026

https://github.com/excaliburhan/littlenews

A news app via electron

crawler electron rss-feed

Last synced: 04 May 2026

https://github.com/ozansz/simple-web-downloader

A simple web page downloader program in C

c crawler curl libcurl web

Last synced: 06 May 2026

https://github.com/openpj/manifoldcf-sdk

Apache ManifoldCF SDK is a Maven project focused on helping developers to extend ManifoldCF with new connectors and extensions

apache crawler docker ecm extensions integrations manifoldcf migration sdk search

Last synced: 07 May 2026

https://github.com/dean9703111/ithelp_total_count

計算 IT邦幫忙文章的瀏覽/Like/留言總數

crawler ithelp total-likes total-responses total-views

Last synced: 07 May 2026

https://github.com/leomaurodesenv/smm-maker-profile

A package to fetching the maker profile - Super Mario Maker

crawler javascript json mario-maker nodejs

Last synced: 08 May 2026

https://github.com/alexzhangs/stockdb

Stock data collecting and analyzing

crawler django pandas scrapy stock tushare

Last synced: 10 May 2026

https://github.com/efishery/wpi-kkp-crawler

This is crawler for fisheries price on wpi.kkp.go.id

crawler kkp wpi

Last synced: 10 May 2026

https://github.com/par7133/splash-bot-crawler

Splash Bot creates splash on the fly of your websites - GPL License 🔥

bot crawler gallery open-source opensource php splash

Last synced: 10 May 2026

https://github.com/octcarp/sustech_cs209a-java2_f24_proj

(Spring Boot + Vue3) Stack Overflow data crawling and visualization: Our project of CS209A 2024 Fall: Computer System Design and Applications A (a.k.a. Java 2), SUSTech. Taught by Dr. Yida Tao @yidatao .

crawler spring-boot stackexchange sustech visualization

Last synced: 10 May 2026

https://github.com/enansari/guess-price-car

Car price estimation based on the information of a car sales site | final project of Maktabkhooneh | حدس قیمت خودرو با ماشین لرنینگ | پروژه نهایی مکتب‌خونه

crawler jadi machine-learning maktabkhoone maktabkhooneh python

Last synced: 10 May 2026

https://github.com/alizdavoodi/mcpdocsearch

This project provides a toolset to crawl websites, generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.

crawler mcp mcp-server

Last synced: 13 May 2026

https://github.com/tsoliangwu0130/ex-dividend-date-notification

crawler email-notification python3 stock-market vanguard

Last synced: 11 Jun 2026

https://github.com/scrape-do/python-sample

Best Rotating Proxy & Scraping API Alternative. Python Example.

capcha-solver captcha crawler crawlers data-mining data-science data-scraping free freeproxy freeproxylist proxy proxy-list rotating-proxy scraper scraping scraping-api scraping-tool web-scrapers web-scrapping

Last synced: 12 Jun 2026

https://github.com/siddhantsharma24/web-scraping-application-jsoup

A web crawler application made using Jsoup Library for scraping information from a webpage.

crawler java jsoup jsoup-crawler scraping

Last synced: 13 Jun 2026

https://github.com/siddhantsharma24/stock-market-scraper-jsoup

A web crawler application made using Jsoup Library for scraping Stock Market data from a webpage.

crawler java jsoup jsoup-html jsoup-library web-scraping

Last synced: 13 Jun 2026

https://github.com/danielemoraschi/go-sitemap-common

Simple GO sitemap generator and crawler.

crawler golang sitemap sitemap-generator

Last synced: 17 Jun 2026

https://github.com/sinkaroid/webnovelcrawler

Simple PHPcurl and getRequest to grab Light Novel and WebNovel, then create parser with DOMpdf.

crawler dompdf webnovel

Last synced: 18 Jun 2026

https://github.com/alatiera/ellinofreneia-crawler

Crawler of ellinofreneianet.gr for offline content consumption

crawler ellinofreneia

Last synced: 19 Jun 2026

https://github.com/chen0040/ios-stock-tracker

Stock tracker implemented using Objective-C for iOS

crawler ios-app objective-c stock-prices

Last synced: 20 Jun 2026

https://github.com/maxgio92/package-crawler

A package crawler for most known Linux distros

crawler go linux package

Last synced: 20 Apr 2026

https://github.com/raphaelalmeidamartins/python-tech-news

Python data science project developed js at the end of Unit 35 (Computer Science Module) of the Trybe's Web Development course

crawler crawler-python data-science pytest python

Last synced: 22 May 2026

https://github.com/curegit/nominium

個人間取引サイトの新着商品をメールなどで通知するクローラーシステム

c2c chromium crawler ecommerce firefox selenium shopping webdriver

Last synced: 12 Mar 2025

https://github.com/cameronnewman/cli.crawler

Simple cli web crawler

cli crawler golang

Last synced: 14 Jan 2026

https://github.com/mlibre/clean-web-scraper

A Node.js web scraper that extracts clean, readable content from websites - perfect for AI/LLM training datasets. Features smart crawling, Mozilla Readability integration, and organized content storage 🤖

ai artificial-intelligence clean crawler data-preprocessing dataset fine-tuning llm recursive-crawling scraper training

Last synced: 17 Mar 2025

https://github.com/jorgeparavicini/medalytik-python

Python crawlers for a job mediation firm

crawler python scrapy

Last synced: 07 Jul 2025

https://github.com/tubone24/askfm-qa-crawler

Crawl Ask.fm QA lists and create corpus for ML.

askfm chromedriver corpus-builder crawler selenium

Last synced: 14 May 2026

https://github.com/f-ca7/movie-cat

A website displaying movies

crawler golang website

Last synced: 19 Apr 2026

https://github.com/woorim960/nate.com-comments-crawler

nate.com-comments-crawler

chromedriver crawler python3 selenium

Last synced: 14 May 2026

https://github.com/dean9703111/humandesign_nodejs

用nodejs爬蟲工具將人類圖網頁上的資訊爬下來，再存到雲端的google excel

crawler googlesheetapi googlesheets nodejs

Last synced: 15 May 2026

https://github.com/elky84/stock-crawler

Naver Stock Crawler & Mock Invest

asp-net asp-net-core crawler csharp dotnet

Last synced: 18 Apr 2026

https://github.com/soulyma/web_crawler

A focused web crawler to extract and structure Arabic content from web pages. Designed for researchers, data analysts, and developers working on Arabic language datasets.

beautifulsoup4 crawler csv data json python structured-data

Last synced: 15 May 2026

https://github.com/rflcnunes/crawler_email_py

In this project I'm creating a web crawler to check email boxes and handle incoming messages.

aws-bucket aws-bucket-s3 aws-s3 crawler crawler-python email python rabbitmq

Last synced: 10 Aug 2025

https://github.com/bitscoper/bitscoper_crawler

Crawls the titles of webpages in series by number and creates a list of the available links.

crawler lister

Last synced: 27 Mar 2025

https://github.com/yowenter/career-roadmap

Oh, how I hate this living death which has swallowed all my teens, if I am cursed with any, will be worn away!

career crawler findjob job-crawler roadmap search-engine

Last synced: 10 Apr 2025

https://github.com/weaming/simple-crawler

my simple crawler

crawler

Last synced: 13 Jun 2025

https://github.com/kbychkov/simplecrawler-app

The GUI for Simplecrawler

crawler simplecrawler spider

Last synced: 12 Jun 2025

https://github.com/moontai0724/auto-notify-pu-courses-quota

A small crawler to fetch remains quota of a list of courses in Providence University every 2 to 10 minutes, then send webhook when change.

crawler javascript nodejs

Last synced: 15 May 2026

https://github.com/ambersun1234/lotto_crawler

web crawler for fetching Taiwan lottery history data

crawler python3

Last synced: 15 Jun 2025

https://github.com/fa7ad/aiub-notes-dl

Download all notes from AIUB's portal

aiub beautifulsoup4 crawler

Last synced: 12 Mar 2025

https://github.com/dimitar0528/crawlitics

An AI-powered Next.js and Python-based ecommerce web crawler, scraper and data-analyst platform that transforms scattered product data into clear market insights.

crawler nextjs product-analysis python scraper

Last synced: 08 Sep 2025