Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with crawling-python

A curated list of projects in awesome lists tagged with crawling-python .

https://github.com/D4Vinci/Scrapling

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

ai ai-scraping automation crawler crawling crawling-python data data-extraction hacktoberfest playwright python python3 scraping selectors stealth web-scraper web-scraping web-scraping-python webscraping xpath

Last synced: 13 May 2025

https://github.com/d4vinci/scrapling

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

ai ai-scraping automation crawler crawling crawling-python data data-extraction hacktoberfest playwright python python3 scraping selectors stealth web-scraper web-scraping web-scraping-python webscraping xpath

Last synced: 13 May 2025

https://github.com/shaohua0116/ICLR2019-OpenReviewData

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

crawler crawling-python openreview tutorial

Last synced: 27 Nov 2024

https://github.com/marshalx/telegram-crawler

🕷 Automatically detect changes made to the official Telegram sites, clients and servers.

crawler crawling crawling-python parser telegram telegram-org telegram-updates

Last synced: 16 May 2025

https://github.com/MarshalX/telegram-crawler

🕷 Automatically detect changes made to the official Telegram sites, clients and servers.

crawler crawling crawling-python parser telegram telegram-org telegram-updates

Last synced: 15 May 2025

https://github.com/wwwwwydev/crawlipt

The script for selenium in python. Make automated testing easier! 使用json脚本驱动selenium

crawling-python crawlipt reptile selenium selenium-python test-automation testing-tools

Last synced: 07 Apr 2025

https://github.com/wwwwwydev/crawlist

A universal solution for web crawling lists. 抓取网页列表的通用解决方案

crawl crawler crawler-python crawling-python crawlist python reptile

Last synced: 01 May 2025

https://github.com/xishandong/android_reverse

此项目分享安卓逆向的实战案例以及学习笔记，适合新手学习，随着作者逐渐变成大神，这个仓库也会适合大神学习~

android-application crawling-python reverse-engineering

Last synced: 19 Dec 2024

https://github.com/galarzaa90/tibia.py

API to parse tibia.com content into python objects.

beautifulsoup crawling-python python python3 tibia webcrawling

Last synced: 06 Apr 2025

https://github.com/helviojunior/filecrawler

File Crawler index files and search hard-coded credentials

crawler crawling-python elasticsearch leaks leaks-scanner

Last synced: 08 Apr 2025

https://github.com/jashakouri/time.ir-crawling

api getting iran holidays per years or months

crawling-python holidays-api jalali timeir

Last synced: 06 May 2025

https://github.com/anzo52/osintbeast

Combining (mostly) Python OSINT tools into a single framework with support for sqlite3 database, currently working on mysql support.

crawling-python information-gathering information-retrieval infosec open-source-intelligence open-source-research osint osint-framework osint-python socmint

Last synced: 22 Apr 2025

https://github.com/deepmancer/advanced-recommender-system

Advance information retrieval system that combines advanced indexing, machine learning, and personalized search to enhance academic research and document discovery.

bigram-model collaborative-filtering crawling-python fine-tuning information-retrieval language-model natural-language-processing nlp positional-indexing pytorch recommender-system selenium spelling-correction tokenization transformers vectorization

Last synced: 01 Apr 2025

https://github.com/kisoo95/naver-cafe-crawling-ver240115

Naver cafe crawling using search keywords / 키워드 검색 위주 네이버 카페 크롤링 코드입니다

beautifulsoup crawling crawling-python naver naver-cafe selenium

Last synced: 23 Apr 2025

https://github.com/alanjumeaucourt/tca-net

Discord bot that send notification 10 minutes before classes as a remind of the classroom and which professor

crawling-python discord-bot python scraping-python

Last synced: 11 Jun 2025

https://github.com/pnguyen215/instagram-crawler

Instagram Crawler is a Python script to download posts from a specified Instagram account.

crawler crawling-python instagram instagram-crawler scraper scraping-python scraping-websites scrapper scrapy-crawler

Last synced: 01 Mar 2025

https://github.com/muhfalihr/pycrawlconnect

Project to connect crawled data to Kafka and monitor using elasticsearch. Still under development, PLEASE UNDERSTAND. Haha:)

apache-kafka beginners books crawl crawler crawling crawling-python elasticsearch indonesian instagram movie news python-script python3 social-media twitter x

Last synced: 06 Apr 2025

https://github.com/minsgy/likelion_apply_crawling

🦁멋쟁이사자처럼 지원자 크롤링 프로젝트

crawling-python opensource-apps selenium

Last synced: 04 May 2025

https://github.com/amirdora/data-scraping-and-fundamental-analysis

Stock crawling and analysis using python with panda, matplotlib and scrapy library. Stock analysis methods taken from the book "Warren Buffett and the Interpretation of Financial Statements" by Mary Buffet

crawling-python data-mining data-science stock valuation visualization

Last synced: 13 Mar 2025

https://github.com/muhfalihr/pyxdtelebot

PyXDTeleBot is a Telegram bot created using the Python programming language, specifically designed to facilitate the seamless sharing of media such as photos and videos from Twitter user posts.

crawler crawling crawling-python crontab python3 telegram-bot telegram-bot-api twitter twitter-api x

Last synced: 06 Apr 2025

https://github.com/captain-woof/zhi-zhu

Zhi-Zhu is a multithreaded spidering script that recursively searches base webpages and all urls appearing in it, for specific (regex) words.

crawler crawler-python crawling-python python3

Last synced: 20 Feb 2025

https://github.com/gabrielmazzotta/web-scrapping-real-state

Web scrapping repo for Real State business, including Cloudflare handling.

cloudflare cloudscraper crawling-python real-state scrapy scrapy-playwright web-scraping

Last synced: 26 Feb 2025

https://github.com/mukhoplus/project

Web Crawling, PyQt, Android App(music player), Kakaotalk Bot

android crawling-python javascript kakaotalk-bot korean-letters leagueoflegends opgg pyqt5

Last synced: 31 Mar 2025

https://github.com/patrik-fredon/python_wallpaper_crawler

Wallpaper Crawler is an advanced web scraping tool designed to crawl websites and download high-resolution wallpapers.

crawler crawling-python image image-recognition images python scraping-websites scrapper selenium-python uv

Last synced: 16 Mar 2025

https://github.com/moe131/webcrawler

Python web crawler designed to scrape websites

crawler crawling-python python python-crawler scraping simhash web-crawler

Last synced: 09 Apr 2025

https://github.com/nxhawk/crawl-data-python

Web crawling (or data crawling) is used for data extraction and refers to collecting data from either the world wide web or, in data crawling cases – any document, file, etc . Traditionally, it is done in large quantities. Therefore, usually done with a crawler agent.

crawling crawling-python get-data python3

Last synced: 01 Mar 2025

https://github.com/william1nguyen/carlist-crawler-python

A Pipeline for extracting data from Carlist.my and load to ElasticSearch

crawling-python elasticsearch etl-pipeline scrapy

Last synced: 22 Feb 2025

https://github.com/m-taghizadeh/dollar_rial_price_dataset

In this dataset, the price of the dollar to the Iranian rial in the years 2011 to 2023 has been collected by our crawler.

crawling-python deep-learning finance huggingface kaggle lstm python rnn selenium time-series time-series-prediction transformer

Last synced: 11 May 2025

https://github.com/surister/scrupy

Python library to create web Crawlers which aims to be powerful yet simple.

crawler crawling-framework crawling-python http library python scraping

Last synced: 12 Mar 2025