Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with crawling-python

A curated list of projects in awesome lists tagged with crawling-python .

https://github.com/D4Vinci/Scrapling

๐Ÿ•ท๏ธ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

ai ai-scraping automation crawler crawling crawling-python data data-extraction hacktoberfest playwright python python3 scraping selectors stealth web-scraper web-scraping web-scraping-python webscraping xpath

Last synced: 13 May 2025

https://github.com/d4vinci/scrapling

๐Ÿ•ท๏ธ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

ai ai-scraping automation crawler crawling crawling-python data data-extraction hacktoberfest playwright python python3 scraping selectors stealth web-scraper web-scraping web-scraping-python webscraping xpath

Last synced: 13 May 2025

https://github.com/shaohua0116/ICLR2019-OpenReviewData

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

crawler crawling-python openreview tutorial

Last synced: 19 Jul 2025

https://github.com/marshalx/telegram-crawler

๐Ÿ•ท Automatically detect changes made to the official Telegram sites, clients and servers.

crawler crawling crawling-python parser telegram telegram-org telegram-updates

Last synced: 16 May 2025

https://github.com/MarshalX/telegram-crawler

๐Ÿ•ท Automatically detect changes made to the official Telegram sites, clients and servers.

crawler crawling crawling-python parser telegram telegram-org telegram-updates

Last synced: 15 May 2025

https://github.com/wwwwwydev/crawlipt

The script for selenium in python. Make automated testing easier! ไฝฟ็”จjson่„šๆœฌ้ฉฑๅŠจselenium

crawling-python crawlipt reptile selenium selenium-python test-automation testing-tools

Last synced: 07 Apr 2025

https://github.com/wwwwwydev/crawlist

A universal solution for web crawling lists. ๆŠ“ๅ–็ฝ‘้กตๅˆ—่กจ็š„้€š็”จ่งฃๅ†ณๆ–นๆกˆ

crawl crawler crawler-python crawling-python crawlist python reptile

Last synced: 01 May 2025

https://github.com/xishandong/android_reverse

ๆญค้กน็›ฎๅˆ†ไบซๅฎ‰ๅ“้€†ๅ‘็š„ๅฎžๆˆ˜ๆกˆไพ‹ไปฅๅŠๅญฆไน ็ฌ”่ฎฐ๏ผŒ้€‚ๅˆๆ–ฐๆ‰‹ๅญฆไน ๏ผŒ้š็€ไฝœ่€…้€ๆธๅ˜ๆˆๅคง็ฅž๏ผŒ่ฟ™ไธชไป“ๅบ“ไนŸไผš้€‚ๅˆๅคง็ฅžๅญฆไน ~

android-application crawling-python reverse-engineering

Last synced: 21 Aug 2025

https://github.com/galarzaa90/tibia.py

API to parse tibia.com content into python objects.

beautifulsoup crawling-python python python3 tibia webcrawling

Last synced: 06 Apr 2025

https://github.com/helviojunior/filecrawler

File Crawler index files and search hard-coded credentials

crawler crawling-python elasticsearch leaks leaks-scanner

Last synced: 08 Apr 2025

https://github.com/jashakouri/time.ir-crawling

api getting iran holidays per years or months

crawling-python holidays-api jalali timeir

Last synced: 18 Oct 2025

https://github.com/anzo52/osintbeast

Combining (mostly) Python OSINT tools into a single framework with support for sqlite3 database, currently working on mysql support.

crawling-python information-gathering information-retrieval infosec open-source-intelligence open-source-research osint osint-framework osint-python socmint

Last synced: 22 Apr 2025

https://github.com/deepmancer/advanced-recommender-system

Advance information retrieval system that combines advanced indexing, machine learning, and personalized search to enhance academic research and document discovery.

bigram-model collaborative-filtering crawling-python fine-tuning information-retrieval language-model natural-language-processing nlp positional-indexing pytorch recommender-system selenium spelling-correction tokenization transformers vectorization

Last synced: 01 Apr 2025

https://github.com/kisoo95/naver-cafe-crawling-ver240115

Naver cafe crawling using search keywords / ํ‚ค์›Œ๋“œ ๊ฒ€์ƒ‰ ์œ„์ฃผ ๋„ค์ด๋ฒ„ ์นดํŽ˜ ํฌ๋กค๋ง ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค

beautifulsoup crawling crawling-python naver naver-cafe selenium

Last synced: 23 Apr 2025

https://github.com/alanjumeaucourt/tca-net

Discord bot that send notification 10 minutes before classes as a remind of the classroom and which professor

crawling-python discord-bot python scraping-python

Last synced: 17 Aug 2025

https://github.com/amirdora/data-scraping-and-fundamental-analysis

Stock crawling and analysis using python with panda, matplotlib and scrapy library. Stock analysis methods taken from the book "Warren Buffett and the Interpretation of Financial Statements" by Mary Buffet

crawling-python data-mining data-science stock valuation visualization

Last synced: 26 Oct 2025

https://github.com/minsgy/likelion_apply_crawling

๐Ÿฆ๋ฉ‹์Ÿ์ด์‚ฌ์ž์ฒ˜๋Ÿผ ์ง€์›์ž ํฌ๋กค๋ง ํ”„๋กœ์ ํŠธ

crawling-python opensource-apps selenium

Last synced: 29 Oct 2025

https://github.com/muhfalihr/pyxdtelebot

PyXDTeleBot is a Telegram bot created using the Python programming language, specifically designed to facilitate the seamless sharing of media such as photos and videos from Twitter user posts.

crawler crawling crawling-python crontab python3 telegram-bot telegram-bot-api twitter twitter-api x

Last synced: 06 Apr 2025

https://github.com/muhfalihr/pycrawlconnect

Project to connect crawled data to Kafka and monitor using elasticsearch. Still under development, PLEASE UNDERSTAND. Haha:)

apache-kafka beginners books crawl crawler crawling crawling-python elasticsearch indonesian instagram movie news python-script python3 social-media twitter x

Last synced: 06 Apr 2025

https://github.com/captain-woof/zhi-zhu

Zhi-Zhu is a multithreaded spidering script that recursively searches base webpages and all urls appearing in it, for specific (regex) words.

crawler crawler-python crawling-python python3

Last synced: 20 Feb 2025

https://github.com/pnguyen215/instagram-crawler

Instagram Crawler is a Python script to download posts from a specified Instagram account.

crawler crawling-python instagram instagram-crawler scraper scraping-python scraping-websites scrapper scrapy-crawler

Last synced: 01 Mar 2025

https://github.com/patrik-fredon/python_wallpaper_crawler

Wallpaper Crawler is an advanced web scraping tool designed to crawl websites and download high-resolution wallpapers.

crawler crawling-python image image-recognition images python scraping-websites scrapper selenium-python uv

Last synced: 14 Sep 2025

https://github.com/m-taghizadeh/dollar_rial_price_dataset

In this dataset, the price of the dollar to the Iranian rial in the years 2011 to 2023 has been collected by our crawler.

crawling-python deep-learning finance huggingface kaggle lstm python rnn selenium time-series time-series-prediction transformer

Last synced: 11 May 2025

https://github.com/mukhoplus/project

Web Crawling, PyQt, Android App(music player), Kakaotalk Bot

android crawling-python javascript kakaotalk-bot korean-letters leagueoflegends opgg pyqt5

Last synced: 20 Jul 2025

https://github.com/manjiro76/searcher-movie-bot

# Searcher Movie BotSearcher Movie Bot helps you find movies and TV series quickly. Just send a title, and the bot will return a link to watch it in seconds! ๐Ÿ™โœจ

async cinema crawling-python elasticsearch entertainment lex line linebot message-api mongodb mongodb-atlas nlp nodejs python rasa streaming telegram tmdb-api-integration

Last synced: 05 Jul 2025

https://github.com/moe131/webcrawler

Python web crawler designed to scrape websites

crawler crawling-python python python-crawler scraping simhash web-crawler

Last synced: 09 Apr 2025

https://github.com/surister/scrupy

Python library to create web Crawlers which aims to be powerful yet simple.

crawler crawling-framework crawling-python http library python scraping

Last synced: 12 Mar 2025

https://github.com/nxhawk/crawl-data-python

Web crawling (or data crawling) is used for data extraction and refers to collecting data from either the world wide web or, in data crawling cases โ€“ any document, file, etc . Traditionally, it is done in large quantities. Therefore, usually done with a crawler agent.

crawling crawling-python get-data python3

Last synced: 01 Mar 2025

https://github.com/william1nguyen/carlist-crawler-python

A Pipeline for extracting data from Carlist.my and load to ElasticSearch

crawling-python elasticsearch etl-pipeline scrapy

Last synced: 15 Jul 2025

https://github.com/narubrown/siteurlcheck-pyton-tool

Site URL Check Pyton Tool / ์‚ฌ์ดํŠธ URL ๋ฐ ์š”์†Œ ์ ๊ฒ€ ๋„๊ตฌ

crawling-python

Last synced: 10 Oct 2025

https://github.com/gabrielmazzotta/web-scrapping-real-state

Web scrapping repo for Real State business, including Cloudflare handling.

cloudflare cloudscraper crawling-python real-state scrapy scrapy-playwright web-scraping

Last synced: 26 Feb 2025