https://github.com/trixsec/bright-data-web-scraper
Web Scraper Using Bright Data API
https://github.com/trixsec/bright-data-web-scraper
Last synced: 11 months ago
JSON representation
Web Scraper Using Bright Data API
- Host: GitHub
- URL: https://github.com/trixsec/bright-data-web-scraper
- Owner: TrixSec
- License: mit
- Created: 2024-12-12T04:52:35.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-12-12T12:39:05.000Z (over 1 year ago)
- Last Synced: 2025-04-04T14:21:29.760Z (about 1 year ago)
- Language: Python
- Size: 31.3 KB
- Stars: 8
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
**Author**: Trix Cyrus
**Copyright**: © 2024 Trixsec Org
**Maintained**: Yes
---
# Web Scraping with BrightData API Tool
This tool allows you to perform web scraping using the BrightData API. It is designed to scrape web pages to collect information without getting blocked or flagged. You can customize the scraping process to fetch data from various web pages and handle large-scale scraping tasks efficiently.
---
## Features
1. **Marketplace**
Scrape data from online marketplace websites, including product listings, reviews, prices, and seller details.
Available In Marketplace
- Apple
- Google
- Olx
- Yapo Chile
2. **B2B (Business to Business)**
Scrape data from B2B platforms, gathering business-related information such as leads, contacts, and product offerings.
Available In B2B
- Linkedin
3. **Web**
General web scraping for a wide range of websites. Ideal for scraping publicly available information such as articles, blog posts, and other content.
Available In Web
- Martindale
4. **E-commerce**
Extract product information, pricing, stock levels, and reviews from e-commerce websites.
Available In E-commerce
- Amazon
- Google
5. **Financial**
Collect financial data such as stock prices, investment articles, cryptocurrency updates, and other financial information.
Available In Financial
- Yahoo
6. **News**
Scrape current news articles, headlines, and related information from news websites.
Available In News
- Google
- BBC
7. **Real Estate**
Extract property listings, prices, locations, and real estate-related data from real estate websites.
Available In Real-estate
- Toctoc
8. **Social Media**
Scrape public social media posts, user profiles, and social interactions.
Available In Social Media
- Instagram
- Quora
9. **Travel**
Collect information on travel sites, including hotels, flights, and tourism-related content.
Available In Travel
- Booking.com
10. **Other**
For scraping any other types of websites not covered in the categories above. This includes niche or specialized sites.
Available In Others
- Wikipedia
--------(Will Add More Features IF I WON)----------
## Requirements
- Python 3.x
- _List any other dependencies or requirements here (e.g., `requests`, `beautifulsoup4`, etc.)_
## Installation
To install and use the Web Scraping with BrightData API tool, follow these steps:
1. **Clone the repository:**
```bash
git clone https://github.com/TrixSec/Bright-Data-Web-Scraper.git
cd Bright-Data-Web-Scraper
```
2. **Install dependencies:**
```bash
pip install -r req.txt
```
## Usage
```bash
python3 main.py
```