https://github.com/slickcharmer/web-scraping-beautifulsoup-selenium

Web scraping using beautiful-soup module
https://github.com/slickcharmer/web-scraping-beautifulsoup-selenium

beautiful-soup python scraping selenium

Last synced: 28 days ago
JSON representation

Web scraping using beautiful-soup module

• Host: GitHub
• URL: https://github.com/slickcharmer/web-scraping-beautifulsoup-selenium
• Owner: slickcharmer
• Created: 2024-03-14T01:23:25.000Z (11 months ago)
• Default Branch: main
• Last Pushed: 2024-03-14T01:23:56.000Z (11 months ago)
• Last Synced: 2024-11-15T01:18:43.873Z (3 months ago)
• Topics: beautiful-soup, python, scraping, selenium
• Language: Python
• Homepage:
• Size: 3.57 MB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Web Scraping

Web scraping is a very powerful tool to learn for any data professional. With web scraping the entire internet becomes your database. In this repository how to parse a web page into a data file (csv) using a Python package called BeautifulSoup Two ways to extract data from a website:

## Use the API of the website (Best way) The data on the websites are unstructured,

Sadly, not all websites provide an API

## Web Scraping: Web scraping is an automated method used to extract useful information from the websites focuses on the transformation of unstructured data (HTML format) on the web into structured data.

**STEPS**: To extract data using web scraping with python,you need to follow these basic steps:

* Find the URL that you want to scrape

* Check wheather is it legal to scrap from that website Goto www.URL/robots.txt if you are using Scrapy you no need to worry because it automatically allow only Legal links. in Settings.py ROBOTSTXT_OBEY=False

* Inspecting the Website

* Find the data you want to extract

* Write the code

* Run the code and extract the data

* Store the data in the required format

## Need of Web Scraping

* **Price Comparison**: Services such as ParseHub use web scraping to collect data from online shopping websites and use it to compare the prices of products.

* **Email address gathering**: Many companies that use email as a medium for marketing, use web scraping to collect email ID and then send bulk emails.

* **travel recommendation**: Scraping a few travel recommendation sites, pull out comments about various do to things and see which property is getting a lot of positive responses from the users! The list of use cases is endless.

* **Social Media Scraping**: Web scraping is used to collect data from Social Media websites such as Twitter to find out what’s trending.

* **Research and Development**: Web scraping is used to collect a large set of data (Statistics, General Information, Temperature, etc.) from websites, which are analyzed and used to carry out Surveys or for R&D.

* **Job listings**: Details regarding job openings, interviews are collected from different websites and then listed in one place so that it is easily accessible to the user.

## Is Web Scraping legal?

To know whether a website allows web scraping or not, you can look at the website’s “robots.txt” file. You can find this file by appending “/robots.txt” to the URL that you want to scrape.