https://github.com/essiencodecraft/webscraping_using_python
https://github.com/essiencodecraft/webscraping_using_python
Last synced: 12 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/essiencodecraft/webscraping_using_python
- Owner: EssienCodeCraft
- Created: 2024-12-10T16:42:30.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-12-10T17:54:09.000Z (over 1 year ago)
- Last Synced: 2025-02-13T09:36:50.588Z (about 1 year ago)
- Language: Jupyter Notebook
- Size: 269 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# WebScraping using Python(BeautifulSoup & Scrapy)
- BeautifulSoup was used to scrape the content of the website including the lxml parser
- Requests was used to get the website URL
- Some transformation was done i.e. creating new columns that stores specific information of the weather needed
- Pandas was used to store the scraped data into a dataframe
- The data was transformed and loaded into Json and CSV file format
- SQLAlchemy was used to create a PostgreSQL engine and connection to load the data into PostgreSQL database
- [DataFrame stored in Database Weather](https://user-images.githubusercontent.com/5301791/137428662-06a7fbad-047e-436a-86f7-abca0dbdc8ed.png)
- [DataFrame stored in Database Weather in Schema forecasts](https://user-images.githubusercontent.com/5301791/137428668-0fe365f7-9c22-4fd1-8e0e-03f94f68d1b5.png)
- Regex pattern was used to extract specific data from the Tonaton web page
- Scrapy framework was used to scrape a Shop website extracted the name, price and link of the items and stored in CSV and JSON.