https://github.com/yanniskatsaros/who-scraped-my-car
Web scraping car listings with Python and BeautifulSoup4
https://github.com/yanniskatsaros/who-scraped-my-car
cars cars-dataset machine-learning python python3 statistics visualization webscraping
Last synced: 28 days ago
JSON representation
Web scraping car listings with Python and BeautifulSoup4
- Host: GitHub
- URL: https://github.com/yanniskatsaros/who-scraped-my-car
- Owner: yanniskatsaros
- Created: 2018-05-06T02:00:07.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2022-12-08T11:08:29.000Z (almost 3 years ago)
- Last Synced: 2023-03-02T22:46:33.301Z (over 2 years ago)
- Topics: cars, cars-dataset, machine-learning, python, python3, statistics, visualization, webscraping
- Language: Python
- Homepage:
- Size: 51.8 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Who Scraped My Car?
Let's build a web-scraper to download and parse car listings from AutoTrader to build a pricing model!
_____
## Requirements
The project depends on `requests`, `beautifulsoup4`, and `pydantic`. You can install the appropriate versions using the following command:
`python -m pip install -r requirements.txt`
## Usage
The project comes with a simple CLI to easily download links to URLs for car listings of a specific brand (make) in a given ZIP code. You can specify your search radius relative to your target ZIP code and limit the total number of entries to download.
To specify a car make, use one of the following "car codes" common to AutoTrader's system.
`python cars/main.py`
```
usage: main.py [-h] [--search-radius SEARCH_RADIUS] [--limit LIMIT]
{AMC,ACURA,ALFA,ASTON,AUDI,BMW,BENTL,BUGATTI,BUICK,CAD,CHEV,CHRY,DAEW,DATSUN,DELOREAN,DODGE,EAGLE,FIAT,FER,FISK,FORD,FREIGHT,GMC,GENESIS,GEO,AMGEN,HONDA,HYUND,INFIN,ISU,JAG,JEEP,KARMA,KIA,LAM,ROV,LEXUS,LINC,LOTUS,MAZDA,MINI,MAS,MAYBACH,MCLAREN,MB,MERC,MIT,NISSAN,OLDS,PLYM,PONT,POR,RAM,RR,SRT,SAAB,SATURN,SCION,SUB,SUZUKI,TESLA,TOYOTA,VOLKS,VOLVO,YUGO,SMART}
zip_code
Download car listings for a given car make, within a certain radius of a given
ZIP code
positional arguments:
{AMC,ACURA,ALFA,ASTON,AUDI,BMW,BENTL,BUGATTI,BUICK,CAD,CHEV,CHRY,DAEW,DATSUN,DELOREAN,DODGE,EAGLE,FIAT,FER,FISK,FORD,FREIGHT,GMC,GENESIS,GEO,AMGEN,HONDA,HYUND,INFIN,ISU,JAG,JEEP,KARMA,KIA,LAM,ROV,LEXUS,LINC,LOTUS,MAZDA,MINI,MAS,MAYBACH,MCLAREN,MB,MERC,MIT,NISSAN,OLDS,PLYM,PONT,POR,RAM,RR,SRT,SAAB,SATURN,SCION,SUB,SUZUKI,TESLA,TOYOTA,VOLKS,VOLVO,YUGO,SMART}
A code representing the car make
zip_code ZIP code of the area to search listings for
optional arguments:
-h, --help show this help message and exit
--search-radius SEARCH_RADIUS
A search radius (in miles) within the given ZIP code
--limit LIMIT Limit the total number of entries to search for
```
### Examples
Download URLs for all Porsche listings within 100 miles (default) of the 77056 zip code (output truncated for brevity)
`python cars/main.py POR 77056`
```
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=555304984
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=549029457
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=553937810
...
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=556808514
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=542778710
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=556608600
```
Download URLs for all Jaguar listings in the 78254 zip code for a 20 mile radius
`python cars/main.py JAG 78254 --search-radius 20`
```
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=555182522
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=555481342
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=553690777
...
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=555481327
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=551052772
https://www.autotrader.com/cars-for-sale/vehicledetails.xhtml?listingId=555872157
```
This is a work in progress so stay tuned for more updates in the future!