Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dbeley/gsmarena-scraper
Scraper for gsmarena.com (deprecated)
https://github.com/dbeley/gsmarena-scraper
docker gsmarena tor
Last synced: about 1 month ago
JSON representation
Scraper for gsmarena.com (deprecated)
- Host: GitHub
- URL: https://github.com/dbeley/gsmarena-scraper
- Owner: dbeley
- License: mit
- Archived: true
- Created: 2019-03-16T00:42:23.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2023-11-27T20:49:57.000Z (about 1 year ago)
- Last Synced: 2024-08-03T02:10:27.374Z (4 months ago)
- Topics: docker, gsmarena, tor
- Language: Python
- Homepage:
- Size: 64.5 KB
- Stars: 25
- Watchers: 3
- Forks: 14
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-starred - gsmarena-scraper - Scraper for gsmarena.com (Python)
README
# gsmarena_scraper
**DEPRECATED: even with tor, the rate limiting of gsmarena is too sensitive for the script to complete properly. Use it at your own risk.**
This script extract the mobile specs from all the phones available in gsmarena.com to a csv file (+ one for each brand).
To avoid spam detection, run with TOR (see below).
## Requirements
- python + pip
- docker + docker-compose
## Installation
Clone the repository:
```
git clone https://github.com/dbeley/gsmarena-scraper
cd gsmarena-scraper
```
Install the python dependencies:
```
pip install requests beautifulsoup4 lxml pandas pysocks stem
```
If you prefer, you can also install the requirements in a virtual environment with pipenv (in order to run the python script, you will need to use `pipenv run python gsmarena-scraper.py` instead of `python gsmarena-scraper.py`):
```
pipenv install
```
## Usage
Run the docker container containing the tor proxy (you can tweak the torrc configuration file if you want, but the defaults should be good):
```
docker-compose up -d
```
Run the script:
```
python gsmarena-scraper.py
```
After completion, you can stop the docker container with `docker-compose down`.
## Help
```
python gsmarena_scraper.py -h
```
```
usage: gsmarena-scraper.py [-h] [--debug]
Scraper gsmarena.
optional arguments:
-h, --help show this help message and exit
--debug Display debugging information
```
## Files exported
The exported files will be placed in a folder named Exports. The `all-brands_exports.csv` will contain the data of all brands.