https://github.com/dbeley/gsmarena-scraper

Scraper for gsmarena.com (deprecated)
https://github.com/dbeley/gsmarena-scraper

docker gsmarena tor

Last synced: 3 months ago
JSON representation

Scraper for gsmarena.com (deprecated)

• Host: GitHub
• URL: https://github.com/dbeley/gsmarena-scraper
• Owner: dbeley
• License: mit
• Created: 2019-03-16T00:42:23.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2023-11-27T20:49:57.000Z (12 months ago)
• Last Synced: 2024-06-16T07:31:03.147Z (5 months ago)
• Topics: docker, gsmarena, tor
• Language: Python
• Homepage:
• Size: 64.5 KB
• Stars: 25
• Watchers: 3
• Forks: 14
• Open Issues: 5
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-starred - gsmarena-scraper - Scraper for gsmarena.com (Python)

README

        
# gsmarena_scraper

**DEPRECATED: even with tor, the rate limiting of gsmarena is too sensitive for the script to complete properly. Use it at your own risk.**

This script extract the mobile specs from all the phones available in gsmarena.com to a csv file (+ one for each brand).

To avoid spam detection, run with TOR (see below).

## Requirements

- python + pip

- docker + docker-compose

## Installation

Clone the repository:

```

git clone https://github.com/dbeley/gsmarena-scraper

cd gsmarena-scraper

```

Install the python dependencies:

```

pip install requests beautifulsoup4 lxml pandas pysocks stem

```

If you prefer, you can also install the requirements in a virtual environment with pipenv (in order to run the python script, you will need to use `pipenv run python gsmarena-scraper.py` instead of `python gsmarena-scraper.py`):

```

pipenv install

```

## Usage

Run the docker container containing the tor proxy (you can tweak the torrc configuration file if you want, but the defaults should be good):

```

docker-compose up -d

```

Run the script:

```

python gsmarena-scraper.py

```

After completion, you can stop the docker container with `docker-compose down`.

## Help

```

python gsmarena_scraper.py -h

```

```

usage: gsmarena-scraper.py [-h] [--debug]

Scraper gsmarena.

optional arguments:

  -h, --help  show this help message and exit

  --debug     Display debugging information

```

## Files exported

The exported files will be placed in a folder named Exports. The `all-brands_exports.csv` will contain the data of all brands.