https://github.com/ameshkov/webstorescraper
https://github.com/ameshkov/webstorescraper
Last synced: 8 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/ameshkov/webstorescraper
- Owner: ameshkov
- Created: 2017-12-14T11:04:52.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2023-01-07T19:56:12.000Z (over 3 years ago)
- Last Synced: 2025-04-04T02:01:29.560Z (about 1 year ago)
- Language: JavaScript
- Size: 18.2 MB
- Stars: 20
- Watchers: 3
- Forks: 2
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## How to install
* install nodejs
* clone the repository
* in `webstorescraper/` run `yarn install`
## How to use
```
Usage: node index.js command [limit] [dbProperties]
command can be:
meta - retrieves extensions meta data and saves to 'data/extensions.json'
download - downloads all the extensions in 'data/extensions.json' to data/extensions/*
analyse - analyse the downloaded extensions
database - fills a postgresql database with parsed & downloaded data
For the `download` command you can also pass an optional `limit` parameter.
It controls the minimum downloads extension should have in order to be downloaded.
Default value is 10000.
For the `database` command you can also pass database properties
dbProperties=host port database user password
```
* Parse meta: `node index.js meta > log.txt &`
* Download extensions with 10k+ users: `node index.js download 10000 > log.txt &`
* Database: `eval $(cat .env) node index.js database & > log.txt`
## Setup puppeteer on Debian
```
apt-get install -y wget
apt-get install -y gconf-service
apt-get install -y libasound2
apt-get install -y libatk1.0-0
apt-get install -y libc6
apt-get install -y libcairo2
apt-get install -y libcups2
apt-get install -y libdbus-1-3
apt-get install -y libexpat1
apt-get install -y libfontconfig1
apt-get install -y libgcc1
apt-get install -y libgconf-2-4
apt-get install -y libgdk-pixbuf2.0-0
apt-get install -y libglib2.0-0
apt-get install -y libgtk-3-0
apt-get install -y libnspr4
apt-get install -y libpango-1.0-0
apt-get install -y libpangocairo-1.0-0
apt-get install -y libstdc++6
apt-get install -y libx11-6
apt-get install -y libx11-xcb1
apt-get install -y libxcb1
apt-get install -y libxcomposite1
apt-get install -y libxcursor1
apt-get install -y libxdamage1
apt-get install -y libxext6
apt-get install -y libxfixes3
apt-get install -y libxi6
apt-get install -y libxrandr2
apt-get install -y libxrender1
apt-get install -y libxss1
apt-get install -y libxtst6
apt-get install -y ca-certificates
apt-get install -y fonts-liberation
apt-get install -y libappindicator1
apt-get install -y libnss3
apt-get install -y lsb-release
apt-get install -y xdg-utils
```