https://github.com/firmadyne/scraper

Firmware scraper
https://github.com/firmadyne/scraper

Last synced: 30 days ago
JSON representation

Firmware scraper

• Host: GitHub
• URL: https://github.com/firmadyne/scraper
• Owner: firmadyne
• License: mit
• Created: 2016-01-20T06:50:53.000Z (almost 9 years ago)
• Default Branch: master
• Last Pushed: 2024-07-10T10:05:42.000Z (5 months ago)
• Last Synced: 2024-08-04T01:27:07.078Z (4 months ago)
• Language: Python
• Homepage:
• Size: 128 KB
• Stars: 107
• Watchers: 12
• Forks: 70
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.txt

Awesome Lists containing this project

README

        
Introduction

============

This is a firmware scraper that aims to download firmware images and associated

metadata from supported device vendor websites.

Dependencies

============

* [psycopg2](http://initd.org/psycopg/)

* [scrapy](http://scrapy.org/)

Usage

=====

1. Configure the `firmware/settings.py` file. Comment out `SQL_SERVER` if metadata about

downloaded firmware should not be inserted into a SQL server.

2. To run a specific scraper, e.g. `dlink`:

`scrapy crawl dlink`

To run all scrapers with maximum 4 in parallel, using [GNU Parallel](https://www.gnu.org/software/parallel/):

```parallel -j 4 scrapy crawl ::: `for i in ./firmware/spiders/*.py; do basename ${i%.*}; done` ```