https://github.com/raspi/scrapy-crucial

Web crawler for Crucial (crucial.com)
https://github.com/raspi/scrapy-crucial

crawler hardware memory scrapy spider

Last synced: 2 days ago
JSON representation

Web crawler for Crucial (crucial.com)

• Host: GitHub
• URL: https://github.com/raspi/scrapy-crucial
• Owner: raspi
• License: apache-2.0
• Created: 2020-07-28T20:13:29.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2020-07-28T23:45:11.000Z (over 4 years ago)
• Last Synced: 2024-11-10T23:12:53.105Z (2 months ago)
• Topics: crawler, hardware, memory, scrapy, spider
• Language: Python
• Homepage:
• Size: 11.7 KB
• Stars: 0
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# scrapy-crucial

Web crawler for Crucial ([crucial.com](https://www.crucial.com))

## Requirements

* Python

* [Scrapy](https://scrapy.org/)

## Notes

* 30 day cache is used in `settings.py`

## Spiders

All items are downloaded as JSON in the `items/` directory.

### Memory modules for all motherboards from certain manufacturer

    scrapy crawl manufacturer -a product="supermicro/supermicro-motherboards"

This will generate `items/Memory/supermicro/.json` which then lists all compatible memory modules for this motherboard.

### Memory modules for certain motherboard

    scrapy crawl motherboard -a product="supermicro/a2sdi-ln4f"

This will generate `items/Memory/supermicro/A2SDi-LN4F.json` which then lists all compatible memory modules for this motherboard.