https://github.com/viveckh/qarecewebcrawler
This web crawler gathers the latest details, variations, imagery and pricing informations of a catalog of products given their urls from their corresponding online stores and prepares files ready for upload to your e-commerce platfrom. It was built with the purposes of making product additions easier for e-commerce retailers.
https://github.com/viveckh/qarecewebcrawler
catalog e-commerce ecommerce-platfrom ecommerce-retailers find-products imagery kyliejenner macys pricing-informations qarece-web-crawler scrapy sephora startup-code startup-resources startup-template startups webscraper woocommerce woocommerce-extension wordpress-plugin
Last synced: 7 months ago
JSON representation
This web crawler gathers the latest details, variations, imagery and pricing informations of a catalog of products given their urls from their corresponding online stores and prepares files ready for upload to your e-commerce platfrom. It was built with the purposes of making product additions easier for e-commerce retailers.
- Host: GitHub
- URL: https://github.com/viveckh/qarecewebcrawler
- Owner: Viveckh
- Created: 2019-03-24T14:51:37.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2019-10-29T01:12:31.000Z (almost 6 years ago)
- Last Synced: 2025-02-28T17:25:13.316Z (7 months ago)
- Topics: catalog, e-commerce, ecommerce-platfrom, ecommerce-retailers, find-products, imagery, kyliejenner, macys, pricing-informations, qarece-web-crawler, scrapy, sephora, startup-code, startup-resources, startup-template, startups, webscraper, woocommerce, woocommerce-extension, wordpress-plugin
- Language: Jupyter Notebook
- Homepage: https://www.qarece.com
- Size: 2.36 MB
- Stars: 8
- Watchers: 4
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Qarece Web Crawler
## Author: [(EJ) Vivek Pandey](https://viveckh.com) / [Sujil Maharjan](https://sujilmaharjan.com) / Arjun Bastola
## Save hours spent on manually adding and updating products to your e-commerce site and do it in seconds!
### This web crawler gathers the latest details, variations, imagery and pricing informations of a catalog of products given their urls from their corresponding online stores and prepares files ready for upload to your e-commerce platfrom. It was built with the purposes of making product additions easier for e-commerce retailers.
## Supported Ecommerce Stores
- Macys
- Sephora
- 6pm (Future Enchancement)
- Amazon (Future Enhancement)
## Output Files Customized for Platforms
- Woocommerce (Default)
## Important Notes
- The internal category mapping, profit rates, and shipping rates are to be adjusted by you according to your business logic in the reference files located at `product_scraper/product_scraper/reference_data` folder. The current values are default placeholders.
## Custom Enhancements for Your Business
This crawler prototype was initially built as per the needs of Qarece - New York. If you want a custom implementation or enhancement for your e-commerce business, reach out to the author at `viveckh@hotmail.com`.
Development rates are on a hourly basis or project basis depending on your needs.
## How to Run
- Clone the Repo (Duh)
- Make sure you have Python, Scrapy and Pandas installed in your system
- Add the urls of products you want to crawl by going to `/product_scraper/product_scraper/input/urls.txt`. Check the supported e-commerce stores above.
- Navigate back to the base of the scraper where the scrapy.cfg file is. `cd /product_scraper`
- `python setup.py`
- `scrapy crawl products-spider`
- `python clean-scraped-results.py`
- The output file to upload in woocommerce is ready at `/product_scraper/product_scraper/output/Product-formatted-for-wp.csv`. Run it twice, once in a new products import mode, and again in update mode to update existing products
- The output file to hide products that are no longer available in market will be ready at `/product_scraper/product_scraper/output/products_to_unpublish.csv`. Run it once in update mode to hide products that exist in your catalog but are not available in the market anymore.
- Any products that could not be crawled successfully with associated errors will be detailed in `/product_scraper/product_scraper/output/errors.txt`
## TODO
- An automated way to update catalog in woocommerce using output files without having to upload manually
- An automated script to prepare `urls.txt` by exporting the catalog from woocommerce in csv, and making a list of product's market urls