https://github.com/pocesar/actor-shopify-scraper

Automate monitoring prices on the most popular solution for building online stores and selling products online. Crawl arbitrary Shopify-powered online stores and extract a list of all products in a structured form, including product title, price, description, etc
https://github.com/pocesar/actor-shopify-scraper

apify apify-sdk javascript scraper scraping shopify

Last synced: 17 days ago
JSON representation

Automate monitoring prices on the most popular solution for building online stores and selling products online. Crawl arbitrary Shopify-powered online stores and extract a list of all products in a structured form, including product title, price, description, etc

• Host: GitHub
• URL: https://github.com/pocesar/actor-shopify-scraper
• Owner: pocesar
• Created: 2021-07-17T01:18:26.000Z (over 3 years ago)
• Default Branch: master
• Last Pushed: 2023-06-05T17:12:50.000Z (over 1 year ago)
• Last Synced: 2024-10-18T06:28:47.997Z (27 days ago)
• Topics: apify, apify-sdk, javascript, scraper, scraping, shopify
• Language: JavaScript
• Homepage: https://apify.com/pocesar/shopify-scraper
• Size: 36.1 KB
• Stars: 13
• Watchers: 2
• Forks: 9
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md

Awesome Lists containing this project

README

        
## What does Shopify Scraper do?

Using this tool, you can automate monitoring prices on the most popular solution for building online stores and selling products online. Crawl arbitrary Shopify-powered online stores and extract a list of all products in a structured form, including product title, price, description, etc.

## Need to find product pairs between Shopify and another online shop?

Use the [AI Product Matcher](https://apify.com/equidem/ai-product-matcher). This AI model allows you to compare items from different web stores, identifying exact matches and comparing real-time data obtained via web scraping. With the AI Product Matcher, you can use scraped product data to monitor product matches across the industry, implement dynamic pricing for your website, replace or complement manual mapping, and obtain realistic estimates against your competition for upcoming promo campaigns. 

Most importantly, it is relatively easy to get started with (just follow [this guide](https://blog.apify.com/product-matching-ai-pricing-intelligence-web-scraping/)) and it can match thousands of product pairs.

## Extend Scraper and Output Function

Extend output function allows to filter the items that are output:

```js

async ({ item, customData }) => {

    if (!item.title.includes('cuisine')) {

        return null; // omit the output

    }

    delete item.additional; // remove data from output

    item.requestId = customData.requestId; // add data from the outside

    return item;

}

```

Extend scraper function allows you to interact with scraper phases:

```js

async ({ label, url, filter, fns, filteredSitemapUrls, customData }) => {

    switch (label) {

        case 'FILTER_SITEMAP_URL': {

            // product url, like .../products/cooking-for-dummies-2002-289854

            filter(

                url.includes('cooking') || url.includes(customData.filter)

            );

            break;

        }

        case 'SETUP': {

            // filteredSitemapUrls is a `Set` instance and can be edited in-place

            filteredSitemapUrls.add('https://example.com/secret-unlisted-sitemap.xml');

            filteredSitemapUrls.forEach((sitemapURL) => {

                if (!sitemapURL.includes('en-us')) {

                    filteredSitemapUrls.delete(sitemapURL);

                }

            });

            break;

        }

    }

}

```

## License

Apache 2.0