{"id":42051961,"url":"https://github.com/ndgigliotti/shopify-spy","last_synced_at":"2026-01-26T07:02:21.883Z","repository":{"id":54734569,"uuid":"325655179","full_name":"ndgigliotti/shopify-spy","owner":"ndgigliotti","description":"Extract structured data from Shopify websites.","archived":false,"fork":false,"pushed_at":"2022-04-08T01:41:41.000Z","size":78,"stargazers_count":30,"open_issues_count":0,"forks_count":11,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-11T17:37:49.073Z","etag":null,"topics":["crawler","data","data-acquisition","data-science","dropshipping","ecommerce","scrape","scraper","scraping","scrapy","shopify","spider"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ndgigliotti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-30T21:52:02.000Z","updated_at":"2023-03-04T13:23:16.000Z","dependencies_parsed_at":"2022-08-14T01:10:40.254Z","dependency_job_id":null,"html_url":"https://github.com/ndgigliotti/shopify-spy","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"purl":"pkg:github/ndgigliotti/shopify-spy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndgigliotti%2Fshopify-spy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndgigliotti%2Fshopify-spy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndgigliotti%2Fshopify-spy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndgigliotti%2Fshopify-spy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ndgigliotti","download_url":"https://codeload.github.com/ndgigliotti/shopify-spy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndgigliotti%2Fshopify-spy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28769215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T06:37:25.426Z","status":"ssl_error","status_checked_at":"2026-01-26T06:37:23.039Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","data","data-acquisition","data-science","dropshipping","ecommerce","scrape","scraper","scraping","scrapy","shopify","spider"],"created_at":"2026-01-26T07:02:19.603Z","updated_at":"2026-01-26T07:02:21.870Z","avatar_url":"https://github.com/ndgigliotti.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/ndgigliotti/shopify-spy/master/assets/icon.png\" width=150\u003e\u003c/div\u003e\n\n# Shopify Spy\n\n[![CI](https://github.com/ndgigliotti/shopify-spy/actions/workflows/ci.yml/badge.svg)](https://github.com/ndgigliotti/shopify-spy/actions/workflows/ci.yml)\n[![PyPI version](https://badge.fury.io/py/shopify-spy.svg)](https://badge.fury.io/py/shopify-spy)\n[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\nShopify Spy is a command-line tool for scraping product and collection data from any Shopify store. Built on [Scrapy](https://docs.scrapy.org/en/latest/index.html), it extracts detailed data including high-value information like vendor names and inventory levels.\n\nTo find Shopify stores to scrape, try searching Google with `site:myshopify.com`.\n\n## Installation\n\n[pipx](https://pipx.pypa.io/) and [uv tool](https://docs.astral.sh/uv/) install CLI tools in isolated environments, so they won't conflict with other Python projects:\n\n```bash\n# pipx\npipx install shopify-spy\n\n# uv\nuv tool install shopify-spy\n```\n\nOr install with pip if you want it in a specific virtual environment:\n\n```bash\npip install shopify-spy\n```\n\nRequires Python 3.10+.\n\n## Quick Start\n\n```bash\n# Scrape a single store\nshopify-spy scrape https://www.example.com\n\n# Scrape multiple stores\nshopify-spy scrape https://store1.com https://store2.com https://store3.com\n\n# Download product images\nshopify-spy scrape https://www.example.com --images\n\n# Include collections\nshopify-spy scrape https://www.example.com --collections\n\n# Scrape multiple stores from a file\nshopify-spy scrape --url-file stores.txt\n\n# Specify output directory\nshopify-spy scrape https://www.example.com --output ./my-data\n```\n\nResults are saved as JSON lines in the output directory (default: `./output`).\n\n## Commands\n\n### `scrape`\n\nScrape products and collections from Shopify stores.\n\n```bash\nshopify-spy scrape [URL] [OPTIONS]\n```\n\n**Arguments:**\n- `URL...` - One or more Shopify store URLs (optional if using `--url-file`)\n\n**Options:**\n- `--url-file, -f FILE` - File containing URLs (one per line)\n- `--products / --no-products` - Scrape products (default: yes)\n- `--collections / --no-collections` - Scrape collections (default: no)\n- `--images / --no-images` - Download images (default: no)\n- `--output, -o PATH` - Output directory (default: `./output`)\n- `--config, -c FILE` - Path to YAML config file\n- `--concurrent INT` - Concurrent requests per domain (default: 16)\n- `--throttle / --no-throttle` - Auto-throttle requests (default: yes)\n- `--user-agent, -A TEXT` - Custom User-Agent header\n- `--verbose, -v` - Show debug output\n- `--quiet, -q` - Show only warnings and errors\n\n### `init`\n\nCreate a default configuration file.\n\n```bash\nshopify-spy init [PATH]\n```\n\n**Arguments:**\n- `PATH` - Where to create the config file (default: `./shopify-spy.yaml`)\n\n**Options:**\n- `--force, -f` - Overwrite existing file\n\n## Configuration\n\nShopify Spy can be configured via YAML file. Create one with `shopify-spy init`:\n\n```yaml\n# shopify-spy.yaml\nscrape:\n  products: true      # Scrape product data\n  collections: false  # Scrape collection data\n  images: false       # Download product images\n\noutput:\n  dir: ./output       # Output directory for results\n  images_subdir: images  # Subdirectory for downloaded images\n\nnetwork:\n  concurrent_requests: 16  # Concurrent requests per domain\n  timeout: 180             # Download timeout (seconds)\n  retries: 2               # Retry failed requests\n  # user_agent: MyBot/1.0 (+https://example.com)  # Custom user agent\n  respect_robots_txt: true\n\nthrottle:\n  enabled: true            # Auto-throttle based on server response\n  start_delay: 1           # Initial download delay (seconds)\n  max_delay: 60            # Maximum download delay (seconds)\n  target_concurrency: 1.0  # Target concurrent requests (higher = faster)\n```\n\n**Config file search order:**\n1. Path specified with `--config`\n2. `./shopify-spy.yaml`\n3. `~/.config/shopify-spy/config.yaml`\n\nCLI options override config file settings.\n\n## Output\n\nResults are saved as JSON lines files in the output directory:\n\n```\noutput/\n  shopify_spider_2024-01-15T10-30-00.jsonl\n  images/\n    full/\n      \u003cimage files\u003e\n```\n\nEach line in the JSON file contains a product or collection with full metadata from Shopify's JSON API.\n\n### Image Metadata\n\nWhen using `--images`, each item includes a `scraped_images` field with download info:\n\n```json\n{\n  \"image_urls\": [\"https://cdn.shopify.com/.../product.jpg\"],\n  \"scraped_images\": [\n    {\n      \"url\": \"https://cdn.shopify.com/.../product.jpg\",\n      \"path\": \"full/abc123def.jpg\",\n      \"checksum\": \"d41d8cd98f00b204e9800998ecf8427e\",\n      \"status\": \"downloaded\"\n    }\n  ],\n  \"product\": { ... }\n}\n```\n\nThe `path` is relative to the images directory (`output/images/` by default).\n\n### Parsing Output\n\n**With jq:**\n```bash\n# Extract product titles\ncat output/*.jsonl | jq '.product.title'\n\n# Get prices\ncat output/*.jsonl | jq '{title: .product.title, price: .product.variants[0].price}'\n```\n\n**With Python:**\n```python\nimport json\n\nwith open(\"output/shopify_spider_2024-01-15.jsonl\") as f:\n    for line in f:\n        item = json.loads(line)\n        print(item[\"product\"][\"title\"])\n```\n\n**With pandas:**\n```python\nimport pandas as pd\n\ndf = pd.read_json(\"output/shopify_spider_2024-01-15.jsonl\", lines=True)\nproducts = pd.json_normalize(df[\"product\"])\n```\n\n**With polars:**\n```python\nimport polars as pl\n\ndf = pl.read_ndjson(\"output/shopify_spider_2024-01-15.jsonl\")\n```\n\n## Limitations\n\n**Standard Shopify stores only.** This tool works with standard Shopify stores using Liquid themes, which represent nearly all Shopify sites. The small number of headless stores built on [Hydrogen](https://hydrogen.shopify.dev/) or other custom storefronts are not supported, as they use the Storefront GraphQL API instead of the JSON endpoints this tool relies on.\n\n**Rate limiting.** Scraping very large stores may still result in temporary bans. Auto-throttling is enabled by default, but you can adjust the settings or disable it for faster scraping:\n\n```bash\n# Disable throttling (faster but riskier)\nshopify-spy scrape https://example.com --no-throttle\n```\n\n## Advanced Usage\n\nFor advanced Scrapy configuration or custom pipelines, you can use Shopify Spy as a library:\n\n```python\nfrom scrapy.crawler import CrawlerProcess\nfrom scrapy.utils.project import get_project_settings\nfrom shopify_spy.spiders.shopify import ShopifySpider\n\nprocess = CrawlerProcess(get_project_settings())\nprocess.crawl(ShopifySpider, url=\"https://example.com\", products=True)\nprocess.start()\n```\n\n## Feedback\n\nFound a bug or have a suggestion? [Open an issue](https://github.com/ndgigliotti/shopify-spy/issues).\n\n## License\n\n[MIT](https://choosealicense.com/licenses/mit/)\n\n## Credits\n\nIcon by [Bartama Graphic](https://www.flaticon.com/authors/bartama-graphic).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndgigliotti%2Fshopify-spy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fndgigliotti%2Fshopify-spy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndgigliotti%2Fshopify-spy/lists"}