{"id":21409129,"url":"https://github.com/usernam3/shopify-app-store-scraper","last_synced_at":"2025-04-07T12:03:40.507Z","repository":{"id":38213354,"uuid":"225490539","full_name":"usernam3/shopify-app-store-scraper","owner":"usernam3","description":"Crawler behind the Shopify App Marketplace dataset","archived":false,"fork":false,"pushed_at":"2024-11-28T00:35:42.000Z","size":70,"stargazers_count":74,"open_issues_count":1,"forks_count":21,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-31T11:03:46.282Z","etag":null,"topics":["crawler","dataset-creation","shopify"],"latest_commit_sha":null,"homepage":"https://www.kaggle.com/usernam3/shopify-app-store","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/usernam3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-02T23:47:27.000Z","updated_at":"2025-03-22T10:03:36.000Z","dependencies_parsed_at":"2025-01-15T17:35:54.697Z","dependency_job_id":"d7e79bf4-9b20-457e-8344-5c0ec8bcde0a","html_url":"https://github.com/usernam3/shopify-app-store-scraper","commit_stats":null,"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usernam3%2Fshopify-app-store-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usernam3%2Fshopify-app-store-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usernam3%2Fshopify-app-store-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usernam3%2Fshopify-app-store-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/usernam3","download_url":"https://codeload.github.com/usernam3/shopify-app-store-scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247648976,"owners_count":20972945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","dataset-creation","shopify"],"created_at":"2024-11-22T17:22:27.950Z","updated_at":"2025-04-07T12:03:40.483Z","avatar_url":"https://github.com/usernam3.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Shopify App Store scraper\n\n## About\n\nHere you can find the code which scrapes and saves data from the [Shopify App Store](https://apps.shopify.com/).\n\nThe scraper is used to collect [Shopify app store dataset on Kaggle](https://www.kaggle.com/usernam3/shopify-app-store) and includes these files:\n- `apps`\n- `apps_categories`\n- `categories`\n- `key_benefits`\n- `pricing_plan_features`\n- `pricing_plans`\n- `reviews`\n\nWhile the dataset published on Kaggle is regularly updated, this repository allows keeping the local copy up to date independently of the released version.\n\nDetailed dataset description can be found [here](https://www.kaggle.com/usernam3/shopify-app-store).\n\n## How to use it\n\n### Docker (recommended)\n\n[Authenticate to GitHub Container Registry](https://docs.github.com/en/packages/getting-started-with-github-container-registry/migrating-to-github-container-registry-for-docker-images#authenticating-with-the-container-registry) (if not already)\n\n```bash\ndocker login ghcr.io -u USERNAME -p TOKEN\n```\n\nPull container\n\n```bash\ndocker pull ghcr.io/usernam3/shopify-app-store-scraper\n```\n\nRun container\n\n```bash\ndocker run -v `pwd`/output/:/app/output/ ghcr.io/usernam3/shopify-app-store-scraper\n```\n\nAfter container finished the execution check the `output` folder (in current directory)\n\n```bash\nls -la output/\n```\n\n### Python\n\nInstall requirements\n\n```bash\npip install -r requirements.txt\n``` \n\nRun scraper\n\n```bash\nscrapy crawl app_store\n``` \n \nAfter container finished the execution check the `output` folder (in current directory)\n\n```bash\nls -la output/\n```\n\n---\n\nPlease don't hesitate to open issues or PRs at any time if you need help with anything.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fusernam3%2Fshopify-app-store-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fusernam3%2Fshopify-app-store-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fusernam3%2Fshopify-app-store-scraper/lists"}