{"id":16796268,"url":"https://github.com/zoranpandovski/prodirectscraper","last_synced_at":"2025-08-07T16:32:19.344Z","repository":{"id":22479641,"uuid":"96368921","full_name":"ZoranPandovski/ProdirectScraper","owner":"ZoranPandovski","description":":necktie: Web scraper for http://www.prodirectselect.com/ :shoe:","archived":false,"fork":false,"pushed_at":"2024-10-28T10:57:23.000Z","size":117,"stargazers_count":15,"open_issues_count":9,"forks_count":9,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-12-01T10:53:29.466Z","etag":null,"topics":["python","scraper","scrapy","scrapy-crawler","scrapy-spider","spider","webscraper","webscraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ZoranPandovski.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-05T23:23:53.000Z","updated_at":"2024-10-28T11:06:31.000Z","dependencies_parsed_at":"2023-01-11T21:38:34.465Z","dependency_job_id":"2f904426-d203-4b40-b566-d3e4eea1bdf3","html_url":"https://github.com/ZoranPandovski/ProdirectScraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZoranPandovski%2FProdirectScraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZoranPandovski%2FProdirectScraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZoranPandovski%2FProdirectScraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZoranPandovski%2FProdirectScraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ZoranPandovski","download_url":"https://codeload.github.com/ZoranPandovski/ProdirectScraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229067269,"owners_count":18014946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","scraper","scrapy","scrapy-crawler","scrapy-spider","spider","webscraper","webscraping"],"created_at":"2024-10-13T09:18:46.932Z","updated_at":"2024-12-10T13:26:14.166Z","avatar_url":"https://github.com/ZoranPandovski.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://img.shields.io/travis/ZoranPandovski/ProdirectScraper/master.svg?logo=travis)](https://travis-ci.org/ZoranPandovski/ProdirectScraper)\n[![Maintainability](https://api.codeclimate.com/v1/badges/7628ad67f9b52d2b1e66/maintainability)](https://codeclimate.com/github/ZoranPandovski/ProdirectScraper/maintainability)\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/a17cb028c594405e9235c724d6b45b50)](https://app.codacy.com/app/ZoranPandovski/ProdirectScraper?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=ZoranPandovski/ProdirectScraper\u0026utm_campaign=badger)\n[![BCH compliance](https://bettercodehub.com/edge/badge/ZoranPandovski/ProdirectScraper?branch=master)](https://bettercodehub.com/)\n[![Known Vulnerabilities](https://snyk.io/test/github/ZoranPandovski/ProdirectScraper/badge.svg?targetFile=requirements.txt)](https://snyk.io/test/github/ZoranPandovski/ProdirectScraper?targetFile=requirements.txt)\n[![License](https://img.shields.io/badge/license-MIT%20License-brightgreen.svg)](https://opensource.org/licenses/MIT)\n[![Coverage Status](https://coveralls.io/repos/github/ZoranPandovski/ProdirectScraper/badge.svg?branch=master)](https://coveralls.io/github/ZoranPandovski/ProdirectScraper?branch=master)\n[![Total alerts](https://img.shields.io/lgtm/alerts/g/ZoranPandovski/ProdirectScraper.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/ZoranPandovski/ProdirectScraper/alerts/)\n[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/ZoranPandovski/ProdirectScraper.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/ZoranPandovski/ProdirectScraper/context:python)\n\n# ProdirectScraper\n\n## Installation\n\nInstalling Scrapy inside a virtual environment on all platforms.\n\nPython packages can be installed either globally (a.k.a system wide), or in user-space. We do not recommend installing scrapy system wide.\n\nInstead, we recommend that you install scrapy within a so-called “virtual environment” (virtualenv).\n\nVirtualenvs allow you to not conflict with already-installed Python system packages (which could break some of your system tools and scripts), and still install packages normally with pip (without sudo and the likes).\n\nTo install it globally (having it globally installed actually helps here), it should be a matter of running:\n\n```\n$ [sudo] pip install virtualenv\n```\n\nInside virtual env install ProdirectScraper dependencies:\n\n```\npip install -r requirements.txt\n```\n\n## Config Settings\nThese are the basic options:\n```\n# available currency EUR,USD,GBP\ncurrency =\n# Number of pieces to display in the email\npp =\n\n# mailer configuration options\nsmtp_host =\nmail_from =\nmail_to =\nsmtp_user =\nsmtp_pass =\nsmtp_port =\nsmtp_tls =\nsmtp_ssl =\n```\n\nAfter that edit the configuration specific to the category of product you would like to scrape.\n\nFor trainers:\n```\n#available sizes are from 4 to 12, e.g 4 or 4,5,10\nsize =\n\n```\n\nFor men's clothing:\n```\n# available options:  One size, ONE-SIZE, S/M, L/XL, S, M, L, XL, XXL\nsize =\n```\n\nFor women's clothing:\n```\n# available options: OSFM,One Size,8,10,12,14,16,7 - 10,4½ - 7½,ONE-SIZE,32C,6,3½,4,4½,5,5½,6½,7,7½,XXS,XS,S,M,L,XL\nsize =\n```\n\n\n## Running the Spiders\n\nTo put our spider to work, go to the project’s top level directory and run:\n\n```\nscrapy crawl SCRAPER\n```\nwhere \"SCRAPER\" must be one of the following:\n- trainers\n- mensclothing\n- womensclothing\n\nThis command runs the spider with name trainers, that will crawl http://www.prodirectselect.com/ website and send mail with lowest prices, model description and link to trainers, which size is specified in configuration.ini \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzoranpandovski%2Fprodirectscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzoranpandovski%2Fprodirectscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzoranpandovski%2Fprodirectscraper/lists"}