{"id":24669232,"url":"https://github.com/amgawishx/item-grapper","last_synced_at":"2025-07-24T14:34:53.782Z","repository":{"id":83131547,"uuid":"593606287","full_name":"amgawishx/item-grapper","owner":"amgawishx","description":"An application to send email notifications about new products on hermes.com","archived":false,"fork":false,"pushed_at":"2023-04-14T18:35:11.000Z","size":82,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-07T12:07:14.640Z","etag":null,"topics":["bot","data-mining","selenium","webscraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amgawishx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-01-26T12:36:19.000Z","updated_at":"2024-03-28T15:57:49.000Z","dependencies_parsed_at":null,"dependency_job_id":"4115652a-9fe6-482d-8e27-1cae5bae890a","html_url":"https://github.com/amgawishx/item-grapper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/amgawishx/item-grapper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amgawishx%2Fitem-grapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amgawishx%2Fitem-grapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amgawishx%2Fitem-grapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amgawishx%2Fitem-grapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amgawishx","download_url":"https://codeload.github.com/amgawishx/item-grapper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amgawishx%2Fitem-grapper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266856278,"owners_count":23995663,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bot","data-mining","selenium","webscraping"],"created_at":"2025-01-26T09:20:04.979Z","updated_at":"2025-07-24T14:34:53.774Z","avatar_url":"https://github.com/amgawishx.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Requirements\nSystem requirements:\n- python3\n- firefox\n- geckodriver\n- pip3\n\n\nTo install them run the following commands:\n```\nsudo apt install python3\nsudo apt install firefox\nsudo apt install pip3\nwget https://github.com/mozilla/geckodriver/releases/download/v0.24.0/geckodriver-v0.24.0-linux64.tar.gz\ntar -xvzf geckodriver*\nchmod +x geckodriver\nsudo mv geckodriver /usr/local/bin\n```\n\nAlso RAM must be at least 2GB\n\nPython  requirements: \n- scrapy==2.6.3\n- selenium==3.141.0\n\nTo install them, run the following after running the system requirements:\n```\npip3 install scrapy==2.6.3 selenium==3.141.0\n```\n## Usage\nThe file named `bot.py` is the entry point to the application, it can be simply run using: `python3 bot.py` but the better approach is to create  a `cronjob` to daily run the launch command to clean up the log files preventing using all available disk space, refresh the proxies that update every 24 hours and keep the application running in background even after ending a SSH session.\n\n## Structure\nThe whole code is a `scrapy` project created by the `scrapy startproject hermes`, for more information about `scrapy` projects are found at: https://docs.scrapy.org/en/latest/intro/tutorial.html\nThe `bot.py`file contains the main code utilizing `webdriver`to navigate to the target URL.\nThe `proxies.json`is a file generated automatically using the following command:  `scrapy crawl arachne -O proxies.json`.\nThere are a few needed environment variables to run the application:\n- `SMTP_SERVER`: The SMTP server used to send emails via.\n- `COMPANY_EMAIL`: The email at the SMTP server that we use to send emails from.\n- `APP_PASSWORD`: The password used to login to the SMTP server.\n- `COMPANY_NAME`: The name to send emails as.\n- `CLIENT_EMAIL`: The email to send the notifications to.\n\n## Notes\nUsing the default run settings, the average time for a subsequent successful acquisition of products' data from the site is around 10-30 minutes, to reduce this time try experimenting with different `TIMEOUT`values or use paid proxy services.\n\nCurrently, the application only sends email notifications regarding new unique products, it doesn't detect new colors/variations of already existing products.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famgawishx%2Fitem-grapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famgawishx%2Fitem-grapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famgawishx%2Fitem-grapper/lists"}