{"id":28543852,"url":"https://github.com/aniket-16-s/product-sraper","last_synced_at":"2025-06-24T10:10:07.763Z","repository":{"id":292293150,"uuid":"978465556","full_name":"Aniket-16-S/product-Sraper","owner":"Aniket-16-S","description":"Scrapping product information from well known websites.","archived":false,"fork":false,"pushed_at":"2025-06-08T04:09:13.000Z","size":108,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-08T04:29:34.826Z","etag":null,"topics":["automation","chromedriver","ecommerce-website","selenium","selenium-python","selenium-webdriver","webscraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aniket-16-S.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-06T03:09:42.000Z","updated_at":"2025-06-08T04:09:17.000Z","dependencies_parsed_at":"2025-06-08T04:29:57.551Z","dependency_job_id":null,"html_url":"https://github.com/Aniket-16-S/product-Sraper","commit_stats":null,"previous_names":["aniket-16-s/product-sraper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Aniket-16-S/product-Sraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aniket-16-S%2Fproduct-Sraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aniket-16-S%2Fproduct-Sraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aniket-16-S%2Fproduct-Sraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aniket-16-S%2Fproduct-Sraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aniket-16-S","download_url":"https://codeload.github.com/Aniket-16-S/product-Sraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aniket-16-S%2Fproduct-Sraper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261649858,"owners_count":23189755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","chromedriver","ecommerce-website","selenium","selenium-python","selenium-webdriver","webscraping"],"created_at":"2025-06-09T22:06:01.508Z","updated_at":"2025-06-24T10:10:07.741Z","avatar_url":"https://github.com/Aniket-16-S.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ⚡ Multi-Web Product Scraper (Stealth Mode | Blazing Fast | NLP for queries | Cache Management ) ⚡\n\n### Search Amazon, Flipkart, and Myntra — *all at once*, way before your coffee is ready.\n### Built with async Playwright for stealth, speed.\n\n---\n\n## 🚀 What's Cookin’ :\n\n- 🔍 **Multi-Site Support**: Scrapes all products from Amazon, Flipkart, and Myntra ( more coming soon . . . ).\n- 🧠 **`Natural Language Processing`** : Uses all-MiniLM-L6-v2 model for understanding user queries.\n- 🛢️ **`Cache Management`** : Uses aiosqlite to cache the searched results to avoid computation expences.\n- 📈 **150+ Products in 18 Seconds**: Yup, tested — ~152 products loaded in one go ( 18-20s sec ).\n- 🖼️ **`Photo Gallery`** : Product photos are saved with dedicated folder.\n- ⚡ **SuperFast Scraping**: Built with asyncio, aiohttp, and Playwright, scrapes *around 150 products within 20 sec* 💨\n- 🕶️ **Headless \u0026 Stealth Mode**: Runs in the background and mimics human behavior to dodge bot detection.\n- 🧠 **Why This Slaps**: Manually opening 3 sites, scrolling forever, and remembering deals? Nah. Just type what you want — this gives you 150+ options instantly.\n\n---\n\n## 💻 How To Run\nClone this repo :\n```bash\ngit clone https://github.com/Aniket-16-S/product-Sraper.git\n```\n```bash\ncd product-Sraper \n``` \ninstall dependencies :\n```bash\npip install -r requirements.txt\n```\n```bash\nplaywright install\n```\nRun the async scraper :\n```bash\npython main_scraper.py\n```\nEnter your product keyword and let it cook 🔥 for 20 secs.\nNote : Please Wait for 40 to 50 sec for initial setup on first run.\n\n---\n\n## 📂 Output\n#### Products from all 3 sites shown in your terminal.\n\n#### Images of all products store in your File Mansger.\n\n#### Clean, structured format — easy to compare.\n\n#### Great for research, trend tracking, or flexing that dev power.\n\n---\n\n## 🔮 Future Goals\n\n#### 🛍️ Add support for more sites like Ajio, Meesho, Nykaa, etc.\n\n#### 🎛️ GUI version for non-coders\n\n---\n## ⭐ Why This Could Blow Up\n#### Tired of opening 10 tabs to compare products ? This solves that.\n#### Has advance features like Natural Language Processing for query matching and Cache Management.\n#### Get more than 100 products scraped at your terminal in 16 - 20 secs. ( assuming decent network connectivity. )\n#### It’s async and hence faster.\n#### With a clean UI or web version, this could go viral as a price-compare tool.\n#### Dev-friendly = easy stars + forks on GitHub.\n#### Great side-project flex for portfolios or hackathons.\n---\n\n## ⚠️ Legal Note\n### This project is for educational use only. Respect each site’s terms of service and robots.txt.\n\n---\n\n## 🤝 Contributions Welcome\n Got new site targets or bug fixes? Open a PR or drop ideas in issues!\n#### Let’s build the ultimate cross-site scraper 🧃\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faniket-16-s%2Fproduct-sraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faniket-16-s%2Fproduct-sraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faniket-16-s%2Fproduct-sraper/lists"}