{"id":15198894,"url":"https://github.com/rohanrvpatil/scraping_concepts","last_synced_at":"2026-02-18T15:31:09.953Z","repository":{"id":255726182,"uuid":"850618797","full_name":"rohanrvpatil/scraping_concepts","owner":"rohanrvpatil","description":"This project covers all scraping concepts.","archived":false,"fork":false,"pushed_at":"2024-09-11T18:07:25.000Z","size":3485,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-14T00:28:08.431Z","etag":null,"topics":["beautifulsoup4","httpx","python","scrapy","selenium"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rohanrvpatil.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-01T09:55:43.000Z","updated_at":"2024-09-11T18:08:28.000Z","dependencies_parsed_at":"2024-09-28T02:00:23.685Z","dependency_job_id":"84c81f16-3d8d-43d0-9908-c3f5385ec05e","html_url":"https://github.com/rohanrvpatil/scraping_concepts","commit_stats":{"total_commits":34,"total_committers":1,"mean_commits":34.0,"dds":0.0,"last_synced_commit":"aa55f682a35bc1def762324fff5d8ab276f1cdd3"},"previous_names":["rohanrvpatil/scraping_concepts"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohanrvpatil%2Fscraping_concepts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohanrvpatil%2Fscraping_concepts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohanrvpatil%2Fscraping_concepts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohanrvpatil%2Fscraping_concepts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rohanrvpatil","download_url":"https://codeload.github.com/rohanrvpatil/scraping_concepts/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241678127,"owners_count":20001679,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautifulsoup4","httpx","python","scrapy","selenium"],"created_at":"2024-09-28T02:00:18.193Z","updated_at":"2025-10-12T18:41:47.311Z","avatar_url":"https://github.com/rohanrvpatil.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cmark\u003eThis GitHub repository includes web scraping projects built with Scrapy, Selenium, BeautifulSoup, httpx.\u003c/mark\u003e\n\n## basic_scrapy_rugshop:\n\n* **Website:** https://www.therugshopuk.co.uk/rugs-by-room/bedroom-rugs.html\n* **Purpose:** Extracts product data of rugs\n* **Fields extracted:** name, price, link\n* **Scraping tool:** Scrapy\n* **Libraries/Methods used:** selectors\n* **Exported data:** [output.csv](https://github.com/rohanrvpatil/scraping_concepts/blob/main/basic_scrapy_rugshop/output.csv)\n\n\n\n## beautiful_soup_proxies:\n\n* **Website:** https://free-proxy-list.net/\n* **Purpose:** Extracting free proxies and verifying them\n* **Fields extracted:** proxy(with port)\n* **Scraping tool:** BeautifulSoup\n* **Libraries/Methods used:** requests\n* **Exported data:** [verified_proxies.csv](https://github.com/rohanrvpatil/scraping_concepts/blob/main/scraping_proxies/verified_proxies.csv)\n\n\n\n## dynamic_hidden_api_json:\n\n* **Website:** https://www.petsathome.com/\n* **Purpose:** Extracts product data of pet toys, accessories, food essentials\n* **Fields extracted:** 28 columns of product details\n* **Scraping tool:** Fetch/XHR tool in Network tab of Console (Extracted json from API)\n* **Libraries/Methods used:** requests\n* **Exported data:** [products_data.xlsx](https://github.com/rohanrvpatil/scraping_concepts/blob/main/extracting_json/files/products_data.xlsx), [response_data.json](https://github.com/rohanrvpatil/scraping_concepts/blob/main/extracting_json/files/response_data.json)\n\n\n\n## httpx_scraping:\n\n* **Website:** https://www.rei.com/c/downhill-ski-boots\n* **Purpose:** Extracts product data of downhill ski-boots\n* **Fields extracted:**  link, name, product_id, price, rating\n* **Scraping tool:** python-httpx\n* **Libraries/Methods used:** selectors, urljoin, HTMLParser, dataclasses, export functions for csv/xlsx/json\n* **Exported data:** [data.csv](https://github.com/rohanrvpatil/scraping_concepts/blob/main/html_scraping/data_exports/data.csv), [data.json](https://github.com/rohanrvpatil/scraping_concepts/blob/main/html_scraping/data_exports/data.json), [data.xlsx](https://github.com/rohanrvpatil/scraping_concepts/blob/main/html_scraping/data_exports/data.xlsx)\n\n\n\n## dynamic_scrapy_splash_beerwulf (test project):\n\n* **Website:** https://www.beerwulf.com/en-gb/c/mixedbeercases\n* **Purpose:** Extracts beer product data\n* **Fields extracted:**  name, price\n* **Scraping tool:** scrapy-splash\n* **Libraries/Methods used:** None\n* **Exported data:** None\n\n\n\n## selenium_amazon_products:\n\n* **Website:** [Amazon searching for \"dell i7 laptop\"](https://www.amazon.com/s?k=dell+i7+laptop\u0026crid=3OIV4GP9RPUT3\u0026sprefix=dell+i7%2Caps%2C687\u0026ref=nb_sb_ss_ts-doa-p_1_7)\n* **Purpose:** Extracting product details of laptops related to \"dell i7 laptop\"\n* **Fields extracted:** link, title, price, brand, model name, screen size, about this item, technical details: summary, rating (out of 5)\n* **Scraping tool:** Selenium\n* **Libraries/Methods used:** user agent rotation, chrome_options\n* **Exported data:** [laptop_details.xlsx](https://github.com/rohanrvpatil/scraping_concepts/blob/main/selenium_amazon_products/search_laptop_details/data/laptop_details.xlsx)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frohanrvpatil%2Fscraping_concepts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frohanrvpatil%2Fscraping_concepts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frohanrvpatil%2Fscraping_concepts/lists"}