{"id":20710043,"url":"https://github.com/oxylabs/pricing-data-collection-from-ecommerce-stores","last_synced_at":"2025-03-11T05:48:56.435Z","repository":{"id":162961277,"uuid":"638176502","full_name":"oxylabs/pricing-data-collection-from-ecommerce-stores","owner":"oxylabs","description":"Appache Airflow DAGs for e-commerce pricing collection.","archived":false,"fork":false,"pushed_at":"2024-06-03T10:56:50.000Z","size":15,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-17T21:07:28.812Z","etag":null,"topics":["appache","e-commerce-scraper","ebay-search","ebay-searches","ecommerce-scraper","ecommerce-website","pricing-data","scraping","web-crawling","web-scraping"],"latest_commit_sha":null,"homepage":"https://oxylabs.io/products/scraper-api/web","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-05-09T08:39:52.000Z","updated_at":"2024-06-03T10:56:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"7953cd93-5bc3-415e-98dd-c3b7af93d21b","html_url":"https://github.com/oxylabs/pricing-data-collection-from-ecommerce-stores","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpricing-data-collection-from-ecommerce-stores","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpricing-data-collection-from-ecommerce-stores/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpricing-data-collection-from-ecommerce-stores/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpricing-data-collection-from-ecommerce-stores/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/pricing-data-collection-from-ecommerce-stores/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242980784,"owners_count":20216285,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["appache","e-commerce-scraper","ebay-search","ebay-searches","ecommerce-scraper","ecommerce-website","pricing-data","scraping","web-crawling","web-scraping"],"created_at":"2024-11-17T02:09:39.562Z","updated_at":"2025-03-11T05:48:56.428Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pricing Data Collection from E-Commerce Stores\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.go2cloud.org/aff_c?offer_id=7\u0026aff_id=877\u0026url_id=112)\n\n\n\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/GbxmdGhZjq)\n\nDuring the webinar on “Driving E-Commerce Success Through Pricing Data: Are You Getting It Right?”, our Engineering Manager, Povilas Kudriavcevas, showcased how to collect product pricing data from https://books.toscrape.com, using Oxylabs [\u003cu\u003eWeb Crawler\u003c/u\u003e](https://oxylabs.io/features/web-crawler) and [\u003cu\u003eWeb Scraper API\u003c/u\u003e](https://oxylabs.io/products/scraper-api/web). In this repository, you will find the Apache Airflow DAGs he used during the webinar. \n\n## Apache Airflow setup\nFollow official documentation `https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html`\n\n## Setup\n1. Copy the contents of the files into your Apache Airflow project.\n2. Configure your Oxylabs credentials in the `settings.py` file.\n3. Specify the absolute path to the `database.db` file located in the `dags` folder.\n4. Execute the DAGs using the Apache Airflow UI.\n\nWatch the webinar to learn more about pricing data collection and follow the steps Povilas took to do it successfully. \n\nThe webinar recording will be uploaded soon on our [\u003cu\u003eweb\u003c/u\u003e](https://oxylabs.io/resources/webinars), next to all other web scraping-related webinars. If you stumbled here before the 10th of May, you can still watch this webinar live. Make sure to [\u003cu\u003eregister\u003c/u\u003e](https://www.bigmarker.com/oxylabs/Driving-E-Commerce-Success-Through-Pricing-Data-Are-You-Getting-It-Right?)!\n\nWe wish you a smooth data collection, and if you have any questions, feel free to reach out to us at support@oxylabs.io or by live chat on our [\u003cu\u003edashboard\u003c/u\u003e](https://dashboard.oxylabs.io/en/registration). \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fpricing-data-collection-from-ecommerce-stores","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fpricing-data-collection-from-ecommerce-stores","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fpricing-data-collection-from-ecommerce-stores/lists"}