https://github.com/oxylabs/pricing-data-collection-from-ecommerce-stores
Appache Airflow DAGs for e-commerce pricing collection.
https://github.com/oxylabs/pricing-data-collection-from-ecommerce-stores
appache e-commerce-scraper ebay-search ebay-searches ecommerce-scraper ecommerce-website pricing-data scraping web-crawling web-scraping
Last synced: over 1 year ago
JSON representation
Appache Airflow DAGs for e-commerce pricing collection.
- Host: GitHub
- URL: https://github.com/oxylabs/pricing-data-collection-from-ecommerce-stores
- Owner: oxylabs
- Created: 2023-05-09T08:39:52.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-06-03T10:56:50.000Z (about 2 years ago)
- Last Synced: 2025-01-17T21:07:28.812Z (over 1 year ago)
- Topics: appache, e-commerce-scraper, ebay-search, ebay-searches, ecommerce-scraper, ecommerce-website, pricing-data, scraping, web-crawling, web-scraping
- Language: Python
- Homepage: https://oxylabs.io/products/scraper-api/web
- Size: 14.6 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Pricing Data Collection from E-Commerce Stores
[](https://oxylabs.go2cloud.org/aff_c?offer_id=7&aff_id=877&url_id=112)
[](https://discord.gg/GbxmdGhZjq)
During the webinar on “Driving E-Commerce Success Through Pricing Data: Are You Getting It Right?”, our Engineering Manager, Povilas Kudriavcevas, showcased how to collect product pricing data from https://books.toscrape.com, using Oxylabs [Web Crawler](https://oxylabs.io/features/web-crawler) and [Web Scraper API](https://oxylabs.io/products/scraper-api/web). In this repository, you will find the Apache Airflow DAGs he used during the webinar.
## Apache Airflow setup
Follow official documentation `https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html`
## Setup
1. Copy the contents of the files into your Apache Airflow project.
2. Configure your Oxylabs credentials in the `settings.py` file.
3. Specify the absolute path to the `database.db` file located in the `dags` folder.
4. Execute the DAGs using the Apache Airflow UI.
Watch the webinar to learn more about pricing data collection and follow the steps Povilas took to do it successfully.
The webinar recording will be uploaded soon on our [web](https://oxylabs.io/resources/webinars), next to all other web scraping-related webinars. If you stumbled here before the 10th of May, you can still watch this webinar live. Make sure to [register](https://www.bigmarker.com/oxylabs/Driving-E-Commerce-Success-Through-Pricing-Data-Are-You-Getting-It-Right?)!
We wish you a smooth data collection, and if you have any questions, feel free to reach out to us at support@oxylabs.io or by live chat on our [dashboard](https://dashboard.oxylabs.io/en/registration).