https://github.com/allendema/rewe_dl
Call store APIs. Parse responses and save the data to SQL, JSON or any custom format.
https://github.com/allendema/rewe_dl
apprise discounts httpx json jsonl market price-comparison price-tracker products python3 ruff sqlite3 webhooks
Last synced: 2 months ago
JSON representation
Call store APIs. Parse responses and save the data to SQL, JSON or any custom format.
- Host: GitHub
- URL: https://github.com/allendema/rewe_dl
- Owner: allendema
- License: gpl-3.0
- Created: 2024-07-31T20:52:14.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2025-02-25T22:38:08.000Z (2 months ago)
- Last Synced: 2025-02-25T23:30:36.596Z (2 months ago)
- Topics: apprise, discounts, httpx, json, jsonl, market, price-comparison, price-tracker, products, python3, ruff, sqlite3, webhooks
- Language: Python
- Homepage:
- Size: 585 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
## About
Python library to call the APIs that the store itself calls.
To get good parsed results from those API calls see [`parser.py`](https://github.com/allendema/rewe_dl/blob/main/rewe_dl/parser.py).
[](https://github.com/allendema/rewe_dl/main/pyproject.toml)
[](https://github.com/astral-sh/ruff)
[](https://github.com/allendema/rewe_dl/commits/main)
## Usage
Prerequisites
sudo apt-get install git
python -m ensurepip --upgrade
```bash
git clone https://github.com/allendema/rewe_dl && cd rewe_dl
```
```bash
pip install requirements.txt
```
```bash
pip install -e rewe_dl
```
Optional [apprise](https://github.com/caronc/apprise/wiki/) can be used to send send notifications to more than 100 services/apps.
See [Config](https://github.com/allendema/rewe_dl/main/README.md#config) options!
## Examples
You can:
- save product infos to JSON
- save product infos to JSONL
- save product infos to SQL
- save product images
- save product calories or other details
- monitor your _basket_ with your fav products
- compare prices and get notified when prices change
- save products to SQL for further analysis
- analyse the output data the way you like, for example: inflation analysis.
- _and_ whatever you want.
examples/discounted_to_json.py
https://github.com/allendema/rewe_dl/blob/7948de8ebc401af4791cdd66fd928daac58d749e/rewe_dl/examples/discounted_to_json.py#L26-L37
examples/discounted_to_sql.py
https://github.com/allendema/rewe_dl/blob/7948de8ebc401af4791cdd66fd928daac58d749e/rewe_dl/examples/discounted_to_sql.py#L23-L35
examples/my_basket.py
https://github.com/allendema/rewe_dl/blob/7948de8ebc401af4791cdd66fd928daac58d749e/rewe_dl/examples/my_basket.py#L23-L37
examples/newest_products_to_sql.py
https://github.com/allendema/rewe_dl/blob/7948de8ebc401af4791cdd66fd928daac58d749e/rewe_dl/examples/newest_products_to_sql.py#L19-L26
examples/notify_when_price.py
https://github.com/allendema/rewe_dl/blob/7948de8ebc401af4791cdd66fd928daac58d749e/rewe_dl/examples/notify_when_price.py#L54-L66
examples/raw_responses_to_json.py
https://github.com/allendema/rewe_dl/blob/7948de8ebc401af4791cdd66fd928daac58d749e/rewe_dl/examples/raw_responses_to_json.py#L26-L43
See `rewe_dl/examples` folder for some more starting examples to embedd it into your own project.
## Config
### Getting the cookies
Depending on store location, some stores will have different prices.
To not use the default store, go to [https://shop.rewe.de](https://shop.rewe.de)
and select your pickup store.
Right click -> Inspect source -> Web Storage -> Cookies
and copy the value from `marketsCookie` into the `config.json` file.
### Notifications/Webhooks
If you don't want to install apprise, you can use [matrix.org](https://matrix.org) or [telegram.org](https://telegram.org).
- Matrix:
Fill the `matrix` dict in `rewe_dl/config.json` file.
- Telegram:
Fill the `telegram` dict in `rewe_dl/config.json` file.
- Apprise (Optional): The config file in `~/.config/apprise` (without any extension) will be used.
Configure your wanted notifications/webhooks options based on the [wiki](https://github.com/caronc/apprise/wiki/).
## Changes
API endpoints or data structure might change!
If you have any issues report them or open a pull request.
This project will be a part of another project which uses multiple stores using an REST API with Fastapi.
Development
### Development
If you want to modify something:
- `pip install requirements-dev.txt`
- The `pyproject.toml` file is used for linting/formatting with `ruff`.
- Make your changes.
- If you add tests, run them with `python3 ./scripts/run_tests.py`.
- Run `ruff check --fix .`
- Run `ruff format .`
- Create a [Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request).
## Credits
Inspired from [gallery-dl](https://github.com/mikf/gallery-dl), the code structure is similiar and some is code straight reused.
Where applicable - docstrings point to original code creators.