https://github.com/pyprogrammerblog/progress-updater
The Progress-Updater help you to track your tasks using backends like Redis, Mongo or SQL
https://github.com/pyprogrammerblog/progress-updater
data-engineering mongo pipelines pydantic python redis sql
Last synced: 6 months ago
JSON representation
The Progress-Updater help you to track your tasks using backends like Redis, Mongo or SQL
- Host: GitHub
- URL: https://github.com/pyprogrammerblog/progress-updater
- Owner: pyprogrammerblog
- License: mit
- Created: 2022-09-03T14:35:21.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2022-09-17T12:22:07.000Z (over 3 years ago)
- Last Synced: 2025-06-24T07:39:10.543Z (10 months ago)
- Topics: data-engineering, mongo, pipelines, pydantic, python, redis, sql
- Language: Python
- Homepage:
- Size: 63.5 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
progress-updater
=================
[](https://progress-updater.readthedocs.io/en/latest/?badge=latest)
[](https://github.com/pyprogrammerblog/progress-updater/blob/master/LICENSE)
[](https://github.com/pyprogrammerblog/progress-updater/workflows/CI/badge.svg/)
[](https://badge.fury.io/py/progress-updater)
Writing the progress of a task to a backend!
Installation
-------------
Install it using ``pip``
```shell
pip install progress-updater
```
Basic usage
-------------
```python
from progress_updater import ProgressUpdater
from progress_updater.backends.mongo import MongoSettings
settings = MongoSettings(
mongo_connection="mongodb://user:pass@mongo:27017",
mongo_db="db",
mongo_collection="logs",
)
updater = ProgressUpdater(task_name="My Task", settings=settings)
with updater(block_name="First part"):
# doing things
updater.notify("doing first block...")
# doing more things
with updater(block_name="Second part"):
# doing things
updater.notify("doing second block...")
# doing more things
updater.raise_latest_exception() # if exists
```
The output is:
```shell
- Task: My task
- Entering First part
doing first block...
Time spent: 0h0m
Successfully completed
- Entering Second part
doing second block...
Time spent: 0h0m
Successfully completed
```
Backends
-------------------
The available backends to store logs are **Mongo**, **Redis** and **SQL**.
Please read the [docs](https://progress-updater.readthedocs.io/en/latest/)
for further information.
Setting your backend with environment variables
--------------------------------------------------
You can set your backend by defining env vars.
The `PU__` prefix indicates that it belongs to `ProgressUpdater`.
```shell
# SQL
PU__SQL_DSN='postgresql+psycopg2://user:pass@postgres:5432/db'
PU__SQL_TABLE='logs'
...
# Redis
PU__REDIS_HOST='redis'
PU__REDIS_DB='1'
PU__REDIS_PASSWORD='pass'
...
# Mongo
PU__MONGO_CONNECTION='mongodb://user:pass@mongo:27017'
PU__MONGO_DB='db'
PU__MONGO_COLLECTION='logs'
...
```
And then when creating a `ProgressUpdater` object, the backend will be
automatically configured.
```python
from progress_updater import ProgressUpdater
with ProgressUpdater(task_name="My Task") as updater:
pass
```
Documentation
--------------
Please visit this [link](https://progress-updater.readthedocs.io/en/latest/) for documentation.