https://github.com/gjovs/batch-manager
https://github.com/gjovs/batch-manager
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/gjovs/batch-manager
- Owner: gjovs
- Created: 2024-10-28T01:29:09.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2024-10-28T01:47:05.000Z (6 months ago)
- Last Synced: 2024-10-28T05:26:42.231Z (6 months ago)
- Language: Python
- Size: 7.81 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README_EN.md
Awesome Lists containing this project
README
# 🗃️ Batch Insertion in PostgreSQL with Multiprocessing
This project implements a batch insertion solution in PostgreSQL using Python with `multiprocessing.Pool` and `ThreadPoolExecutor`, providing support for parallel threads and processes. Ideal for handling large volumes of data, the project distributes the workload across multiple threads or processes, reducing insertion time and improving performance.
## 🚀 Overview
Handling large volumes of data can be challenging when insertions are performed sequentially. This project uses the `concurrent.futures.ThreadPoolExecutor` and `multiprocessing.Pool` modules to implement parallel or pseudo-parallel batch insertions in PostgreSQL.
## 🔧 Features
- 📑 **PostgreSQL Connection**: Simple and flexible configuration of connection parameters.
- ⚙️ **Parallel Execution**: Support for both processes and threads via `multiprocessing.Pool` and `ThreadPoolExecutor`.
- 📦 **Batch Insertions**: Divides data into batches to minimize the number of transactions and optimize performance.