https://github.com/alefrp/text_to_db

A Python tool for efficiently converting CSV tax data into a PostgreSQL database, featuring column name translation and error logging.
https://github.com/alefrp/text_to_db

converter database postgresql python

Last synced: 2 days ago
JSON representation

A Python tool for efficiently converting CSV tax data into a PostgreSQL database, featuring column name translation and error logging.

• Host: GitHub
• URL: https://github.com/alefrp/text_to_db
• Owner: AlefRP
• License: mit
• Created: 2023-11-07T16:52:24.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-01-24T16:18:37.000Z (10 months ago)
• Last Synced: 2024-01-24T17:32:40.683Z (10 months ago)
• Topics: converter, database, postgresql, python
• Language: Python
• Homepage:
• Size: 3.28 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Text To DB Converter

## Overview 📈

This utility is designed to process CSV files containing tax data, translate the column names based on a provided dictionary, and then load the processed data into a PostgreSQL database. If errors occur during processing, the problematic files are recorded in an `error-log.csv` for subsequent review.



## Requirements 🐍

- Python 3.11.5

- Libraries: `pandas`, `sqlalchemy`, `csv`, `os`, `shutil`, `tqdm`

## Configuration ⚙️

1. Database credentials should be set up in the `config_db.py` file:

   ```python

   db_username = 'your_username'

   db_password = 'your_password'

   db_host = 'db_host'

   db_port = 'db_port'

   db_name = 'db_name'

   db_schema = 'test'

   ```

2. Ensure that the PostgreSQL database is running and that the credentials provided in the `config_db.py` file are correct.

    Remember to create the schema in the PostgreSQL database if it does not exist.

3. After configuring the above parameters, run the script to start processing the CSV files. In case of errors, problematic files will be moved to the `temp/processed` folder and errors logged in the `error-log.csv` file.

## Usage 🚀

To use the Text To DB Converter, follow these steps:

1. Open your command line interface (CLI).

2. Navigate to the directory where the `main.py` file is located.

3. Run the following command:

   ```bash

   python main.py

   ```

## License 📜

This project is licensed under the MIT License - see the [LICENSE](LICENSE) for details.