https://github.com/nitor-infotech-oss/python-bulk-data-parser
A Component for reading excel files data and used similar data for bulk data upload after perform validation. Extremely fast, flexible, and easy to use. Using the result of this parser you can perform insert/update as per your database structure.
https://github.com/nitor-infotech-oss/python-bulk-data-parser
bulk-loader data-parser python3
Last synced: 6 months ago
JSON representation
A Component for reading excel files data and used similar data for bulk data upload after perform validation. Extremely fast, flexible, and easy to use. Using the result of this parser you can perform insert/update as per your database structure.
- Host: GitHub
- URL: https://github.com/nitor-infotech-oss/python-bulk-data-parser
- Owner: nitor-infotech-oss
- Created: 2021-01-04T11:37:54.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2021-01-05T09:15:48.000Z (almost 5 years ago)
- Last Synced: 2025-02-01T09:22:21.680Z (8 months ago)
- Topics: bulk-loader, data-parser, python3
- Language: Python
- Homepage:
- Size: 43 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Bulk Excel/CSV Data Parser
A Component for reading excel files data and used similar data for bulk data upload after perform validation. Extremely fast, flexible, and easy to use.
Using the result of this parser you can perform insert/update as per your database structure.
This component is based on the Python Pandas Library. We use Pandas ExcelReader & CSVReader for reading excel & csv respectively.
### Installation
#### Prerequisite:
- Python3
Use the following command to download pip directly,
```bash
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
```
Now execute the downloaded file using below command
```bash
python3 get-pip.py
```
```bash
pip3 install pandas
pip3 install xlrd
pip3 install xlsxwriter
```
### Documentation
We have given below functionality here
#### Supported EXCEL & CSV File Format
1. Read Excel/CSV file data
2. Config file validation
3. Data Validations
- Added file type validation
- Added required column validation on header
- Added validation for each row for required data
- Added Boolean Fields validation
4. Downloads Excel/CSV for status of data.
- Here we will export excel which will have uploaded data status.
- Added Errors column if any data for in the row
5. Supports multi threading for data processing
#### Configurations:
- Config file contains all the configuration needed to perform validations on the data. so if you want to add your own entity validations then you have to create new config file similar to user.json in Config-Files folder.
- config.py contains basic configurations like supported files type, validation messages, etc.
- You can refer files from Sample-Data-Files folder for initial run.
#### To Run
```bash
python3 data_parser.py -i -e
```
### Upcoming fetures
1. Validations on date, time fields
2. Google sheet support