{"id":22415026,"url":"https://github.com/nitor-infotech-oss/python-bulk-data-parser","last_synced_at":"2025-03-27T04:16:46.191Z","repository":{"id":98592260,"uuid":"326664345","full_name":"nitor-infotech-oss/python-bulk-data-parser","owner":"nitor-infotech-oss","description":"A Component for reading excel files data and used similar data for bulk data upload after perform validation. Extremely fast, flexible, and easy to use. Using the result of this parser you can perform insert/update as per your database structure.","archived":false,"fork":false,"pushed_at":"2021-01-05T09:15:48.000Z","size":44,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-01T09:22:21.680Z","etag":null,"topics":["bulk-loader","data-parser","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nitor-infotech-oss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-04T11:37:54.000Z","updated_at":"2021-01-05T09:19:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"e6399200-1b0f-4b6e-a644-3261bef5dc4e","html_url":"https://github.com/nitor-infotech-oss/python-bulk-data-parser","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nitor-infotech-oss%2Fpython-bulk-data-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nitor-infotech-oss%2Fpython-bulk-data-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nitor-infotech-oss%2Fpython-bulk-data-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nitor-infotech-oss%2Fpython-bulk-data-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nitor-infotech-oss","download_url":"https://codeload.github.com/nitor-infotech-oss/python-bulk-data-parser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245779481,"owners_count":20670686,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bulk-loader","data-parser","python3"],"created_at":"2024-12-05T15:10:49.345Z","updated_at":"2025-03-27T04:16:46.155Z","avatar_url":"https://github.com/nitor-infotech-oss.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Bulk Excel/CSV Data Parser\n\nA Component for reading excel files data and used similar data for bulk data upload after perform validation. Extremely fast, flexible, and easy to use. \n\nUsing the result of this parser you can perform insert/update as per your database structure.\n\nThis component is based on the Python Pandas Library. We use Pandas ExcelReader \u0026 CSVReader for reading excel \u0026 csv respectively.\n\n### Installation\n\n#### Prerequisite:\n- Python3\n\n\nUse the following command to download pip directly,\n\n```bash\ncurl https://bootstrap.pypa.io/get-pip.py -o get-pip.py\n```\nNow execute the downloaded file using below command\n\n```bash\npython3 get-pip.py\n```\n\n```bash\npip3 install pandas\npip3 install xlrd\npip3 install xlsxwriter\n```\n### Documentation\n\nWe have given below functionality here\n\n#### Supported EXCEL \u0026 CSV File Format\n\n1. Read Excel/CSV file data\n2. Config file validation\n3. Data Validations\n    - Added file type validation\n\t- Added required column validation on header\n\t- Added validation for each row for required data\n    - Added Boolean Fields validation\n4. Downloads Excel/CSV for status of data.\n\t- Here we will export excel which will have uploaded data status.\n\t- Added Errors column if any data for in the row\n5. Supports multi threading for data processing\n\n#### Configurations:\n- Config file contains all the configuration needed to perform validations on the data. so if you want to add your own entity validations then you have to create new config file similar to user.json in Config-Files folder. \n- config.py contains basic configurations like supported files type, validation messages, etc.\n- You can refer files from Sample-Data-Files folder for initial run. \n\n#### To Run \n\n```bash\npython3 data_parser.py -i \u003cinputfile\u003e -e \u003centitytype\u003e\n```\n\n### Upcoming fetures\n\n1. Validations on date, time fields\n2. Google sheet support ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnitor-infotech-oss%2Fpython-bulk-data-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnitor-infotech-oss%2Fpython-bulk-data-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnitor-infotech-oss%2Fpython-bulk-data-parser/lists"}