Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/andrekuratomi/tables_to_db
A python module that automatizes csv tables insertion to a MySQL database.
https://github.com/andrekuratomi/tables_to_db
mysql pandas pymysql python sqlalchemy
Last synced: about 1 month ago
JSON representation
A python module that automatizes csv tables insertion to a MySQL database.
- Host: GitHub
- URL: https://github.com/andrekuratomi/tables_to_db
- Owner: AndreKuratomi
- Created: 2023-09-11T21:07:50.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-10-28T22:05:48.000Z (3 months ago)
- Last Synced: 2024-10-28T23:18:28.249Z (3 months ago)
- Topics: mysql, pandas, pymysql, python, sqlalchemy
- Homepage:
- Size: 71.3 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# tables_to_db
- [Translations](#translations)
- [About](#about)
- [Instalation](#instalation)
- [How to use this module](#How-does-the-module-work)
- [References](#references)
## Translations
- [🇧🇷 Português brasileiro / Brazilian portuguese](/.multilingual_readmes/README.pt-br.md)
- [🇬🇧 / 🇺🇸 English](https://github.com/AndreKuratomi/tables_to_db)
## About
tables_to_db is a python module that automatises csv tables insertion in a MySQL database using PyMySQL an Pandas. With this a manual insertion via some interface such as MySQL Workbench, for instance, is dispensable.
This module can be easily used in any major project that may need its functionality.
## Instalation
0. Before using this module it is first necessary to have instaled the following devices:
- The code versioning [Git](https://git-scm.com/downloads).
- The language [Python](https://www.python.org/downloads/)
- A code editor, also known as IDE. For instance, [Visual Studio Code (VSCode)](https://code.visualstudio.com/).
-
And versioning your directory to receive the aplication clone:
```
git init
```
1. Clone the repository tables_to_db by your machine terminal or by the IDE:
```
git clone https://github.com/AndreKuratomi/tables_to_db.git
```
WINDOWS:
Obs: In case of any mistake similar to this one:
```
unable to access 'https://github.com/AndreKuratomi/tables_to_db.git/': SSL certificate problem: self-signed certificate in certificate chain
```
Configure git to disable SSL certification:
```
git config --global http.sslVerify "false"
```
Enter the directory:
```
cd tables_to_db
```
2. After cloning the repository install:
Virtual enviroment* and update its dependencies with the following command:
LINUX:
```
python3 -m venv venv --upgrade-deps
```
WINDOWS:
```
py -m venv venv --upgrade-deps
```
In case an error like this one is returned just follow the command displayed:
```
The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt install python3.10-venv
You may need to use sudo with that command. After installing the python3-venv
package, recreate your virtual environment.
```
*It is a good practice to work with virtual enviroments because different projects may need different dependencies. A virtual enviroment is only a separated enviroment from the user machine. If not used, the user's machine may have lots of dependencies intalled that may only be used in a single project.
Activate your virtual enviroment with the command:
LINUX:
```
source/venv/bin/activate
```
WINDOWS:
On Windows operational system it is necessary to configure the Execution Policy at PowerShell:
```
Get-ExecutionPolicy # to check the Execution policy type
Set-ExecutionPolicy RemoteSigned # to change the type of policy if the command above shows 'Restricted'
```
Obs: It may often be necessary to open PowerShell as administrador for that.
```
.\env\Scripts\activate
```
Install the module's dependencies:
```
pip install -r requirements.txt
```
WINDOWS:
In case any error similar to the one bellow be returned:
```
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\\Users\\andre.kuratomi\\OneDrive - Company\\Área de Trabalho\\tables_to_db_mail_for_finances\\tables_to_db_and_mail_finances\\env\\Lib\\site-packages\\jedi\\third_party\\django-stubs\\django-stubs\\contrib\\contenttypes\\management\\commands\\remove_stale_contenttypes.pyi'
HINT: This error might have occurred since this system does not have Windows Long Path support enabled. You can find information on how to enable this at https://pip.pypa.io/warnings/enable-long-paths
```
Run cmd as adminstrador with the following command:
```
reg.exe add HKLM\SYSTEM\CurrentControlSet\Control\FileSystem /v LongPathsEnabled /t REG_DWORD /d 1 /f
```
3. Open the application on your IDE:
```
code .
```
4. Create .env file in the root directoy:
```
touch .env
```
Inside it we need to put our enviroment variables taking as reference the given file .env.example:
```
MYSQL_USER=user
MYSQL_PASSWORD=password
```
Obs: Do not share info from .env file. It is already mentioned in .gitignore for not being pushed to the repo.
## How to use this module
The module tables_to_db.insert_tables_to_a_db() has two parameters: 'db' and 'tables_path'.
It was originally developed for a project that used MySQL (db) and CSV tables located in a separated folder (tables_path). This module came in substitution of manual insertion of this tables to MySQL using MySQL Workbench, saving time.
So this module can be simply instatiated in a project like this:
```
db = 'my_mysql_database'
tables_path = '/path/to/csv/tables'
insert_tables_to_a_db(db, tables_path)
```
## References
- [Git](https://git-scm.com/downloads)
- [MySQL](https://https://www.mysql.com/)
- [Pandas](https://pandas.pydata.org/docs/)
- [Python](https://www.python.org/downloads/)
- [PyMySQL](https://pypi.org/project/PyMySQL/)