https://github.com/ilyass-bougati/cdb

This is a project is all about turning a big CSV file to a postgres table, it automatically detects the type and generates a schema and create the table for you
https://github.com/ilyass-bougati/cdb

data-science docker-compose pandas postgresql python

Last synced: 2 months ago
JSON representation

This is a project is all about turning a big CSV file to a postgres table, it automatically detects the type and generates a schema and create the table for you

• Host: GitHub
• URL: https://github.com/ilyass-bougati/cdb
• Owner: Ilyass-Bougati
• Created: 2025-02-05T16:54:20.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2025-06-29T09:54:34.000Z (about 1 year ago)
• Last Synced: 2025-06-29T10:35:00.449Z (about 1 year ago)
• Topics: data-science, docker-compose, pandas, postgresql, python
• Language: Python
• Homepage:
• Size: 688 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# CDB - CSV to DB

![python](https://img.shields.io/badge/Python-FFD43B?style=for-the-badge&logo=python&logoColor=blue)

![pandas](https://img.shields.io/badge/Pandas-2C2D72?style=for-the-badge&logo=pandas&logoColor=white)

![postgres](https://img.shields.io/badge/PostgreSQL-316192?style=for-the-badge&logo=postgresql&logoColor=white)

![docker](https://img.shields.io/badge/Docker-2CA5E0?style=for-the-badge&logo=docker&logoColor=white)

This is a project is all about turning a big CSV file to a [postgres](https://www.postgresql.org/) table, it automatically detects the type and generates a schema and create the table for you, all you need to do is ensure there are no duplicate column names and that the column names don't go over the postgres limit of size (31 as I know).

# Environment variables

Before running the project, you must create a `.env` file where you define the following variables

```env

POSTGRES_USER=...

POSTGRES_PASSWORD=...

TABLE_NAME=...

```

# Database

If you don't have postgres installed, you can use [docker](https://www.docker.com/), and run the following command to start a postgresql container

```bash

docker compose up

```

# Cloning

Since I've used one submodule [plog](https://github.com/Ilyass-Bougati/plog) you'll have to clone this repo recursively

```bash

git clone --recursive git@github.com:Ilyass-Bougati/CDB.git

```

# Usage

running the program after setting things up, is as easy as it could get

```bash

python cdb.py filepath

```