https://github.com/crazyoptimist/data-loader-go-poc

Load data into DB in a scalable manner
https://github.com/crazyoptimist/data-loader-go-poc

Last synced: 13 days ago
JSON representation

Load data into DB in a scalable manner

• Host: GitHub
• URL: https://github.com/crazyoptimist/data-loader-go-poc
• Owner: crazyoptimist
• Created: 2024-02-01T10:52:06.000Z (9 months ago)
• Default Branch: main
• Last Pushed: 2024-02-02T22:04:23.000Z (9 months ago)
• Last Synced: 2024-10-03T21:41:14.882Z (about 1 month ago)
• Language: Go
• Size: 10.7 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Data Loader POC

This is a proof of concept utilizing PostgreSQL/MySQL connection pooling along with Go's concurrency worker pool pattern.

It downloads a CSV dataset file from the internet, and loads one million records into DB.

### Instructions

Run a PostgreSQL/MySQL database locally either using Docker or not.

Create `.env` file from `.env.example`.

Run

```bash

go run ./cmd/pgloader/main.go

```

```bash

go run ./cmd/mysqlloader/main.go

```

[_crazyoptimist_](https://crazyoptimist.net)