https://github.com/soartec-lab/go-gcs-to-pg-data-importer

Data import from GCS to PostgreSQL by golang
https://github.com/soartec-lab/go-gcs-to-pg-data-importer

Last synced: about 1 month ago
JSON representation

Data import from GCS to PostgreSQL by golang

• Host: GitHub
• URL: https://github.com/soartec-lab/go-gcs-to-pg-data-importer
• Owner: soartec-lab
• Created: 2022-01-23T07:47:59.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2022-01-30T07:34:46.000Z (almost 3 years ago)
• Last Synced: 2024-10-22T02:13:14.613Z (2 months ago)
• Language: Go
• Size: 23.4 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# go-gcs-to-pg-data-importer

Data import from GCS to PostgreSQL by golang

# Setup

## Run docker containers

1. Run docker-compose

```bash

$ docker-compose up

```

2. connect GCS fake server:

```bash

$ docker-compose exec app curl --insecure https://gcs:4443/storage/v1/b/tables/o

#=> {"kind":"storage#objects","items":[{"kind":"storage#object","name":"users.json","id":"tables/users.json","bucket":"tables","size":"63","contentType":"application/json","crc32c":"VuKJWg==","md5Hash":"2lbYdXcwrlOlP6Q7efN/pQ==","timeCreated":"2022-01-23T08:05:22.41268Z","timeDeleted":"0001-01-01T00:00:00Z","updated":"2022-01-23T08:05:22.412714Z","generation":"0"}]}

```

3. connect PostgreSQL server:

```bash

$ docker-compose exec app curl http://postgresql:5432

#=> curl: (52) Empty reply from server

```

## Database

Create inital data:

```bash

$ dc exec postgresql psql -f /init/create_users.sql -U postgres

#=> CREATE TABLE

```

# Docker images

## GCS images

Using [fake-gcs-server](https://github.com/fsouza/fake-gcs-server)