https://github.com/holistics/pgcp

Copying tables between Postgres databases (for analytics purpose)
https://github.com/holistics/pgcp

data-engineering tools

Last synced: 3 months ago
JSON representation

Copying tables between Postgres databases (for analytics purpose)

• Host: GitHub
• URL: https://github.com/holistics/pgcp
• Owner: holistics
• Archived: true
• Created: 2015-10-29T07:38:03.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2023-03-16T16:37:24.000Z (almost 2 years ago)
• Last Synced: 2024-08-13T07:15:44.329Z (6 months ago)
• Topics: data-engineering, tools
• Language: Ruby
• Homepage:
• Size: 27.3 KB
• Stars: 7
• Watchers: 4
• Forks: 0
• Open Issues: 5
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• jimsghstars - holistics/pgcp - Copying tables between Postgres databases (for analytics purpose) (Ruby)

README

        
# pgcp - Copy PostgreSQL tables between databases (for analytics purpose)

pgcp is a simple tool to copy tables from a source postgres DB to a destination postgres DB.

The main use case of the tool is to make a dump of data from production databases

 to a centralized database for analytics purpose. Don't expect the copy to be an exact copy,

 we do drop a lot of table's metadata and only retain what we think is important for analytics.

What we retain:

* Table's schema structure

* Table's indexes

* Table's data

What we don't clone over (non-exhaustive):

* Sequences (auto increment)

* Triggers

* ...

Requirements:

* Make sure you have `psql` installed on the machine running `pgcp`

## Installation

Install gem from RubyGems

    gem install pgcp

Create file `~/.pgcp.yml` that contains the credentials of your interested databases:

    databases:

      production_db:

        user: postgres

        password:

        dbname: 

        host: your_production.server.com

      analytics_db:

        user: postgres

        dbname: analytics

        password:

        host: your_analytics.server.com

## Usage

For usage details, runs: `pgcp help`

Copy a single `bookings` table from production to analytics:

    pgcp -s production_db -d analytics_db -t public.bookings 

Copy multiple tables:

    pgcp -s production_db -d analytics_db -t public.bookings public.registrations

Copy all tables in schema public to destination database

    pgcp -s production_db -d analytics_db -t public.*

Copy all tables in schema public to destination database, but to a different schema

    pgcp -s production_db -d analytics_db -t public.* --force-schema