https://github.com/chfern/bucardo-sandbox
Bucardo sandbox to sync 2 DBs
https://github.com/chfern/bucardo-sandbox
Last synced: 5 months ago
JSON representation
Bucardo sandbox to sync 2 DBs
- Host: GitHub
- URL: https://github.com/chfern/bucardo-sandbox
- Owner: chfern
- Created: 2021-01-24T16:03:32.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2021-01-24T16:14:26.000Z (over 5 years ago)
- Last Synced: 2025-09-07T08:21:05.060Z (10 months ago)
- Language: Dockerfile
- Size: 4.88 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Bucardo Sandbox
A sandbox about replicating data from 1 postgres DB to another using [bucardo](https://bucardo.org/).
This sandbox uses docker-compose for linux package installation and database creation only (this is intentional).
Versions used:
- PostgreSQL - 11.9
- Bucardo - 5.6.0
## Scenario
We would have 2 databases (source database and destination database). Data in some tables would then be synced from src to dest DB.
Source host: 192.168.99.3
Source database: booking-db
Source port: 5432
Source username: postgres
Source password: AAAaaa123!@#
Destination host: 192.168.99.3
Destination database: booking-db
Destination port: 6432
Destination username: postgres
Destination password: AAAaaa123!@#
Tables to-sync from source database:
- `schema_to_migrate.bookings`
- `schema_to_migrate.booking_histories`
# Running the Sandbox
## Pre-requisites
You need to have these installed in your system beforehand:
- Docker
- Docker Compose
- Any DB GUI or psql to inspect the database later on
## Sandbox Setup
Setup static IP on host
`sudo ifconfig lo0 alias 192.168.99.3`
In the root project directory, run `docker-compose up`. This spawns the 2 database containers with required linux packages and postgres plugin.
Execute bash using root user in source postgres container by running `docker exec -u root -it $(docker ps -aqf "name=postgres-11-src$") /bin/bash`
### Environment Variables Setup
For convenience purpose, set the informations to env variables
```
export SOURCE_HOST=192.168.99.3
export SOURCE_PORT=5432
export SOURCE_DATABASE=booking-db
export SOURCE_USERNAME=postgres
export SOURCE_PASSWORD='AAAaaa123!@#'
export DEST_HOST=192.168.99.3
export DEST_PORT=6432
export DEST_DATABASE=booking-db
export DEST_USERNAME=postgres
export DEST_PASSWORD='AAAaaa123!@#'
export TABLES="schema_to_migrate.bookings schema_to_migrate.booking_histories"
```
### Source Database & Destination Database Setup
Connect to the src database using psql to create tables and optionally seed dummy data.
```
psql -U $SOURCE_USERNAME -h $SOURCE_HOST $SOURCE_DATABASE
CREATE SCHEMA schema_to_migrate;
CREATE TABLE schema_to_migrate.bookings(booking_id int primary key);
CREATE TABLE schema_to_migrate.booking_histories(booking_history_id int primary key);
INSERT INTO schema_to_migrate.bookings(booking_id) values (1);
\q
```
Because destination database is freshly deployed, dump schema info from source database and update destination database with the schema information
```
cd /tmp && \
pg_dump -U $SOURCE_USERNAME -d $SOURCE_DATABASE --schema-only --no-privileges --no-owner --no-security-labels --no-synchronized-snapshots --no-tablespaces > schema.sql && \
psql -h $DEST_HOST -p $DEST_PORT -d $DEST_DATABASE -U $DEST_USERNAME -f schema.sql
```
### Bucardo Setup
Create a new linux user for bucardo
```
export BUCARDO_USER=bucardo
sudo useradd $BUCARDO_USER -m && \
sudo passwd $BUCARDO_USER
```
Download bucardo distribution in well-known `src` folder
```
cd /usr/local/src && \
wget https://github.com/bucardo/bucardo/archive/5.6.0.tar.gz && \
tar zxvf 5.6.0.tar.gz && \
rm 5.6.0.tar.gz && \
cd bucardo-5.6.0 && \
perl Makefile.PL && \
make && \
make install && \
sudo chown -R $BUCARDO_USER:$BUCARDO_USER /usr/local/src/bucardo-5.6.0
```
Create other folders required by bucardo
```
sudo mkdir -p /var/log/bucardo /var/run/bucardo && \
sudo chown $BUCARDO_USER:$BUCARDO_USER /var/log/bucardo /var/run/bucardo
```
Switch to `bucardo` user then [re-apply the env variables](#environment-variables-setup)
```
su -l bucardo
```
### Configuring Bucardo's PostgreSQL
Create .pgpass file to connect to local and destination DB, then secure it due to it containing credential information
```
cat > $HOME/.pgpass < $HOME/.bucardorc <