{"id":23028945,"url":"https://github.com/maiha/pg-copy-ch","last_synced_at":"2025-08-14T12:34:21.775Z","repository":{"id":42009796,"uuid":"273719206","full_name":"maiha/pg-copy-ch","owner":"maiha","description":"Simply copy the current PostgreSQL data to ClickHouse","archived":false,"fork":false,"pushed_at":"2022-11-02T11:04:21.000Z","size":59,"stargazers_count":10,"open_issues_count":2,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2023-04-10T09:13:01.448Z","etag":null,"topics":["backup","clickhouse","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Crystal","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maiha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-20T14:03:11.000Z","updated_at":"2022-11-02T11:06:51.000Z","dependencies_parsed_at":"2023-01-20T20:02:48.429Z","dependency_job_id":null,"html_url":"https://github.com/maiha/pg-copy-ch","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maiha%2Fpg-copy-ch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maiha%2Fpg-copy-ch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maiha%2Fpg-copy-ch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maiha%2Fpg-copy-ch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maiha","download_url":"https://codeload.github.com/maiha/pg-copy-ch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229827764,"owners_count":18130394,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","clickhouse","postgresql"],"created_at":"2024-12-15T14:13:28.725Z","updated_at":"2024-12-15T14:13:29.463Z","avatar_url":"https://github.com/maiha.png","language":"Crystal","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pg-copy-ch [![Build Status](https://travis-ci.org/maiha/pg-copy-ch.svg?branch=master)](https://travis-ci.org/maiha/pg-copy-ch)\n\nSimply copy the current PostgreSQL data to ClickHouse\n* Simple    : Just dumps and imports, so it works with older versions like 9.x.\n* Handy     : Static single binary, so you can install it just by wget or cp.\n* Automatic : Create ClickHouse tables from PostgreSQL automatically.\n* Easy      : Include/Exclude tables by regex. Skip by max data count, ttl.\n\n```console\n$ psql mydb -c \"SELECT count(*) FROM users\"             # =\u003e 3835\n$ clickhouse-client -q \"CREATE DATABASE pg\"\n\n$ pg-copy-ch init config --pg-db=mydb --ch-db=pg\n$ pg-copy-ch copy -t users\n[09:05:28] (1/1) users REPLACED 3835 (0.35s)\n\n$ clickhouse-client -q \"SELECT count(*) FROM pg.users\"  # =\u003e 3835\n```\n\n## Installation\n* **psql** : required for PostgreSQL client\n* **clickhouse-client** : required for ClickHouse client\n* x86_64 static binary: https://github.com/maiha/pg-copy-ch/releases\n\n```console\n$ wget https://github.com/maiha/pg-copy-ch/releases/latest/download/pg-copy-ch\n```\n\n## Usage\n\n### config\n\nFirst, create config file by `init config`.\n\n```console\n$ pg-copy-ch init config\nInitialized empty config in .pg-copy-ch/config\n```\n\nThen, edit `config` file about connection settings for PostgreSQL and ClickHouse.\n\n```console\n$ vi .pg-copy-ch/config\n[postgres]\nhost = \"pg-server1\"\nport = 5432\nuser = \"postgres\"\ndb   = \"mydb\"\npsql = \"psql -h %host -p %port -U %user %db -w\"\n...\n```\n\n### Copy\n\nOnce you setup config, you can run `copy` with specifying the table by one of '-a', '-t', '-f', '-F'.\n\n```console\n$ pg-copy-ch copy -t users,orders # Copy only the specified tables\n$ pg-copy-ch copy -a              # Copy all tables\n$ pg-copy-ch copy -f \u003cALLOW_FILE\u003e # Copy all tables both in the config and in \u003cALLOW＿FILE\u003e.\n$ pg-copy-ch copy -F \u003cDENY_FILE\u003e  # Copy all tables in the config and NOT in \u003cDENY_FILE\u003e.\n```\n\n### Filter by allow\n\n'-f \u003cFILE\u003e' obtains allow table names from the FILE, one per line.\nWe can create it by the `init tables` command.\n\n```console\n$ pg-copy-ch init tables\nCreated .pg-copy-ch/tables\n```\n\nThen, edit or comment out as you like.\n\n```console\n$ vi .pg-copy-ch/tables\n# budgets\ncreatives\norders\n...\n\n$ pg-copy-ch copy -f .pg-copy-ch/tables\n```\n\n### Filter by deny\n\n'-F \u003cFILE\u003e' skips table names written in the FILE.\nThis works as a regular expression if the line contains '^' or '$'.\nFor example, this will ignore all PostgreSQL system tables.\n\n```console\n$ vi ignores\n^pg_\n\n$ pg-copy-ch copy -F ignores\n```\n\n### Dryrun\n\n'-n' just prints the actions that would be executed, but do not execute them.\n\n```console\n$ pg-copy-ch copy -t users,orders,xxx -n\nTable  PostgreSQL Action\n------ ---------- ----------------------------\nusers  FOUND      (will) Replace\norders FOUND      (will) Replace\nxxx    N/A        Ignore (PG schema not found)\n```\n\n## Find performance killers\n\nFirst, run and generate log.\n\n```console\n$ pg-copy-ch -a | tee log\n```\n\n##### worst record counts\n\n```console\n$ grep REPLACED log | sort -n -k 5 -r | head -3\n[05:28:26] (092/428) creatives    REPLACED 11989140 (37.01s)\n[05:29:28] (157/428) constraints  REPLACED 5765600 (2.79s)\n[05:31:15] (280/428) eviews       REPLACED 4460582 (5.67s)\n```\n\n##### worst time\n\n```console\n$ grep REPLACED log | sort -n -t'(' -k 3 -r | head -3\n[05:28:26] (092/428) creatives    REPLACED 11989140 (37.01s)\n[05:26:38] (008/428) schedules    REPLACED 1115075 (22.90s)\n[05:32:47] (343/428) statistics   REPLACED 2443266 (8.98s)\n```\n\n## PostgreSQL\n\n### psql\n\nFor example, if you want to use SSL, you can specify the command directly to `psql` in config.\nHere, '%' prefixed words are replaced by those settings automatically.\n\n```toml\n[postgres]\npsql = \"psql -h %host -p %port -U %user %db -w --dbname=postgres --set=sslmode=require --set=sslrootcert=./sslcert.crt\"\n```\n\n### authorization\n\nUsing `~/.pgpass` is a easiest way to specify a password.\n\nIf it is difficult to write to HOME by cron execution, you can embed it directly into the config file using `psql` above.\n\n```toml\n[postgres]\npsql = \"PGPASSWORD=foo psql -h %host -p %port -U %user %db -w\"\n```\n\n### before_sql\n\n\nThe `before_sql` allows you to insert an arbitrary SQL.\nThis is useful for using not `public` schema but a custom schema.\n\n```toml\n[postgres]\nbefore_sql      = \"SET search_path to myschema,public;\"\n```\n\n## Development\n\n* using [Crystal](http://crystal-lang.org/) on docker\n\n```console\n$ make\n```\n\n## Test\n\n```\n$ make ci\n```\n\n## Contributing\n\n1. Fork it (\u003chttps://github.com/maiha/pg-copy-ch/fork\u003e)\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create a new Pull Request\n\n## Contributors\n\n- [maiha](https://github.com/maiha) - creator and maintainer\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaiha%2Fpg-copy-ch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaiha%2Fpg-copy-ch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaiha%2Fpg-copy-ch/lists"}