{"id":13538689,"url":"https://github.com/timescale/timescaledb-parallel-copy","last_synced_at":"2025-05-14T07:08:22.826Z","repository":{"id":37885231,"uuid":"95589699","full_name":"timescale/timescaledb-parallel-copy","owner":"timescale","description":"A binary for parallel copying of CSV data into a TimescaleDB hypertable","archived":false,"fork":false,"pushed_at":"2025-04-29T11:16:07.000Z","size":417,"stargazers_count":397,"open_issues_count":15,"forks_count":56,"subscribers_count":24,"default_branch":"main","last_synced_at":"2025-05-07T04:58:48.133Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.timescale.com/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/timescale.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-06-27T18:42:07.000Z","updated_at":"2025-04-29T11:16:11.000Z","dependencies_parsed_at":"2022-08-15T17:20:57.585Z","dependency_job_id":"cb905f23-4ca6-42c9-b9a2-6f8bb88087ee","html_url":"https://github.com/timescale/timescaledb-parallel-copy","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timescale%2Ftimescaledb-parallel-copy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timescale%2Ftimescaledb-parallel-copy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timescale%2Ftimescaledb-parallel-copy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timescale%2Ftimescaledb-parallel-copy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/timescale","download_url":"https://codeload.github.com/timescale/timescaledb-parallel-copy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254092659,"owners_count":22013290,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T09:01:14.853Z","updated_at":"2025-05-14T07:08:22.801Z","avatar_url":"https://github.com/timescale.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# timescaledb-parallel-copy\n\n`timescaledb-parallel-copy` is a command line program for parallelizing\nPostgreSQL's built-in `COPY` functionality for bulk inserting data\ninto [TimescaleDB.](//github.com/timescale/timescaledb/)\n\n## Installation\n\n\u003cdetails\u003e\n\u003csummary\u003eDocker\u003c/summary\u003e\n\n```sh\ndocker pull timescale/timescaledb-parallel-copy\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eGo\u003c/summary\u003e\n\nYou need the Go runtime (1.13+) installed, then simply `go get` this repo:\n\n```sh\ngo install github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy@latest\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eBrew\u003c/summary\u003e\n\n-   Add the TimescaleDB Homebrew tap.\n\n```sh\nbrew tap timescale/tap\n```\n\n-   Install timescaledb-parallel-copy.\n\n```sh\nbrew install timescaledb-tools\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eDebian\u003c/summary\u003e\n\n-   Install packages needed for the installation.\n\n```sh\nsudo apt install gnupg lsb-release wget\n```\n\n-   Add the TimescaleDB repository.\n\n```sh\necho \"deb https://packagecloud.io/timescale/timescaledb/debian/ $(lsb_release -c -s) main\" | sudo tee /etc/apt/sources.list.d/timescaledb.list\n```\n\n-   Install the TimescaleDB GPG key.\n\n```sh\nwget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg\n```\n\n-   Install the tools package which contains `timescaledb-parallel-copy`.\n\n```sh\nsudo apt install timescaledb-tools\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eUbuntu\u003c/summary\u003e\n\n-   Install packages needed for the installation.\n\n```sh\nsudo apt install gnupg lsb-release wget\n```\n\n-   Add the TimescaleDB repository.\n\n```sh\necho \"deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main\" | sudo tee /etc/apt/sources.list.d/timescaledb.list\n```\n\n-   Install the TimescaleDB GPG key.\n\n```sh\nwget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg\n```\n\n-   Install the tools package which contains `timescaledb-parallel-copy`.\n\n```sh\nsudo apt install timescaledb-tools\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eRedHat\u003c/summary\u003e\n\n-   Add the TimescaleDB repository.\n\n```sh\nsudo tee /etc/yum.repos.d/timescale_timescaledb.repo \u003c\u003cEOL\n[timescale_timescaledb]\nname=timescale_timescaledb\nbaseurl=https://packagecloud.io/timescale/timescaledb/el/$(rpm -E %{rhel})/\\$basearch\nrepo_gpgcheck=1\ngpgcheck=0\nenabled=1\ngpgkey=https://packagecloud.io/timescale/timescaledb/gpgkey\nsslverify=1\nsslcacert=/etc/pki/tls/certs/ca-bundle.crt\nmetadata_expire=300\nEOL\n```\n\n-   Install the tools package which contains `timescaledb-parallel-copy`.\n\n```sh\nsudo yum install timescaledb-tools\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eFedora\u003c/summary\u003e\n\n-   Add the TimescaleDB repository.\n\n```sh\nsudo tee /etc/yum.repos.d/timescale_timescaledb.repo \u003c\u003cEOL\n[timescale_timescaledb]\nname=timescale_timescaledb\nbaseurl=https://packagecloud.io/timescale/timescaledb/el/9/\\$basearch\nrepo_gpgcheck=1\ngpgcheck=0\nenabled=1\ngpgkey=https://packagecloud.io/timescale/timescaledb/gpgkey\nsslverify=1\nsslcacert=/etc/pki/tls/certs/ca-bundle.crt\nmetadata_expire=300\nEOL\n```\n\n-   Install the tools package which contains `timescaledb-parallel-copy`.\n\n```sh\nsudo yum install timescaledb-tools\n```\n\u003c/details\u003e\n\n## Usage\n\nBefore using this program to bulk insert data, your database should\nbe installed with the TimescaleDB extension and the target table\nshould already be made a hypertable.\n\nIf you want to bulk insert data from a file named `foo.csv` into a\n(hyper)table named `sample` in a database called `test`:\n\n```bash\n# single-threaded\n$ timescaledb-parallel-copy --connection $DATABASE_URL --table sample --file foo.csv\n\n# 2 workers\n$ timescaledb-parallel-copy --connection $DATABASE_URL --table sample --file foo.csv \\\n    --workers 2\n\n# 2 workers, report progress every 30s\n$ timescaledb-parallel-copy --connection $DATABASE_URL --table sample --file foo.csv \\\n    --workers 2 --reporting-period 30s\n\n# Treat literal string 'NULL' as NULLs:\n$ timescaledb-parallel-copy --connection $DATABASE_URL --table sample --file foo.csv \\\n    --copy-options \"NULL 'NULL' CSV\"\n```\n\nOther options and flags are also available:\n\n```\n$ timescaledb-parallel-copy --help\n\nUsage of timescaledb-parallel-copy:\n  -batch-error-output-dir string\n        directory to store batch errors. Settings this will save a .csv file with the contents of the batch that failed and continue with the rest of the data.\n  -batch-size int\n        Number of rows per insert (default 5000)\n  -columns string\n        Comma-separated columns present in CSV\n  -connection string\n        PostgreSQL connection url (default \"host=localhost user=postgres sslmode=disable\")\n  -copy-options string\n        Additional options to pass to COPY (e.g., NULL 'NULL') (default \"CSV\")\n  -db-name string\n        (deprecated) Database where the destination table exists\n  -escape character\n        The ESCAPE character to use during COPY (default '\"')\n  -file string\n        File to read from rather than stdin\n  -header-line-count int\n        Number of header lines (default 1)\n  -limit int\n        Number of rows to insert overall; 0 means to insert all\n  -log-batches\n        Whether to time individual batches.\n  -quote character\n        The QUOTE character to use during COPY (default '\"')\n  -reporting-period duration\n        Period to report insert stats; if 0s, intermediate results will not be reported\n  -schema string\n        Destination table's schema (default \"public\")\n  -skip-batch-errors\n        if true, the copy will continue even if a batch fails\n  -skip-header\n        Skip the first line of the input\n  -split string\n        Character to split by (default \",\")\n  -table string\n        Destination table for insertions (default \"test_table\")\n  -truncate\n        Truncate the destination table before insert\n  -verbose\n        Print more information about copying statistics\n  -version\n        Show the version of this tool\n  -workers int\n        Number of parallel requests to make (default 1)\n\n```\n\n## Purpose\n\nPostgreSQL native `COPY` function is transactional and single-threaded, and may not be suitable for ingesting large\namounts of data. Assuming the file is at least loosely chronologically ordered with respect to the hypertable's time\ndimension, this tool should give you great performance gains by parallelizing this operation, allowing users to take\nfull advantage of their hardware.\n\nThis tool also takes care to ingest data in a more efficient manner by roughly preserving the order of the rows. By\ntaking a \"round-robin\" approach to sharing inserts between parallel workers, the database has to switch between chunks\nless often. This improves memory management and keeps operations on the disk as sequential as possible.\n\n## Contributing\n\nWe welcome contributions to this utility, which like TimescaleDB is released under the Apache2 Open Source License. The same [Contributors Agreement](//github.com/timescale/timescaledb/blob/master/CONTRIBUTING.md) applies; please sign the [Contributor License Agreement](https://cla-assistant.io/timescale/timescaledb-parallel-copy) (CLA) if you're a new contributor.\n\n### Running Tests\n\nSome of the tests require a running Postgres database. Set the `TEST_CONNINFO`\nenvironment variable to point at the database you want to run tests against.\n(Assume that the tests may be destructive; in particular it is not advisable to\npoint the tests at any production database.)\n\nFor example:\n\n```\n$ createdb gotest\n$ TEST_CONNINFO='dbname=gotest user=myuser' go test -v ./...\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimescale%2Ftimescaledb-parallel-copy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimescale%2Ftimescaledb-parallel-copy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimescale%2Ftimescaledb-parallel-copy/lists"}