{"id":19223692,"url":"https://github.com/wmfs/supercopy","last_synced_at":"2025-05-13T10:48:57.657Z","repository":{"id":21730978,"uuid":"136929037","full_name":"wmfs/supercopy","owner":"wmfs","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-18T15:10:27.000Z","size":574,"stargazers_count":2,"open_issues_count":5,"forks_count":3,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-14T03:54:37.428Z","etag":null,"topics":["package","postgresql","tymly"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wmfs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-11T13:18:58.000Z","updated_at":"2025-03-18T15:10:30.000Z","dependencies_parsed_at":"2024-01-05T18:32:31.755Z","dependency_job_id":"a0b9aab2-a8d9-4734-9bc8-f4f9cc99683c","html_url":"https://github.com/wmfs/supercopy","commit_stats":null,"previous_names":[],"tags_count":53,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wmfs%2Fsupercopy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wmfs%2Fsupercopy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wmfs%2Fsupercopy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wmfs%2Fsupercopy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wmfs","download_url":"https://codeload.github.com/wmfs/supercopy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249977433,"owners_count":21354859,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["package","postgresql","tymly"],"created_at":"2024-11-09T15:09:16.745Z","updated_at":"2025-04-20T23:31:40.878Z","avatar_url":"https://github.com/wmfs.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# supercopy\n[![Tymly Package](https://img.shields.io/badge/tymly-package-blue.svg)](https://tymly.io/)\n[![npm (scoped)](https://img.shields.io/npm/v/@wmfs/supercopy.svg)](https://www.npmjs.com/package/@wmfs/supercopy)\n[![CircleCI](https://circleci.com/gh/wmfs/supercopy.svg?style=svg)](https://circleci.com/gh/wmfs/supercopy)\n[![codecov](https://codecov.io/gh/wmfs/supercopy/branch/master/graph/badge.svg)](https://codecov.io/gh/wmfs/supercopy)\n[![CodeFactor](https://www.codefactor.io/repository/github/wmfs/supercopy/badge)](https://www.codefactor.io/repository/github/wmfs/supercopy)\n[![Dependabot badge](https://img.shields.io/badge/Dependabot-active-brightgreen.svg)](https://dependabot.com/)\n[![Commitizen friendly](https://img.shields.io/badge/commitizen-friendly-brightgreen.svg)](http://commitizen.github.io/cz-cli/)\n[![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://standardjs.com)\n[![license](https://img.shields.io/github/license/mashape/apistatus.svg)](https://github.com/wmfs/tymly/blob/master/packages/pg-concat/LICENSE)\n\n\n\n\n\n\u003e Takes a specifically-named directory structure of CSV files and conjures bulk insert, update and delete statements and applies them to a PostgreSQL database. \n\n## \u003ca name=\"install\"\u003e\u003c/a\u003eInstall\n```bash\n$ npm install supercopy --save\n```\n\n## \u003ca name=\"usage\"\u003e\u003c/a\u003eUsage\n\n```javascript\nconst pg = require('pg')\nconst supercopy = require('supercopy')\n\n// Make a new Postgres client\nconst client = new pg.Client('postgres://postgres:postgres@localhost:5432/my_test_db')\n\nsupercopy(\n  {\n    sourceDir: '/dir/that/holds/deletes/inserts/updates/and/upserts/dirs',\n    headerColumnNamePkPrefix: '.',\n    topDownTableOrder: ['departments', 'employees'],\n    client: client,\n    schemaName: 'my_schema',\n    truncateTables: true,\n    debug: true,\n    multicopy: false,\n    directoryNames: { ... }\n  },\n  function (err) {\n    // Done!\n  }\n)\n\n```\n\n## supercopy(`options`, `callback`)\n\n### Options\n\n| Property              | Type       | Notes |\n| --------              | ----       | ------ |\n| `sourceDir`           | `function` | An absolute path pointing to a directory containing action folders. See the [File Structure](#structure) section for more details.\n| `headerColumnNamePkPrefix` | `string` | When conjuring an `update` statement, Supercopy will need to know which columns in the CSV file constitute a primary key. It does this by expecting the first line of each file to be a header containing `,` delimited column names. However, column names prefixed with this value should be deemed a primary-key column. Only use in update CSV-file headers.|\n| `topDownTableOrder`   | `[string]` | An array of strings, where each string is a table name. Table inserts will occur in this order and deletes in reverse - use to avoid integrity-constraint errors. If no schema prefix is supplied to a table name, then it's inferred from `schemaName`. \n| `client`              | `client`   | Either a [pg](https://www.npmjs.com/package/pg) client or pool (something with a `query()` method) that's already connected to a PostgreSQL database.\n| `schemaName`          | `string`   | Identifies a PostgreSQL schema where the tables that are to be affected by this copy be found.\n| `truncateTables`      | `boolean`  | A flag to indicate whether or not to truncate tables before supercopying into them\n| `debug`               | `boolean`  | Show debugging information on the console\n| `multicopy`           | `boolean`  | Enables 'sourceDir' to house many typical Supercopy 'sourceDir' shaped directories. Defaults to false.\n| `quote`               | `string`   | Override the the default quote character, \". It isn't necessary to quote fields but occasionally (especially when importing JSON fields) you need to, and this option will help.  \n| `directoryNames`      | `object`   | Overrides the default directory names - see below. \n\n### \u003ca name=\"structure\"\u003e\u003c/a\u003eFile structure\n\nThe directory identified by the `sourceDir` option should be structured in the following way:\n\n```\n/someDir\n  /inserts\n    table1.csv\n    table2.csv\n  /updates\n    table1.csv\n    table2.csv\n  /upserts\n    table1.csv\n    table2.csv  \n  /deletes\n    table1.csv\n    \nOR IF USING MULTICOPY\n\n/manyDirs\n /someDir\n  /inserts\n    table1.csv\n    table2.csv\n /someDir\n  /inserts\n    table1.csv\n    table2.csv   \n\n```\n\n#### Notes\n\n* The sub-directories here refer to the type of action that should be performed using CSV data files contained in it. Supported directory names are `insert`, `update`, `upsert` (try to update, failing that insert) and `delete`.\n* Directories are optional. A directory maybe missing or empty.\n* The `directoryNames` option can be used to apply actions to directories if the names don't meet the above structure. Eg \n`directoryName : { 'inserts': 'new', 'deletes': 'old' }` would insert the contents of the directory named `new` and remove the contents of the `old` directory.\n* The filename of each file should refer to a table name in the schema identified by the `schemaName` option. \n* The expected format of the .csv files is:\n  * One line per record\n  * The first line to be a comma delimited list of column names (i.e. a header record)\n  * For update and upsert files, ensure columns-names in the header record that are part of the primary key are identified with a `headerColumnNamePkPrefix` character.\n  * All records to be comma delimited, and any text columns containing a `,` should be quoted with a `\"`. The [csv-string](https://www.npmjs.com/package/csv-string#stringifyinput--object-separator--string--string) package might help.\n* Note that only primary key values should be provided in a 'delete' file.\n\n## \u003ca name=\"test\"\u003e\u003c/a\u003eTesting\n\nBefore running these tests, you'll need a test PostgreSQL database available and set a `PG_CONNECTION_STRING` environment variable to point to it, for example:\n\n```PG_CONNECTION_STRING=postgres://postgres:postgres@localhost:5432/my_test_db```\n\n\n```bash\n$ npm test\n```\n\n\n## \u003ca name=\"license\"\u003e\u003c/a\u003eLicense\n[MIT](https://github.com/wmfs/supercopy/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwmfs%2Fsupercopy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwmfs%2Fsupercopy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwmfs%2Fsupercopy/lists"}