{"id":31807859,"url":"https://github.com/gwinans/knex-ptosc-plugin","last_synced_at":"2026-04-28T21:35:15.928Z","repository":{"id":309432850,"uuid":"1036178979","full_name":"gwinans/knex-ptosc-plugin","owner":"gwinans","description":"Knex plugin to use pt-osc instead of the internal runner for online alters. How does this not exist already?","archived":false,"fork":false,"pushed_at":"2026-02-18T02:15:26.000Z","size":465,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-18T07:20:51.302Z","etag":null,"topics":["database","knex","mariadb","migrations","mysql","pt-online-schema-change","pt-osc"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gwinans.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-08-11T17:12:51.000Z","updated_at":"2026-02-18T02:15:29.000Z","dependencies_parsed_at":"2025-08-11T22:23:24.952Z","dependency_job_id":"3f62d47f-74ca-45e9-a92d-2c48974c59b7","html_url":"https://github.com/gwinans/knex-ptosc-plugin","commit_stats":null,"previous_names":["gwinans/knex-ptosc-plugin"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gwinans/knex-ptosc-plugin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gwinans%2Fknex-ptosc-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gwinans%2Fknex-ptosc-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gwinans%2Fknex-ptosc-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gwinans%2Fknex-ptosc-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gwinans","download_url":"https://codeload.github.com/gwinans/knex-ptosc-plugin/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gwinans%2Fknex-ptosc-plugin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32400868,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-28T19:38:08.556Z","status":"ssl_error","status_checked_at":"2026-04-28T19:37:55.688Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","knex","mariadb","migrations","mysql","pt-online-schema-change","pt-osc"],"created_at":"2025-10-11T04:38:22.919Z","updated_at":"2026-04-28T21:35:15.923Z","avatar_url":"https://github.com/gwinans.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# knex-ptosc-plugin\n\n## AI Disclosure\n\nAn LLM (GPT-5) was used heavily in the creation of this plugin.\n\n## WARNING\n\n**This code is a very early swing at extending Knex to use\n`pt-online-schema-change` for DB migrations. Use in production at your own\nrisk.**\n\n## Introduction\n\nA [Knex](https://knexjs.org/) helper for running `ALTER TABLE` operations online\nusing\n[Percona Toolkit's `pt-online-schema-change`](https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html)\n(pt-osc).\n\nThis plugin offers an alternative to the normal schema builder, routing the\n`.alterTable()` builder through **pt-online-schema-change** so changes can be\nexecuted with minimal locking and downtime. It also exposes\n`alterTableWithPtoscRaw` for executing raw `ALTER TABLE` statements. For\noperations that can be run INSTANT in the engine, `pt-osc` will not be invoked;\ninstead knex will handle it.\n\n**Github Repo:** https://github.com/gwinans/knex-ptosc-plugin\n\n**Test App:** https://github.com/gwinans/kpp-test-app\n\nPlease, come contribute! Star the project!\n\n---\n\n## Features\n\n- **Safe execution**: No shell concatenation, no command injection — arguments\n  are passed directly to the pt-osc binary.\n- **Password safety**: The database password is passed via `MYSQL_PWD`\n  environment variable (never on the command line or in logs).\n- **Atomic migration lock**: Uses Knex’s migrations lock row\n  (`knex_migrations_lock` by default) to prevent concurrent schema changes.\n  Table names can be customized with `migrationsTable` and\n  `migrationsLockTable`. Both tables must exist before running migrations;\n  otherwise an error is thrown.\n- **Dry-run first**: Always runs a pt-osc `--dry-run` before executing.\n- **Full ALTER support**: Works with Knex’s `.alterTable()` builder or raw\n  `ALTER TABLE` strings via `alterTableWithPtoscRaw`.\n- **Respects Knex bindings**: Correctly interpolates values from `.toSQL()`\n  output.\n- **Instant alters when possible**: Attempts native\n  `ALTER TABLE ... ALGORITHM=INSTANT` and falls back to pt-osc when unsupported or\n  when MySQL returns error 4092 (\"Maximum row versions\").\n- **Native index operations**: `ADD INDEX` and `DROP INDEX` statements run\n  directly via Knex without pt-osc.\n\nServers running MySQL 5.6 or 5.7 skip the instant-alter attempt and always use\npt-online-schema-change. This will not be changed - MySQL 5.6 and 5.7 are\nlong-past End-of-Life.\n\n## Requirements\n\n- Node.js 16+\n- Knex configured for MySQL or MariaDB\n- [Percona Toolkit](https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html)\n  installed and `pt-online-schema-change` available in `$PATH`\n- MySQL user with appropriate privileges, as required by Knex.\n\n---\n\n## Installation\n\n```sh\nnpm install knex-ptosc-plugin\n```\n\n---\n\n## Options\n\n| Option | Type | Default | Description |\n| --- | --- | --- | --- |\n| `password` | `string` | from Knex connection | Override DB password; passed via `MYSQL_PWD` env |\n| `maxLoad` | `number` | `undefined` | Passed to `--max-load` (e.g. `Threads_connected=150`) |\n| `maxLoadMetric` | `string` | `'Threads_running'` | Metric name used in `--max-load` (e.g. `Threads_connected`) |\n| `criticalLoad` | `number` | `undefined` | Passed to `--critical-load` (e.g. `Threads_running=50`) |\n| `criticalLoadMetric` | `string` | `'Threads_running'` | Metric name used in `--critical-load` (e.g. `Threads_running`) |\n| `alterForeignKeysMethod` | `'auto' \\| 'rebuild_constraints' \\| 'drop_swap' \\| 'none'` | `'auto'` | Passed to `--alter-foreign-keys-method` |\n| `ptoscPath` | `string` | `'pt-online-schema-change'` | Path to pt-osc binary |\n| `forcePtosc` | `boolean` | `false` | Skip the instant-alter attempt and always run pt-osc |\n| `ptoscMinRows` | `number` | `0` | Minimum row count required to use pt-osc; below this, ALTER runs natively |\n| `analyzeBeforeSwap` | `boolean` | `true` | `--analyze-before-swap` or `--noanalyze-before-swap` |\n| `checkAlter` | `boolean` | `true` | `--check-alter` or `--nocheck-alter` |\n| `checkForeignKeys` | `boolean` | `true` | `--check-foreign-keys` or `--nocheck-foreign-keys` |\n| `checkInterval` | `number` | `undefined` | Passed to `--check-interval` |\n| `checkPlan` | `boolean` | `true` | `--check-plan` or `--nocheck-plan` |\n| `checkReplicationFilters` | `boolean` | `true` | `--check-replication-filters` or `--nocheck-replication-filters` |\n| `checkReplicaLag` | `boolean` | `false` | Adds `--check-replica-lag` |\n| `chunkIndex` | `string` | `undefined` | Passed to `--chunk-index` |\n| `chunkIndexColumns` | `number` | `undefined` | Passed to `--chunk-index-columns` |\n| `chunkSize` | `number` | `1000` | Passed to `--chunk-size` |\n| `chunkSizeLimit` | `number` | `4.0` | Passed to `--chunk-size-limit` |\n| `chunkTime` | `number` | `0.5` | Passed to `--chunk-time` |\n| `dropNewTable` | `boolean` | `true` | `--drop-new-table` or `--nodrop-new-table` |\n| `dropOldTable` | `boolean` | `true` | `--drop-old-table` or `--nodrop-old-table` |\n| `dropTriggers` | `boolean` | `true` | `--drop-triggers` or `--nodrop-triggers` |\n| `checkUniqueKeyChange` | `boolean` | `true` | `--check-unique-key-change` or `--nocheck-unique-key-change` |\n| `maxLag` | `number` | `25` | Passed to `--max-lag` |\n| `maxBuffer` | `number` | `10485760` | Combined stdout/stderr capture limit in bytes |\n| `logger` | `{ log: Function, error: Function }` | `console` | Override default logging methods |\n| `onProgress` | `(pct: number, eta?: string) =\u003e void` | `undefined` | Callback for progress percentage and optional ETA parsed from output; logs include pt-osc ETA when available |\n| `statistics` | `boolean` | `false` | Adds `--statistics`; log and collect internal pt-osc counters |\n| `onStatistics` | `(stats: Record\u003cstring, number\u003e) =\u003e void` | `undefined` | Invoked with parsed statistics object when `statistics` is true |\n| `migrationsTable` | `string` | `'knex_migrations'` | Overrides migrations table name used for lock checks |\n| `migrationsLockTable` | `string` | `'knex_migrations_lock'` | Overrides migrations lock table name used when acquiring lock |\n| `timeoutMs` | `number` | `30000` | Timeout in ms when acquiring migration lock |\n| `intervalMs` | `number` | `500` | Delay between lock retries in ms |\n\nAll options are validated up front. Invalid booleans, callbacks, logger methods,\nor empty string values now throw a `TypeError` before migrations start.\n\n### Statistics example\n\nWhen `statistics: true`, pt-online-schema-change prints internal counters at the\nend of the run. These are parsed into an object, logged via the provided logger,\nand returned (or sent to `onStatistics`). Example output:\n\n```\n# Event          Count\n# =====          =====\n# chunk-size     1000\n# copy_rows      12345\n```\n\n## Debugging\n\nThe full `pt-online-schema-change` command is logged before execution regardless\nof `DEBUG`. Set `DEBUG=knex-ptosc-plugin` to enable verbose, line-by-line output.\n\n```sh\nDEBUG=knex-ptosc-plugin knex migrate:latest\n```\n\nWithout `DEBUG`, only high-level progress percentages are logged. If\npt-online-schema-change exits with an error, its full stdout and stderr are\nlogged regardless of `DEBUG`, including any trailing lines that lack a newline.\n\n## Testing\n\nRun `npm test` to execute linting and unit tests. Run `npm run test:coverage`\nto include coverage reporting. Continuous integration also clones the\n[kpp-test-app](https://github.com/gwinans/kpp-test-app) and runs its migrations\nagainst a MySQL instance (`root`/`test`, database `ptosc`) to verify end-to-end\nbehavior.\n\n---\n\n## Safety \u0026 Security\n\n- **No shell**: Commands are executed via\n  [`child_process.spawn`](https://nodejs.org/api/child_process.html#child_processspawncommand-args-options)\n  with an **args array**, so backticks, semicolons, or other shell\n  metacharacters in table names or SQL will not be interpreted by a shell.\n- **Password is hidden**: Never appears in process list, logs, or command\n  history.\n- **Atomic locks**: Lock acquisition uses `UPDATE ... WHERE is_locked=0` to\n  avoid stealing locks from another process.\n- **Dry-run first**: Always verifies the migration before execution.\n\n---\n\n## Example Migration\nThe `alterTableWithPtosc` helper accepts a table name and a schema builder\ncallback, ensuring all changes use Knex's builder API.\n### CommonJS\n\n```js\nconst { alterTableWithPtosc } = require('knex-ptosc-plugin');\n\nexports.up = function (knex) {\n  return alterTableWithPtosc(\n    knex,\n    'widgets',\n    (table) =\u003e {\n      table.bigInteger('qty').alter();\n    },\n    { chunkSize: 500 }\n  );\n};\n\nexports.down = function (knex) {\n  return alterTableWithPtosc(\n    knex,\n    'widgets',\n    (table) =\u003e {\n      table.integer('qty').alter();\n    },\n    { chunkSize: 500 }\n  );\n};\n```\n\n### ESM\n\n```js\nimport { alterTableWithPtosc } from 'knex-ptosc-plugin';\n\nexport function up(knex) {\n  return alterTableWithPtosc(\n    knex,\n    'widgets',\n    (table) =\u003e {\n      table.bigInteger('qty').alter();\n    },\n    { chunkSize: 500 }\n  );\n}\n\nexport function down(knex) {\n  return alterTableWithPtosc(\n    knex,\n    'widgets',\n    (table) =\u003e {\n      table.integer('qty').alter();\n    },\n    { chunkSize: 500 }\n  );\n}\n```\n\n### Raw SQL\n\n```js\nimport { alterTableWithPtoscRaw } from 'knex-ptosc-plugin';\n\nexport function up(knex) {\n  return alterTableWithPtoscRaw(\n    knex,\n    'ALTER TABLE widgets ALTER COLUMN qty TYPE BIGINT',\n    { statistics: true }\n  );\n}\n```\n\n## Advanced Usage\n\n### Handling Foreign Keys\n\n`pt-online-schema-change` may refuse to run when tables have foreign key constraints. Use `alterForeignKeysMethod: 'drop_swap'` or `'rebuild_constraints'` to control how pt-osc manages them:\n\n```js\nreturn alterTableWithPtosc(\n  knex,\n  'widgets',\n  (table) =\u003e {\n    table.dropColumn('category_id');\n  },\n  { alterForeignKeysMethod: 'drop_swap' } // or 'rebuild_constraints'\n);\n```\n\n### Replica Lag Safeguard\n\nEnable `checkReplicaLag: true` to have pt-osc abort if replicas fall behind, useful in production environments with read replicas:\n\n```js\nreturn alterTableWithPtosc(\n  knex,\n  'widgets',\n  (table) =\u003e {\n    table.string('status').defaultTo('pending');\n  },\n  { checkReplicaLag: true }\n);\n```\n\n### Capture Statistics\n\nCollect pt-osc statistics by setting `statistics: true` and handling them in the `onStatistics` callback:\n\n```js\nreturn alterTableWithPtosc(\n  knex,\n  'widgets',\n  (table) =\u003e {\n    table.bigInteger('qty').alter();\n  },\n  {\n    statistics: true,\n    onStatistics: (stats) =\u003e {\n      console.log('pt-osc stats', stats);\n    },\n  }\n);\n```\n\n---\n\n## Troubleshooting\n\n- **`pt-online-schema-change: command not found`**\\\n  The plugin runs `which pt-online-schema-change` during initialization and will\n  throw if the binary cannot be found. Make sure Percona Toolkit is installed\n  and in your PATH, or pass `ptoscPath` in options.\n\n- **Permission errors**\\\n  Verify you followed the installation instructions for Knex.\n\n- **Foreign key issues**\\\n  Use `alterForeignKeysMethod: 'rebuild_constraints'` or `'drop_swap'` if pt-osc\n  refuses to run due to FK constraints. Be careful with self-referencing FKs.\n  Even ptosc can't handle them correctly.\n\n---\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n\n© 2025 Geoff Winans\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgwinans%2Fknex-ptosc-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgwinans%2Fknex-ptosc-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgwinans%2Fknex-ptosc-plugin/lists"}