{"id":32805284,"url":"https://github.com/ryanch741/pt-osc-incremental-update-plugin","last_synced_at":"2026-04-14T07:33:57.513Z","repository":{"id":322644631,"uuid":"1090343540","full_name":"ryanch741/pt-osc-incremental-update-plugin","owner":"ryanch741","description":"A Perl plugin for pt-online-schema-change to incrementally execute custom SQL updates on the new table during the row-copying process.","archived":false,"fork":false,"pushed_at":"2025-11-05T14:47:55.000Z","size":9,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-05T16:24:35.916Z","etag":null,"topics":["data-backfill","database-migration","mysql","percona-toolkit","perl","plugin","pt-online-schema-change-"],"latest_commit_sha":null,"homepage":"","language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ryanch741.png","metadata":{"files":{"readme":"README.en.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-05T14:41:57.000Z","updated_at":"2025-11-05T14:57:58.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ryanch741/pt-osc-incremental-update-plugin","commit_stats":null,"previous_names":["ryanch741/pt-osc-incremental-update-plugin"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ryanch741/pt-osc-incremental-update-plugin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanch741%2Fpt-osc-incremental-update-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanch741%2Fpt-osc-incremental-update-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanch741%2Fpt-osc-incremental-update-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanch741%2Fpt-osc-incremental-update-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ryanch741","download_url":"https://codeload.github.com/ryanch741/pt-osc-incremental-update-plugin/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanch741%2Fpt-osc-incremental-update-plugin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31787039,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T02:24:21.117Z","status":"ssl_error","status_checked_at":"2026-04-14T02:24:20.627Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-backfill","database-migration","mysql","percona-toolkit","perl","plugin","pt-online-schema-change-"],"created_at":"2025-11-06T12:00:58.185Z","updated_at":"2026-04-14T07:33:57.497Z","avatar_url":"https://github.com/ryanch741.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pt-osc-incremental-update-plugin\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\nA Perl plugin for Percona Toolkit's `pt-online-schema-change` that allows you to execute custom SQL update statements **incrementally in batches** on the new table during an online schema change process (e.g., while adding an index).\n\nThis is particularly useful for scenarios like data backfilling or data cleansing concurrently with a data migration.\n\n[中文版本](./README.md)\n\n---\n\n## ✨ Features\n\n- **Incremental Updates**: The plugin executes your SQL after each data chunk is copied, ensuring that only newly copied rows are processed, thus avoiding full table scans.\n- **Custom SQL**: You can provide any complex update logic through a temporary SQL file.\n- **Atomic Operation**: Binds the data update logic with the schema change in a single `pt-online-schema-change` command, simplifying the workflow and ensuring data consistency.\n- **Safety Checks**: The plugin validates the existence and format of the SQL file and automatically deletes it after use for security.\n- **Compatibility**: Leverages the standard hook mechanism of `pt-online-schema-change`.\n\n## ⚙️ How It Works\n\nThe plugin hooks into several key execution points of `pt-online-schema-change`:\n\n1.  **`new` (Initialization)**:\n    *   On startup, the plugin reads the SQL statement from `/tmp/pt_osc_update_rows.sql`.\n    *   It validates the SQL format (must contain the placeholder `%new_table% t`).\n    *   The SQL is stored in memory, and the temporary file is immediately deleted.\n\n2.  **`after_create_new_table` (After New Table Creation)**:\n    *   This hook is triggered after `pt-online-schema-change` creates the empty temporary table with the new schema.\n    *   The plugin captures and saves the actual name of this new table.\n\n3.  **`on_copy_rows_after_nibble` (After Row Copy)**:\n    *   `pt-online-schema-change` copies data from the old table to the new one in chunks (nibbles). This hook is fired after each chunk is copied.\n    *   The plugin performs the following actions:\n        1.  Retrieves the table's primary key information (only single-column PKs are supported).\n        2.  Replaces the `%new_table%` placeholder in the SQL template with the actual new table name.\n        3.  Dynamically appends a `WHERE` clause (e.g., `AND t.primary_key \u003e last_pk_value`) to ensure only the most recently copied rows are updated.\n        4.  Executes the constructed SQL statement.\n        5.  Records the maximum primary key value processed so far, to be used in the next incremental update.\n\n## 🚀 How to Use\n\n#### 1. Deploy the Plugin Script\n\nPlace the `plugin-pt-osc-update-rows.pl` script in a directory accessible by `pt-online-schema-change`. If you are using Docker, you can mount the directory as follows:\n\n```bash\n# Example: Mount a local plugin directory to /opt/plugin-perls in the container\ndocker run -it --rm \\\n  -v /path/to/your/plugins:/opt/plugin-perls \\\n  registry.cn-hangzhou.aliyuncs.com/flyhand/perconalab-toolkit:3.7.0 bash```\n\n#### 2. Prepare the Custom SQL File\n\nCreate a temporary SQL file at `/tmp/pt_osc_update_rows.sql`. **You must adhere to the following rules**:\n\n- Use `%new_table%` as the placeholder for the new table name.\n- You must alias the new table as `t`, e.g., `%new_table% t`.\n\n**Example SQL file content:**\n\n```bash\n# Suppose we want to backfill the enterprise_id column in the new table\n# based on data from the 'cart' and 'restaurant' tables.\necho \"UPDATE %new_table% t\nLEFT JOIN cart c ON c.id = t.cart_id\nLEFT JOIN restaurant r ON r.id = c.restaurant_id\nSET t.enterprise_id = r.enterprise_id\nWHERE t.enterprise_id = 0\" \u003e /tmp/pt_osc_update_rows.sql\n```\n\n#### 3. Execute pt-online-schema-change\n\nWhen invoking the `pt-online-schema-change` command, specify the plugin using the `--plugin` argument.\n\n```bash\n/usr/bin/pt-online-schema-change \\\n  --alter \"ADD INDEX idx_enterprise_id(enterprise_id)\" \\\n  --plugin /opt/plugin-perls/plugin-pt-osc-update-rows.pl \\\n  --preserve-triggers \\\n  --chunk-size=1000 \\\n  --execute \\\n  h=127.0.0.1,P=3306,u=root,p=\"PASSWORD\",D=DATABASE_NAME,t=TABLE_NAME\n```\n\n## ⚠️ Important Notes\n\n- **SQL File Path**: The plugin is hardcoded to read from `/tmp/pt_osc_update_rows.sql`. Ensure the path is correct and that you have the necessary read/write permissions.\n- **SQL Alias**: Your update statement must use `t` as the alias for the new table (e.g., `UPDATE %new_table% t ...`).\n- **Primary Key Limitation**: The incremental update logic of this plugin only supports **single-column primary keys**. Composite primary keys are not supported.\n- **Temporary File**: The plugin will **immediately and automatically delete** `/tmp/pt_osc_update_rows.sql` after successfully reading its content.\n\n## 📄 License\n\nThis project is licensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanch741%2Fpt-osc-incremental-update-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fryanch741%2Fpt-osc-incremental-update-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanch741%2Fpt-osc-incremental-update-plugin/lists"}