{"id":23086686,"url":"https://github.com/spokeywheeler/batcher","last_synced_at":"2025-08-16T05:32:00.657Z","repository":{"id":40001160,"uuid":"312824623","full_name":"SpokeyWheeler/batcher","owner":"SpokeyWheeler","description":"A command line utility to perform concurrent mass updates or deletes against an SQL database","archived":false,"fork":false,"pushed_at":"2025-08-06T21:40:00.000Z","size":6040,"stargazers_count":4,"open_issues_count":6,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-06T23:33:05.591Z","etag":null,"topics":["cockroach","cockroachdb","go","golang","mariadb","mysql","postgres","postgresql","sql"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SpokeyWheeler.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-11-14T13:40:35.000Z","updated_at":"2025-06-20T09:54:28.000Z","dependencies_parsed_at":"2023-02-18T17:15:54.894Z","dependency_job_id":"fd88a071-f86d-44a2-8464-4bc6161b7125","html_url":"https://github.com/SpokeyWheeler/batcher","commit_stats":null,"previous_names":[],"tags_count":99,"template":false,"template_full_name":null,"purl":"pkg:github/SpokeyWheeler/batcher","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpokeyWheeler%2Fbatcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpokeyWheeler%2Fbatcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpokeyWheeler%2Fbatcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpokeyWheeler%2Fbatcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SpokeyWheeler","download_url":"https://codeload.github.com/SpokeyWheeler/batcher/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpokeyWheeler%2Fbatcher/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270670770,"owners_count":24625607,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cockroach","cockroachdb","go","golang","mariadb","mysql","postgres","postgresql","sql"],"created_at":"2024-12-16T19:29:34.993Z","updated_at":"2025-08-16T05:32:00.394Z","avatar_url":"https://github.com/SpokeyWheeler.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://spokeywheeler.semaphoreci.com/badges/batcher/branches/main.svg?style=shields)](https://spokeywheeler.semaphoreci.com/projects/batcher)  [![Go Report Card](https://goreportcard.com/badge/github.com/SpokeyWheeler/batcher)](https://goreportcard.com/report/github.com/SpokeyWheeler/batcher)  [![PkgGoDev](https://pkg.go.dev/badge/github.com/SpokeyWheeler/batcher)](https://pkg.go.dev/github.com/SpokeyWheeler/batcher)  [![Total alerts](https://img.shields.io/lgtm/alerts/g/SpokeyWheeler/batcher.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/SpokeyWheeler/batcher/alerts/)  [![Codacy Badge](https://app.codacy.com/project/badge/Grade/132d19460c42416bb371f98bb0c94fc6)](https://www.codacy.com/gh/SpokeyWheeler/batcher/dashboard?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=SpokeyWheeler/batcher\u0026amp;utm_campaign=Badge_Grade)  [![code-inspector Badge](https://www.code-inspector.com/project/17296/score/svg)](https://www.code-inspector.com/project/17296/score/svg)  [![SourceLevel](https://app.sourcelevel.io/github/SpokeyWheeler/-/batcher.svg)](https://app.sourcelevel.io/github/SpokeyWheeler/-/batcher)  [![GuardRails badge](https://api.guardrails.io/v2/badges/SpokeyWheeler/batcher.svg?token=d09c361974cb1acab7d58f925c6a7dd6f9fc6c05dfd43904043a06f382cdc4d7\u0026provider=github)](https://dashboard.guardrails.io/gh/SpokeyWheeler/52652)  [![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](https://lbesson.mit-license.org/)\n\n# batcher\n\nA utility to perform large updates or deletes in batches to improve performance. `TRUNCATE TABLE` is obviously a faster way to purge an entire table, but in many cases, you have an enormous table, of which you need to remove or update a chunk. Copying the rows you want to keep (for a mass delete) can have all sorts of referential constraint issues.\n\nAfter testing many different approaches, I've created this, which generates singleton updates or deletes of the rows in question. If you want to, you can output the generated SQL to a flat file and process it in some other way. If you use the `-execute` option, batcher will use Go's internal concurrency to perform your mass update without having to worry about long transactions or the excruciating slowness of some databases when doing set operations.\n\nNo names, no packdrill.\n\n## Inspired by\n\n[pg-batch](https://github.com/gabfl/pg-batch) - Thank you!\n\n## Supported Databases\n\n| Database | Version | Supported | CI Test Status | Notes |\n| -------- | ------- | --------- | -------------- | ----- |\n| Cockroach | 20.1.3+ | Yes | 20.2.3  | Versions 19.0+ should work |\n| Informix | 12.10+ | No | No | Next on the list |\n| MariaDB | 10.5+ | Yes | 10.5 | |\n| MySQL | 8.0+ | Yes | 8.0.19 | Earlier (5.x) versions don't work |\n| Oracle | 12+ | No | No | After Informix |\n| PostgreSQL | 13.1+ | Yes | No | CI is an ongoing project |\n| SQLServer | 2019 | No | No | Linux will be first, then Windows (maybe!) |\n\n## Installation\n\nBinaries for Mac, Linux and Windows, as well as the source code in zip and tar.gz  can be found [here](https://github.com/SpokeyWheeler/batcher/releases/latest).\n\nIf you want to build the source, I used Go 1.15.6.\n\nOn Mac, you can also:\n```bash\nbrew tap SpokeyWheeler/tap\nbrew install batcher\n```\n\nIf you're into Docker, you can:\n```bash\ndocker run -it spokey/batcher:latest\n```\n\n## Usage\n\n```bash\n$ ./batcher\n'update', 'delete', 'version' or 'help' subcommand is required\nflags:\n  -concurrency int\n    \tconcurrency (default 20)\n  -database string\n    \tdatabase name\n  -dbtype string\n    \tdatabase type, e.g. postgres, informix, oracle, mysql (default \"postgres\")\n  -execute\n    \texecute the operation ('dry-run' only by default)\n  -host string\n    \thost name or IP (default \"localhost\")\n  -opts string\n    \tJDBC URL options (e.g. sslmode=disable) (default \"sslmode=require\")\n  -password string\n    \tpassword\n  -portnum string\n    \tport number (default \"26257\")\n  -set string\n    \te.g. 'column_name=value, column_name=value ...' (ignored if provided with delete subcommand)\n  -table string\n    \ttable name\n  -user string\n    \tuser name\n  -verbose\n    \tprovide detailed output (will output all statements to the screen)\n  -where string\n    \te.g. 'column=value AND column IS NOT NULL ...'\n```\n\n## CAUTION\n\nThis can seriously mess up your day if you get it wrong. Please dry run first to make sure the statement that will run is the one you want!\n\n## A word about CI\n\nIt turns out that SaaS CI across multiple databases is very, very hard! I've tried with CircleCI, GitHub Actions and Travis-CI, and the best I was able to manage was three out of the four I originally aimed for. (Only two are currently supported because I tried Travis last, I could only get two to work and now can't be bothered to go back!)\n\nMust give massive props to CircleCI support - ridiculously prompt and really, really good.\n\nI haven't given up on it, but I'm going to work on that separately because I'm sick of cluttering up this project with hundreds of meaningless CI-related commit messages and version bumps.\n\nAlso, from a CI perspective, Cockroach's \"download a single binary and put it in your path\" approach has been a delight compared to watching hours of APT output before you can test your change. It's not completely PostgreSQL compatible, but if you're considering a new project with lightweight requirements, it could be an interesting option. PostgreSQL itself isn't too bad, but it's still more work. MariaDB is not a drop-in replacement for MySQL, IMHO, because the process followed after an official APT install is different. Both MariaDB and MySQL need to sort out their character set and collation sequence issues, nobody has time for that shit.\n\n~My next stop will be to try Semaphore CI.~\nSemaphore CI is amazing: stunningly fast and I managed to get everything working!\n\n## Coming soon\n\n  - Commercial databases!\n  - Examples\n  - Trickle truncation?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspokeywheeler%2Fbatcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspokeywheeler%2Fbatcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspokeywheeler%2Fbatcher/lists"}