{"id":16852546,"url":"https://github.com/2called-chaos/db_sucker","last_synced_at":"2026-04-24T23:34:18.439Z","repository":{"id":62556857,"uuid":"83130382","full_name":"2called-chaos/db_sucker","owner":"2called-chaos","description":"Sucks DBs as sucking DBs sucks!","archived":false,"fork":false,"pushed_at":"2023-10-26T11:09:31.000Z","size":278,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-11-30T22:23:34.612Z","etag":null,"topics":["command-line","development","development-tools","dumps","mysql","mysql2","ruby","ssh","utility"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/2called-chaos.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-25T12:31:20.000Z","updated_at":"2022-08-13T13:13:06.000Z","dependencies_parsed_at":"2025-01-24T16:39:50.382Z","dependency_job_id":null,"html_url":"https://github.com/2called-chaos/db_sucker","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/2called-chaos/db_sucker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/2called-chaos%2Fdb_sucker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/2called-chaos%2Fdb_sucker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/2called-chaos%2Fdb_sucker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/2called-chaos%2Fdb_sucker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/2called-chaos","download_url":"https://codeload.github.com/2called-chaos/db_sucker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/2called-chaos%2Fdb_sucker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32245149,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T13:21:15.438Z","status":"ssl_error","status_checked_at":"2026-04-24T13:21:15.005Z","response_time":64,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line","development","development-tools","dumps","mysql","mysql2","ruby","ssh","utility"],"created_at":"2024-10-13T13:47:40.150Z","updated_at":"2026-04-24T23:34:18.417Z","avatar_url":"https://github.com/2called-chaos.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DbSucker\n\n**DbSucker – Sucks DBs as sucking DBs sucks!**\n\n`db_sucker` is an executable which allows you to \"suck\"/pull remote MySQL (others may follow) databases to your local server.\nYou configure your hosts via an YAML configuration in which you can define multiple variations to add constraints on what to dump (and pull).\n\nThis tool is meant for pulling live data into your development environment. **It is not designed for backups!** but you might get away with it.\n\n[![screenshot](https://i.imgur.com/EAjWrEd.png)](https://www.youtube.com/watch?v=jdhyQzJOIkE)\n[▶ see it in action](https://www.youtube.com/watch?v=jdhyQzJOIkE)\n\n---\n## Alpha product (v3 is a rewrite), use at your own risk, always have a backup!\n---\n\n## Features\n\n* independent parallel dump / download / import cycle for each table\n* verifies file integrity via SHA\n* flashy and colorful curses based interface with keyboard shortcuts\n* more status indications than you would ever want (even more if the remote has `pv` (pipeviewer) \u003e= 1.3.8 installed)\n* limit concurrency of certain type of tasks (e.g. limit downloads, imports, etc.)\n* uses more threads than any application should ever use (seriously it's a nightmare)\n\n\n## Requirements\n\nCurrently `db_sucker` only handles the following data-flow constellation:\n\n  - Remote MySQL -\u003e [SSH] -\u003e local MySQL\n\nOn the local side you will need:\n  - unixoid OS\n  - Ruby (\u003e= 2.0, != 2.3.0 see [Caveats](#caveats---bugs))\n  - mysql2 gem\n  - MySQL client (`mysql` command will be used for importing)\n\nOn the remote side you will need:\n  - unixoid OS\n  - Probably SSH access + sftp subsystem (password and/or keyfile)\n  - any folder with write permissions (for the temporary dumps)\n  - mysqldump executable\n  - MySQL credentials :)\n  - Optional: Install \"pv\" aka pipeviewer with a version \u003e= 1.3.8 for progress displays on remote tasks\n\n\n## Installation\n\nSimple as:\n\n    $ gem install db_sucker\n\nYou will need mysql2 as well (it's not a dependency as we might support other DBMS in the future):\n\n    $ gem install mysql2\n\nAt the moment you are advised to adjust the MaxSessions limit on your remote SSH server if you run into issues, see [Caveats](#caveats---bugs).\n\nYou will also need at least one configuration, see [Configuration](#configuration-for-sucking---yaml-format).\n\n\n## Usage\n\nTo get a list of available options invoke `db_sucker` with the `--help` or `-h` option:\n\n    Usage: db_sucker [options] [identifier [variation]]\n\n    # Application options\n            --new NAME                   Generates new container config in /Users/chaos/.db_sucker\n        -a, --action ACTION              Dispatch given action\n        -m, --mode MODE                  Dispatch action with given mode\n        -n, --no-deffer                  Don't use deferred import for files \u003e 50 MB SQL data size.\n        -l, --list-databases             List databases for given identifier.\n        -t, --list-tables [DATABASE]     List tables for given identifier and database.\n                                         If used with --list-databases the DATABASE parameter is optional.\n        -o, --only table,table2          Only suck given tables. Identifier is required, variation is optional (defaults to default).\n                                         WARNING: ignores ignore_always option\n        -e, --except table,table2        Don't suck given tables. Identifier is required, variation is optional (defaults to default).\n        -c, --consumers NUM=10           Maximal amount of tasks to run simultaneously\n            --stat-tmp                   Show information about the remote temporary directory.\n                                         If no identifier is given check local temp directory instead.\n            --cleanup-tmp                Remove all temporary files from db_sucker in target directory.\n            --simulate                   To use with --cleanup-tmp to not actually remove anything.\n\n    # General options\n        -d, --debug [lvl=1]              Enable debug output\n            --monochrome                 Don't colorize output (does not apply to curses)\n            --no-window                  Disables curses window alltogether (no progress)\n        -h, --help                       Shows this help\n        -v, --version                    Shows version and other info\n        -z                               Do not check for updates on GitHub (with -v/--version)\n\n    The current config directory is /Users/chaos/.db_sucker\n\nTo get a list of available interface options and shortcuts press `?` or type `:help` while the curses interface is running (if you just want to see the help without running a task use `db_sucker -a cloop`).\n\n    Key Bindings (case sensitive):\n\n        ?  shows this help\n        ^  eval prompt (app context, synchronized)\n        L  show latest spooled log entries (no scrolling)\n        P  kill SSH polling (if it stucks)\n        T  create core dump and open in editor\n        q  quit prompt\n        Q  same as ctrl-c\n        :  main prompt\n\n    Main prompt commands:\n\n        :? :h(elp)                      shows this help\n        :q(uit)                         quit prompt\n        :q! :quit!                      same as ctrl-c\n        :kill                           (dirty) interrupts all workers\n        :kill!                          (dirty) essentially SIGKILL (no cleanup)\n        :dump                           create and open coredump\n        :eval       [code]              executes code or opens eval prompt (app context, synchronized)\n        :c(ancel)   \u003ctable_name|--all\u003e  cancels given or all workers\n        :p(ause)    \u003ctable_name|--all\u003e  pauses given or all workers\n        :r(esume)   \u003ctable_name|--all\u003e  resumes given or all workers\n\n## Configuration (for sucking) - YAML format\n\n* Note: The name is just for the filename, how you address it later is defined within the file.\n* Create a new configuration with `db_sucker --new \u003cname\u003e`, the name should optimally consist of `a-z_-`.\n* If `ENV[\"EDITOR\"]` is set, the newly generated config file will be opened with that, i.e. `EDITOR=vim db_sucker --new \u003cname\u003e`.\n* Change the file to your liking and be aware that YAML is indendation sensitive (don't mix spaces with tabs).\n* If you want to DbSucker to ignore a certain configuration, rename it to start with two underscores, e.g. `__foo.yml`.\n* The default destination for configuration files is `~/.db_sucker` (indicated in --help) but can be changed with the `DBS_CFGDIR` enviromental variable.\n\n## Configuration (application) - Ruby format\n\nDbSucker has a lot of settings and other mechanisms which you can tweak and utilize by creating a `~/.db_sucker/config.rb` file. You can change settings, add hooks or define own actions. For more information please take a look at the [documented example config](https://github.com/2called-chaos/db_sucker/blob/master/doc/config_example.rb) and/or [complete list of all settings](https://github.com/2called-chaos/db_sucker/blob/master/lib/db_sucker/application.rb#L60-L134).\n\n\n## Deferred import\n\nTables with an uncompressed filesize of over 50MB will be queued up for import. Files smaller than 50MB will be imported concurrently with other tables. When all those have finished the large ones will import one after another. You can skip this behaviour with the `-n` resp. `--no-deffer` option. The threshold is changeable in your `config.rb`, see [Configuration](#configuration-application---ruby-format).\n\n\n## Importer\n\nCurrently there is only the \"binary\" importer which will use the mysql client binary. A [sequel](https://github.com/jeremyevans/sequel) importer has yet to be ported from v2.\n\n* **void10** Used for development/testing. Sleeps for 10 seconds and then exits.\n* **binary** Default import using `mysql` executable\n  * **+dirty** Same as default but the dump will get wrapped:\n    ```\n      (\n        echo \"SET AUTOCOMMIT=0;\"\n        echo \"SET UNIQUE_CHECKS=0;\"\n        echo \"SET FOREIGN_KEY_CHECKS=0;\"\n        cat dumpfile.sql\n        echo \"SET FOREIGN_KEY_CHECKS=1;\"\n        echo \"SET UNIQUE_CHECKS=1;\"\n        echo \"SET AUTOCOMMIT=1;\"\n        echo \"COMMIT;\"\n      ) | mysql -u... -p... target_database\n    ```\n    **The wrapper will only be used on deferred imports (since it alters global MySQL sever variables)!**\n* **sequel** Not yet implemented\n\n## Caveats  / Bugs\n\n### General\n\n* Ruby 2.3.0 has a bug that might segfault your ruby if some exceptions occur, this is fixed since 2.3.1 and later\n\n### SSH errors / MaxSessions\n\nUnder certain conditions the program might softlock when the remote unexpectedly closes the SSH connection or stops responding to it (bad packet error). The same might happen when the remote denies a new connection (e.g. to many connections/sessions). If you think it stalled, try `:kill` (semi-clean) or `:kill!` (basically SIGKILL). If you did kill it make sure to run the cleanup task to get rid of potentially big dump files.\n\n**DbSucker typically needs 2 sessions + 1 for each download and you should have some spare for canceling remote processes**\n\nIf you get warnings that SSH errors occured (and most likely tasks fail), please do any of the following to prevent the issue:\n\n  * Raise the MaxSession setting on the remote SSHd server if you can (recommended)\n  * Lower the amount of slots for concurrent downloads (see [Configuration](#configuration-application---ruby-format))\n  * Lower the amount of consumers (not recommended, use slots instead)\n\nYou can run basic SSH diagnosis tests with `db_sucker \u003cconfig_identifier\u003e -a sshdiag`.\n\n## Todo\n\n* Migrate sequel importer from v2\n* Add dirty features again (partial dumps, dumps with SQL constraints)\n* Figure out a way for consumers to release waiting tasks to prevent logical softlocks\n* Optional encrypted HTTP(s) gateway for faster download of files (I can't figure out why Ruby SFTP is soooo slow)\n\n\n## Contributing\n\n1. Fork it ( http://github.com/2called-chaos/db_sucker/fork )\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create new Pull Request\n6. Get a psych, if you understand what I did here you deserve a medal!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F2called-chaos%2Fdb_sucker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F2called-chaos%2Fdb_sucker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F2called-chaos%2Fdb_sucker/lists"}