{"id":33996447,"url":"https://github.com/qoliber/m2-nanobots-dbdumper","last_synced_at":"2025-12-13T08:04:58.257Z","repository":{"id":157362562,"uuid":"633430866","full_name":"qoliber/m2-nanobots-dbdumper","owner":"qoliber","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-19T12:27:09.000Z","size":171,"stargazers_count":26,"open_issues_count":2,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-08-13T12:57:37.734Z","etag":null,"topics":["magento","magento2","magento2-extension","magento2-module","php"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"osl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qoliber.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-27T13:34:43.000Z","updated_at":"2025-05-05T14:56:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"e9f72714-cbdf-409e-9a1e-065b8984da8e","html_url":"https://github.com/qoliber/m2-nanobots-dbdumper","commit_stats":null,"previous_names":["qoliber/m2-nanobots-dbdumper"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/qoliber/m2-nanobots-dbdumper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qoliber%2Fm2-nanobots-dbdumper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qoliber%2Fm2-nanobots-dbdumper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qoliber%2Fm2-nanobots-dbdumper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qoliber%2Fm2-nanobots-dbdumper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qoliber","download_url":"https://codeload.github.com/qoliber/m2-nanobots-dbdumper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qoliber%2Fm2-nanobots-dbdumper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27702409,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-13T02:00:09.769Z","response_time":147,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["magento","magento2","magento2-extension","magento2-module","php"],"created_at":"2025-12-13T08:04:57.518Z","updated_at":"2025-12-13T08:04:58.246Z","avatar_url":"https://github.com/qoliber.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Open Source Love](https://img.shields.io/badge/open-source-lightgrey?style=for-the-badge\u0026logo=github)\n![](https://img.shields.io/badge/Magento-2.3.x/2.4.x-orange?style=for-the-badge\u0026logo=magento)\n![](https://img.shields.io/badge/Maintained-yes-gren?style=for-the-badge\u0026logo=magento)\n![](https://img.shields.io/badge/PHP-7.4.x-blue?style=for-the-badge\u0026logo=php)\n\n\u003ch1 style=\"text-align: center;\"\u003eQoliber DbDumper \u003c/h1\u003e\n\nThis module was designed simply to create advanced database dumps for Magento 2 Open Source Projects.\nIt does work on Adobe Cloud and Adobe Commerce, but you need to download files manually (this\nmight be changed in the future releases of this module)\n\n## Ok, But why?\n\nIt is very simple! Because we can! But in all seriousness, Magento 2 has been on Market\na long time and nobody has created a module that out of the box has the following features:\n\n* allow to specify SQL filters on any table (for example to limit the product catalog, 2 examples are part of the module)\n* customer and user admin accounts data anonymization by default\n* junk data is never added to dumps\n\nSo, that is why this module was developed.\n\n## Installation\n\nInstallation is via `composer`\n```\ncomposer require qoliber/m2-nanobots-dbdumper\n```\n\nAfter installing the packages just run:\n```\nphp bin/magento setup:upgrade\n```\n\n### Requirements:\n* `PHP 7.4` and higher\n* `Magento 2.4.x` and higher\n\n### Tested on:\n* `Magento 2.4.x` OpenSource\n\n# WIKI\n\nThis section will describe how the module is designed. So let me start with:\n\n## Export Modes\n\nThis module allows you to create any database export profiles you wish, however I've called them\n`Export Models`. The main Export model implements `ExportModelInterface` and all Export Models are defined as `VirtualTypes` so you can\nadd any custom Export Model you wish.\n\nIn this first release of the extension, `Export Model` has 4 params:\n* `$exportModeDescription` - visible in CLI if you specify wrong export Model, just a visual note\n* `$applyFilters` - determines if a custom filters on tables should be applied\n* `$tableGroups` - if you specify this list, you can export only selected tables from the database\n* `$exportTypeForAllTables` - determines if data from custom tables should be exported. Yeah... this variable name will probably be changed\n\nAn export model is defined in the `di.xml`, this module has 2 additional Export Models:\n* `custom` that filters product catalog, more on that lateer\n* `cms` - exports only tables from cms table group\n\nYou can find this in the `di.xml` file:\n```xml\n    \u003cvirtualType name=\"customExportMode\" type=\"Qoliber\\DbDumper\\Model\\ExportModel\"\u003e\n        \u003carguments\u003e\n            \u003cargument name=\"exportModeDescription\" xsi:type=\"string\"\u003eCustom dump with filters\u003c/argument\u003e\n            \u003cargument name=\"applyFilters\" xsi:type=\"boolean\"\u003etrue\u003c/argument\u003e\n            \u003cargument name=\"exportTypeForAllTables\" xsi:type=\"number\"\u003e0\u003c/argument\u003e \u003c!-- ask what to do --\u003e\n        \u003c/arguments\u003e\n    \u003c/virtualType\u003e\n\n    \u003cvirtualType name=\"cmsExport\" type=\"Qoliber\\DbDumper\\Model\\ExportModel\"\u003e\n    \u003carguments\u003e\n        \u003cargument name=\"exportModeDescription\" xsi:type=\"string\"\u003eOnly CMS data\u003c/argument\u003e\n        \u003cargument name=\"applyFilters\" xsi:type=\"boolean\"\u003efalse\u003c/argument\u003e\n        \u003cargument name=\"tableGroups\" xsi:type=\"array\"\u003e\n            \u003citem name=\"cms\" xsi:type=\"object\"\u003ecms\u003c/item\u003e\n        \u003c/argument\u003e\n    \u003c/arguments\u003e\n    \u003c/virtualType\u003e\n```\n\nThose export models need to be added to the `DumpInterface` `exportModels` argument in the `di.xml`\n```xml\n\u003ctype name=\"Qoliber\\DbDumper\\Model\\DumpInterface\"\u003e\n    \u003carguments\u003e\n        \u003cargument name=\"exportModels\" xsi:type=\"array\"\u003e\n            \u003citem name=\"cms\" xsi:type=\"object\"\u003ecmsExport\u003c/item\u003e\n            \u003citem name=\"custom\" xsi:type=\"object\"\u003ecustomExportMode\u003c/item\u003e\n            \u003citem name=\"default\" xsi:type=\"object\"\u003edefaultExportMode\u003c/item\u003e\n            \u003citem name=\"full\" xsi:type=\"object\"\u003efullExportMode\u003c/item\u003e\n        \u003c/argument\u003e\n        [...]\n    \u003c/arguments\u003e\n\u003c/type\u003e\n```\n\nYou can define as many export models as you want.\n\n## Table Groups Interface\nThe main part of this module is based on an assumption that Magento 2 tables can be divided into table groups.\nA group of tables is a `VirtualType` that uses a `TableGroup` model, which implements `TableGroupInterface`.\nThe role of a group is simply to specify a set of actions that can be or should be performed on all tables in the table group so we don't have\nset this for every single table defined in the database - although it is possible, it would make the `di.xml`\nfile too large to maintain.\n\nIn this initial version of the module a group can have 3 group modes, defined by `groupExportMode` in `di.xml`:\n* `default` - it means that each table from this group is exported with all it's data\n* `reduntant` - data from tables in this group will not be exported to database dump file\n* `anonymize` - data from these tables needs to be anonymized. By default, it is related to: admin accounts, customers\ncustomer addresses and sales data.\n\nTable Group export mode is overwritten by the `CLI` param set when running the export manually.\n\n### Grouping Tables by `%` character\n\nIt is not require to define all database tables, you can specify a `wildcard` with the `%` character and use it like this:\n```xml\n    \u003citem name=\"%_idx\" xsi:type=\"null\"/\u003e\n    \u003citem name=\"%_log\" xsi:type=\"null\"/\u003e\n    \u003citem name=\"%_cl\" xsi:type=\"null\"/\u003e\n    \u003citem name=\"%_tmp\" xsi:type=\"null\"/\u003e\n    \u003citem name=\"%_replica\" xsi:type=\"null\"/\u003e\n    \u003c!-- [...] --\u003e\n    \u003citem name=\"sequence_creditmemo_%\" xsi:type=\"null\"/\u003e\n```\nThe code will find all matching tables by running proper queries on the database.\n\n## Table Export Interface\n\nThis is the main interface that transforms data from the database into\n* create table / view statements\n* fetches data from the table\n\nand puts it into a database dump file\n\nA default list of `restrictedColumns` is defined in the `di.xml`. If a table needs to be anonymized\n(which is defined by a TableGroup the table belongs to) table is checked if any of the columns match the\nrestricted column list. If a column is found in the restrictedColumn list, value from that table row will be\n\"faked\" or anonymized.\n\n### Example: email anonymization\n\nEmail DataFakers (or anonymizer) has 2 additional arguments:\n```xml\n\u003ctype name=\"Qoliber\\DbDumper\\Model\\DataFakers\\Email\"\u003e\n    \u003carguments\u003e\n        \u003cargument name=\"emailSchema\" xsi:type=\"string\"\u003e\u003c![CDATA[{{firstname}}.{{lastname}}]]\u003e\u003c/argument\u003e\n        \u003cargument name=\"emailDomain\" xsi:type=\"string\"\u003e\u003c![CDATA[qoliber.info]]\u003e\u003c/argument\u003e\n    \u003c/arguments\u003e\n\u003c/type\u003e\n```\nand has 2 methods:\n```php\n    public function decorateData(int $entityId, ?string $value = null): string\n    {\n        return sprintf(\"%s@%s\", $this-\u003e_getEmailFromSchema($entityId), $this-\u003eemailDomain);\n    }\n\n    private function _getEmailFromSchema(int $entityId): string\n    {\n        return str_replace(\n            [\n                '{{firstname}}',\n                '{{lastname}}'\n            ],\n            [\n                $this-\u003etranslitUrl-\u003efilter($this-\u003efirstname-\u003edecorateData($entityId)),\n                $this-\u003etranslitUrl-\u003efilter($this-\u003elastname-\u003edecorateData($entityId))\n            ],\n            $this-\u003eemailSchema\n        );\n    }\n```\nif you check the `Firstname` DataFaker class you will see that it is using `modulo` division.\n```php\n[...]\n    return $this-\u003efirstNames[$entityId % (count($this-\u003efirstNames) - 1)];\n[...]\n```\nIt is on purpose, so each time you download the database, you will get matching, but randomized names or email\nfor the same ID.\nWhen you work with the team and debug some sales / customer data, it is important to identify data not\njust by an ID.\n\n## Working in CLI\n\nTo start DB dump operation you can simply start the following command in CLI:\n```\nphp bin/magento qoliber:dump --mode=ZZZ\n```\nThis will initiate the `dump` command. For example, let's use an export mode that is not defined:\n\n```\nphp bin/magento qoliber:dump --mode=jakub\n```\nyou will get the following result:\n\n![img_1.png](src/img_1.png)\n\nIn the `custom` Export Model, system will ask you what to do with tables that are not defined in a table group:\n\nby typing `0`, `1` or `2` you can\n* force db dump to export all data from tables without table groups\n* skip the data from additional tables\n* quit\n\n### Recommended actions:\n\nIt is recommended to create your own module and add tables to a specified group for every project you manage:\n* `core` defined as a virtualGroup in the `di.xml` to export all data\n* `empty` defined as a virtualGroup in the `di.xml` to only export table structure\n\n#### Example:\n`di.xml` in your custom module\n\n```xml\n    \u003cvirtualType name=\"core\" type=\"Qoliber\\DbDumper\\Model\\TableGroup\"\u003e\n        \u003carguments\u003e\n            \u003cargument name=\"tableList\" xsi:type=\"array\"\u003e\n                \u003citem name=\"my_custom_table_with_required_data\" xsi:type=\"null\" /\u003e\n            \u003c/argument\u003e\n        \u003c/arguments\u003e\n    \u003c/virtualType\u003e\n\n    \u003cvirtualType name=\"empty\" type=\"Qoliber\\DbDumper\\Model\\TableGroup\"\u003e\n        \u003carguments\u003e\n            \u003cargument name=\"tableList\" xsi:type=\"array\"\u003e\n                \u003citem name=\"my_custom_table_with_redundant_data\" xsi:type=\"null\" /\u003e\n            \u003c/argument\u003e\n        \u003c/arguments\u003e\n    \u003c/virtualType\u003e\n```\n\n## Important!\n* Final DB Dump is setting indexers into **on save** mode.\n* `xsi:type=\"null` is being used a lot as array keys are our final `tables`\n* all `_cl` tables are empty and their autoincrement field is set to 1.\n* all `mviews` are set to `idle` state\n* SQL triggers are not dropped in the SQL dump, you can recreate them by setting indexers to **On Schedule** mode\n\n\n### Types of data:\n\nThe main Module assumption is that we have 3 sets of data:\n* data we would like to anonymize or filter in somehow.\n* data that is redundant and is not required in database dumps\n* undefined data, which is all the data in the database we don't want to filter or is redundant\n\nBased on those assumptions multiple interfaces were created:\n* GroupInterface - used to created VirtualTypes with table groups\n* TableExportInterface\n* BlaBlaInterface\n\n## Release Notes \u0026 Change log\n\n### 1.0.0\n- module release, possibly with few bugs :-)\n\n## Plans and next versions of the module:\n- 1.1.0: adding product media images to a zipped file after `DBDump` creation\n- 1.2.0: adding admin interface to control and download db dumps (they will be secured)\n- 1.3.0: extending admin interface and module with DB dump profiles (more on that with the release)\n- 1.4.0: extending module with task feature to be able to customize and hook into db dump process even further\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqoliber%2Fm2-nanobots-dbdumper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqoliber%2Fm2-nanobots-dbdumper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqoliber%2Fm2-nanobots-dbdumper/lists"}