{"id":13720398,"url":"https://github.com/SwiftOtter/Driver","last_synced_at":"2025-05-07T12:31:21.136Z","repository":{"id":37822171,"uuid":"70206985","full_name":"swiftotter/driver","owner":"swiftotter","description":"A database task-runner (especially for transforming a production database to a sandbox).","archived":false,"fork":false,"pushed_at":"2023-04-03T13:16:29.000Z","size":382,"stargazers_count":61,"open_issues_count":8,"forks_count":10,"subscribers_count":15,"default_branch":"4.1-develop","last_synced_at":"2025-04-08T23:13:22.211Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/swiftotter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.MD","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2016-10-07T01:42:54.000Z","updated_at":"2024-12-15T10:01:08.000Z","dependencies_parsed_at":"2024-02-03T20:49:30.485Z","dependency_job_id":null,"html_url":"https://github.com/swiftotter/driver","commit_stats":{"total_commits":108,"total_committers":7,"mean_commits":"15.428571428571429","dds":"0.40740740740740744","last_synced_commit":"bb51bc383609ad064f2d5e4d213753f66078e9a0"},"previous_names":[],"tags_count":43,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swiftotter%2Fdriver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swiftotter%2Fdriver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swiftotter%2Fdriver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swiftotter%2Fdriver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/swiftotter","download_url":"https://codeload.github.com/swiftotter/driver/tar.gz/refs/heads/4.1-develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252876393,"owners_count":21818173,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T01:01:03.335Z","updated_at":"2025-05-07T12:31:18.067Z","avatar_url":"https://github.com/swiftotter.png","language":"PHP","funding_links":[],"categories":["Tools"],"sub_categories":["Free"],"readme":"# Driver\n\nDriver is a database task-runner that helps you to turn production database into a staging/development database sandbox.\n\n## Description\n\nDriver is the easy way to turn a production database into a staging or development-friendly database.\nIt is built with the aim of complete flexibility. Additionally, with the built-in capability\nof working with RDS, Driver can run transformations on database host that is completely separate from\nyour production environment.\n\nDriver resides on your production machine, with your website's codebase (via Composer).\nYou set up a CRON job to run Driver.\n\n* Driver connects to your production database ONCE via `mysqldump`.\n* Driver dumps that to a file on your system.\n* Driver creates an RDS instance and pushes the database dump up to RDS (via https).\n* Driver runs whatever actions you would like (configured globally or per environment).\n* Driver dumps the transformed data, zips it and pushes it up to S3.\n* You can then download transformed dump from S3 to your staging/development machine, also using Driver.\n\nFor a 3-5GB database, this process could take an hour or two.\nThis depends on how many environments you are creating and what type of RDS instance you are using.\nThis causes no downtime to the frontend thanks to the `--single-transaction` flag on `mysqldump`.\nYes, it does take a while to run, but there is little-to-no impact on the frontend.\n\n## Before We Start\n\nAt Swift Otter we mainly work with Magento 2. We created separate repo for Magento 2 Driver transformations.\nIt includes config for anonymization of all database tables of core Magento 2 system. If you are Magento 2 developer,\nwe recommend you to install that repo, instead of this one (Driver would be installed anyway, as it is a dependency\nof that Magento 2 repo). If you're not a Magento 2 developer, we still recommend you to look at that repo to get\na better idea on how things can be configured for your project.\n\nMagento 2 Driver transformations repo can be found [here](https://github.com/swiftotter/Driver-Magento2).\n\n## Getting Started\n\n### Dependencies\n\n- PHP 7.4 or higher\n\n### Prerequisites\n\nYou need to create an Amazon AWS account. We will be using RDS to perform the data manipulations.\nIt is recommended to create a new IAM user with appropriate permissions to access EC2, RDS and S3\n(exact permissions will be coming). This is explained in details [here](docs/aws-setup.md).\n\n### Installation\n\n```bash\ncomposer require swiftotter/driver\n```\n\n### Configuration\n\n#### Overview of Config Structure \n\nDriver configuration goes into a folder named `.driver`.\nThe files that are recognized inside this folder are:\n\n* `anonymize.yaml`\n* `commands.yaml`\n* `config.yaml`\n* `connections.yaml`\n* `engines.yaml`\n* `environments.yaml`\n* `pipelines.yaml`\n* `reduce.yaml`\n* `update_values.yaml`\n\nThe filenames of these files serve no purpose other than a namespace. The delineation of the configuration\nhappens inside each file. For example, in `pipelines.yaml`, there is a `pipelines` node as the root element.\nIn this way, the YAML itself is providing namespaces. This also has the benefit for you of being able to put\nall of your updates in one file (for example, `config.yaml`).\n\nDriver looks in quite a few places for configuration files. As an example, let's say your application is\nstored in `/var/www/`. Your vendor directory is `/var/www/vendor/` and, of course, Driver's home is\n`/var/www/vendor/swiftotter/driver`. As such, Driver will look in the following locations for configuration\nfiles:\n\n* `/var/www/.driver/`\n* `/var/www/vendor/*/*/.driver/`\n\nYou can symlink any file you want here. Keep in mind that these files do contain sensitive information, and\nit is necessary to include a `.htaccess` into that folder: `Deny from all`\n\n#### Connections Configuration\n\nYou have to configure connections information for your project.\nIn the folder that contains your `vendor` folder, create a folder called `.driver`.\nNext, copy the file `vendor/swiftotter/driver/.driver/connections.yaml.dist` to `.driver/connections.yaml`\nand fill it in with your source MySQL information, as well as destination EC2, RDS and S3 data.\n\nExemplary config can be found in `vendor/swiftotter/driver/.driver/connections.yaml.example`.\n\n#### Environments Configuration\n\nEnvironments allow to make modifications to the database, per-site. You can configure different set of actions for every\nenvironment. For example, for local environment you may want to clear out all admin users as well as set unique URLs\nfor the website and possibly other things.\n\n**Note:** You can execute specific commands per environment. These changes do not revert for each environment,\nbut rather they are applied according to their sort order. If one environment uses the data in the `admin_users` table,\nand another environment clears out all data in the `admin_users` table, you set the sort order for the first environment\nlower (ex. `100`) and the sort order for the second environment higher (ex. `200`).\n\nTo configure your environments copy the file `vendor/swiftotter/driver/.driver/environments.yaml.dist`\nto `.driver/environments.yaml` and fill it in with your environments data.\n\nExemplary config can be found in `vendor/swiftotter/driver/.driver/environments.yaml.example`.\n\n#### Anonymization Configuration\n\nTables can be anonymized by creating `anonymize.yaml` file in `.driver/`. The following type of anonymization entities\nare available in order to provide realistic data and types:\n\n* `email`\n* `company`\n* `firstname`\n* `lastname`\n* `phone`\n* `postcode`\n* `street`\n* `city`\n* `ip`\n* `general`\n* `empty`\n\n**Example File**\n```\nanonymize:\n  tables:\n    quote:\n      customer_email:\n        type: email\n      remote_ip:\n        type: ip\n```\n\n#### Post-Anonymization Values Updates Configuration\n\nSometimes in your database you can store in a single field some value that was originally built by concatenating\nfields from different tables, eg. `full_name` or `shipping_address`. Driver allows to run some specific values updates\nafter anonymization of simple fields is finished. Thanks to this feature you can anonymize such complex fields too.\n\nTo create post-anonymization values updates, create `update_values.yaml` file in `.driver/`.\n\n**Example File**\n```\nupdate-values:\n  tables:\n    customer_grid_flat:\n      joins:\n        - table: customer_entity\n          alias: c\n          on: c.entity_id = customer_grid_flat.entity_id\n        - table: customer_address_entity\n          alias: ba\n          on: ba.entity_id = c.default_billing\n        - table: customer_address_entity\n          alias: sa\n          on: sa.entity_id = c.default_shipping\n      values:\n        - field: name\n          value: CONCAT_WS(' ', c.firstname, c.lastname)\n        - field: shipping_full\n          value: CONCAT(sa.street, ', ', sa.city, ', ', IFNULL(sa.region, '-'), ' ', sa.postcode, ', ', sa.country_id)\n        - field: billing_full\n          value: CONCAT(ba.street, ', ', ba.city, ', ', IFNULL(ba.region, '-'), ' ', ba.postcode, ', ', ba.country_id)\n```\n\n## Usage \n\n### Building Database Sandboxes For All Environments\n\nBe prepared for it to take some time (on the order of hours).\n\n```\n./vendor/bin/driver run\n```\n\n### Building Database Sandbox For Single Environment\n\n```bash\n./vendor/bin/driver run build --environment=envname\n```\n\nwhere `envname` is one of the environments defined in `environments.yaml` config file.\n\n### Tagging Database Sandbox\n\nSandboxes built with a tag will include the tag in the file name.\n\n```bash\n./vendor/bin/driver run build --tag=tagname\n```\n\nwhere `tagname` is an issue number (or whatever is desired).\n\n### Download and Import Sandbox Database from S3 to staging/local environment\n\nThe below command run the export/import pipeline which download the database into your project `var/` directory,\ncreate new database in MySQL, and import the downloaded database in it for your environment.\n\n```\n./vendor/bin/driver run import-s3 --environment=envname --tag=tagname\n```\n\n`tag` is optional.\n\n## Contribution\n\nWe welcome you to contribute in Driver development. You can find some handy information for developers\n[here](docs/development.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSwiftOtter%2FDriver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSwiftOtter%2FDriver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSwiftOtter%2FDriver/lists"}