{"id":21689122,"url":"https://github.com/tinram/database-anonymizer","last_synced_at":"2025-03-20T12:37:13.623Z","repository":{"id":83639844,"uuid":"386010294","full_name":"Tinram/Database-Anonymizer","owner":"Tinram","description":"Remove personal information from MySQL/MariaDB databases and backups.","archived":false,"fork":false,"pushed_at":"2021-11-25T22:56:02.000Z","size":39,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-25T12:41:46.006Z","etag":null,"topics":["anonymisation","anonymise","anonymization","anonymize","anonymized-database","mariadb","mysql","obfuscation"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tinram.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-14T16:46:17.000Z","updated_at":"2024-12-22T23:23:13.000Z","dependencies_parsed_at":"2023-10-16T08:38:01.935Z","dependency_job_id":null,"html_url":"https://github.com/Tinram/Database-Anonymizer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FDatabase-Anonymizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FDatabase-Anonymizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FDatabase-Anonymizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FDatabase-Anonymizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tinram","download_url":"https://codeload.github.com/Tinram/Database-Anonymizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244611848,"owners_count":20481272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anonymisation","anonymise","anonymization","anonymize","anonymized-database","mariadb","mysql","obfuscation"],"created_at":"2024-11-25T17:21:18.955Z","updated_at":"2025-03-20T12:37:13.585Z","avatar_url":"https://github.com/Tinram.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"\r\n# Database Anonymizer\r\n\r\n#### Remove Personally Identifiable Information (PII) from MySQL and MariaDB database copies and backups.\r\n\r\n\r\n## Purpose\r\n\r\n### Primary\r\n\r\nRemove identifiable personal data stored in a database copy and/or exported logical backup file.\r\n\r\nPersonally Identifiable Information (PII) identifies an individual such as a full name or passport number, or when a number of identifiers can be combined to identify an individual.\r\n\r\nFor database development work, the PII needs to be removed to meet the requirements of data protection laws such as EU GDPR, UK GDPR etc. From the UK Information Commissioner's Office website: *\"If personal data can be truly anonymised then the anonymised data is not subject to the UK GDPR.\"*\r\n\r\nTo maintain data context, the PII is replaced with pseudo-data.\r\n\r\n### Secondary\r\n\r\nReduce the size of backup files for development usage via database table truncation and clipping.\r\n\r\n#### Processed Table Example\r\n\r\n[1]: https://tinram.github.io/images/database-anonymizer.png\r\n![database-anonymizer][1]\r\n\r\n\u003cbr\u003e\r\n\r\n## Background\r\n\r\nSome MySQL databases are huge and contain sensitive personal data.  \r\nFor development usage, only a small subset of the database may be required.  \r\n*Database Anonymizer* provides one way of creating a database export that is much smaller than the original, and with suitable configuration, contains no PII.\r\n\r\n\r\n## Personal Data Columns\r\n\r\nData types recognised:\r\n\r\n+ `CHAR, VARCHAR, TEXT`\r\n+ `INT types`\r\n+ `DECIMAL, FLOAT`\r\n+ `DATE, DATETIME, TIMESTAMP`\r\n\r\nMost personal data columns are likely to be `CHAR` and `VARCHAR` types.\r\n\r\nThe users table in the example schema *anon_test.sql* provides a reference.\r\n\r\n`DECIMAL` AND `FLOAT` processing is basic (integer overwrites), but avoids using expensive function calls to generate the fractional part, and the resulting *.00* fraction shows the column data has been changed.\r\n\r\n`ENUM`s are not supported.\r\n\r\n\r\n## Usage\r\n\r\nSet-up in *runner.php*\r\n\r\n*Database Anonymizer* is reasonably modular.\r\n\r\nIf logging is not required, comment out the line:\r\n\r\n```php\r\n$log = Logger::setup($config);\r\n```\r\n\r\nand nothing will be logged in *src/Anonymizer/log/anon.log*\r\n\r\nIf tables do not require clipping (partial truncation) comment out the line:\r\n\r\n```php\r\n$c = new Clip($db, $numRows);\r\n```\r\n\r\n\r\n## Example Usage\r\n\r\n```bash\r\n$ mysql -h localhost -u root -p \u003c src/Anonymizer/sql/anon_test.sql\r\n```\r\n\r\nChange any database credentials as needed in *src/Anonymizer/Configuration.php*\r\n\r\nFill the database with test data (use a tool such as Spawner, or a script such as [Database_Filler](https://github.com/Tinram/Database-Filler)) and create 150 rows of junk data for the next steps.\r\n\r\n```sql\r\nmysql\u003e USE anon_test;\r\nmysql\u003e SELECT email FROM users;           # displays the default email column data\r\n```\r\n\r\nConfigure *runner.php* or use the default parameters as they are, which specify:\r\n\r\n```php\r\n$tablesToTruncate = ['misc'];             # table called 'misc' to be truncated (wiped, preserving table schema)\r\n$numRows = 100;                           # number of rows to remain after clipping (reduced table rows, i.e. partial truncation)\r\n$c = new Clip($db, $numRows);             # all database tables to be clipped\r\n$anonymize = [ 'users' =\u003e ['email'] ];    # users.email column data will be anonymized/obliterated\r\n```\r\n\r\n```bash\r\n$ php runner.php                          # execute the runner script\r\n```\r\n\r\n```sql\r\nmysql\u003e SELECT email FROM users;           # returns 100 rows of pseudo email addresses\r\n\r\nmysql\u003e SELECT COUNT(*) FROM misc;         # returns 0, as table has been truncated\r\n\r\nmysql\u003e SELECT COUNT(*) FROM posts;        # returns 100, as table has been clipped\r\n```\r\n\r\n```bash\r\n$ cat src/Anonymizer/log/anon.log         # shows the action log\r\n```\r\n\r\n    \u003ctimestamp\u003e | Truncated table misc.\r\n    \u003ctimestamp\u003e | Clipped table misc.\r\n    \u003ctimestamp\u003e | Clipped table posts.\r\n    \u003ctimestamp\u003e | Clipped table users.\r\n    \u003ctimestamp\u003e | Filtered table users.\r\n\r\nCreate a database export of reduced data and anonymized email addresses:\r\n\r\n```bash\r\n$ mysqldump -h localhost -u root -p --single-transaction anon_test | gzip -9 \u003e anon_test_reduced.sql.gz\r\n```\r\n\r\n\r\n## Other\r\n\r\nAlthough probably obvious, only run this script on a database copy, not the master.\r\n\r\nTested on MySQL 5.7 and 8.0, and MariaDB 10.1 and 10.4\r\n\r\n\r\n## License\r\n\r\nDatabase Anonymizer is released under the [GPL v.3](https://www.gnu.org/licenses/gpl-3.0.html).\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinram%2Fdatabase-anonymizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftinram%2Fdatabase-anonymizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinram%2Fdatabase-anonymizer/lists"}