{"id":15713662,"url":"https://github.com/josacar/triki","last_synced_at":"2025-08-23T13:39:13.627Z","repository":{"id":48381646,"uuid":"267150797","full_name":"josacar/triki","owner":"josacar","description":"Mysql, PostgreSQL and SQL dump obfuscator aka anonimizer","archived":false,"fork":false,"pushed_at":"2024-08-18T17:12:02.000Z","size":430,"stargazers_count":41,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-14T06:40:18.706Z","etag":null,"topics":["crystal","crystal-lang","crystal-language","crystal-shard","database","database-dump","hacktoberfest","mysql","mysqldump","obfuscator","postgresql","shards","sql","sqlserver"],"latest_commit_sha":null,"homepage":"","language":"Crystal","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/josacar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-26T20:59:08.000Z","updated_at":"2025-03-24T00:48:22.000Z","dependencies_parsed_at":"2023-01-23T14:30:28.266Z","dependency_job_id":"8768aa87-f15e-4d9b-a5d6-13cb7ecce41c","html_url":"https://github.com/josacar/triki","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/josacar/triki","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/josacar%2Ftriki","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/josacar%2Ftriki/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/josacar%2Ftriki/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/josacar%2Ftriki/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/josacar","download_url":"https://codeload.github.com/josacar/triki/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/josacar%2Ftriki/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271749189,"owners_count":24814142,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crystal","crystal-lang","crystal-language","crystal-shard","database","database-dump","hacktoberfest","mysql","mysqldump","obfuscator","postgresql","shards","sql","sqlserver"],"created_at":"2024-10-03T21:32:47.170Z","updated_at":"2025-08-23T13:39:13.606Z","avatar_url":"https://github.com/josacar.png","language":"Crystal","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Triki\n\n[![Build Status](https://github.com/josacar/triki/workflows/Crystal%20CI/badge.svg)](https://github.com/josacar/triki/actions)\n\nYou want to develop against real production data, but you don't want to violate your users' privacy.  Enter Triki: standalone Crystal code for the selective rewriting of SQL dumps in order to protect user privacy.\n\n# Documentation\n\n[Here](https://josacar.github.io/triki/) you can find the latest generated API documentation about this library.\n\n# Supported databases and versions\n\n## Servers\n\n- MySQL\n- Postgres\n- SQL Server\n\n## Clients\n\n- `pg_dump` ( Postgresql ) up to 15.x\n- `mysqldump` ( MySQL ) up to 8.2\n- `mysqldump` ( MariaDB ) aka `mariadb-dump` ( since v0.3.0 ) up to 10.11\n\n**Note**: Clients may break current SQL dump parsing as by now there is no proper integration testing in CI with all combinations of servers and clients versions, above versions have been partially manually tested.\n\n# Install\n\nAdd this in your `shard.yml`\n\n```\ndependencies:\n  triki:\n    github: josacar/triki\n```\n\nAnd then run `shards install`\n\n# Example Usage\n\nMake an obfuscator.cr script:\n\n```crystal\nrequire \"triki\"\n\nobfuscator = Triki.new({\n  \"people\" =\u003e {\n    \"email\"                     =\u003e { :type =\u003e :email, :skip_regexes =\u003e [/^[\\w\\.\\_]+@my_company\\.com$/i] },\n    \"ethnicity\"                 =\u003e :keep,\n    \"crypted_password\"          =\u003e { :type =\u003e :fixed, :string =\u003e \"SOME_FIXED_PASSWORD_FOR_EASE_OF_DEBUGGING\" },\n    \"bank_account\"              =\u003e { :type =\u003e :fixed, :string =\u003e  -\u003e(row : Triki::RowAsHash) { \"#{row[\"bank_account\"].to_s[0..4]}#{\"*\" * (row[\"email\"].to_s.size - 5)}\".as(Triki::RowContent) } },\n    \"salt\"                      =\u003e { :type =\u003e :fixed, :string =\u003e \"SOME_THING\" },\n    \"remember_token\"            =\u003e :null,\n    \"remember_token_expires_at\" =\u003e :null,\n    \"age\"                       =\u003e { :type =\u003e :null, :unless =\u003e -\u003e(row : Triki::RowAsHash) { row[\"email\"] == \"hello@example.com\" } },\n    \"photo_file_name\"           =\u003e :null,\n    \"photo_content_type\"        =\u003e :null,\n    \"photo_file_size\"           =\u003e :null,\n    \"photo_updated_at\"          =\u003e :null,\n    \"postal_code\"               =\u003e { :type =\u003e :fixed, :string =\u003e \"94109\", :unless =\u003e -\u003e(person : Triki::RowAsHash) { person[\"postal_code\"] == \"12345\"} },\n    \"name\"                      =\u003e :name,\n    \"full_address\"              =\u003e :address,\n    \"bio\"                       =\u003e { :type =\u003e :lorem, :number =\u003e 4 },\n    \"relationship_status\"       =\u003e { :type =\u003e :fixed, :one_of =\u003e [\"Single\", \"Divorced\", \"Married\", \"Engaged\", \"In a Relationship\"] },\n    \"has_children\"              =\u003e { :type =\u003e :integer, :between =\u003e 0..1 },\n  },\n  \"invites\"                     =\u003e :truncate,\n  \"invite_requests\"             =\u003e :truncate,\n  \"tags\"                        =\u003e :keep,\n  \"relationships\" =\u003e {\n    \"account_id\"                =\u003e :keep,\n    \"code\"                      =\u003e { :type =\u003e :string, :length =\u003e 8, :chars =\u003e Triki::USERNAME_CHARS }\n  }\n})\nobfuscator.fail_on_unspecified_columns = true # if you want it to require every column in the table to be in the above definition\nobfuscator.globally_kept_columns = %w[id created_at updated_at] # if you set fail_on_unspecified_columns, you may want this as well\n\nobfuscator.obfuscate(STDIN, STDOUT)\n```\n\nAnd to get an obfuscated dump:\n\n```\nmysqldump -c --add-drop-table --hex-blob -u user -ppassword database | obfuscator \u003e obfuscated_dump.sql\n```\n\nNote that the -c option on mysqldump is required to use triki.  Additionally, the default behavior of mysqldump\nis to output special characters. This may cause trouble, so you can request hex-encoded blob content with `--hex-blob`.\nIf you get MySQL errors due to very long lines, try some combination of `--max_allowed_packet=128M`, `--single-transaction`, `--skip-extended-insert`, and `--quick`.\n\n## Database Server\n\nBy default the database type is assumed to be MySQL, but you can use the builtin SQL Server support by specifying:\n\n```crystal\nobfuscator.database_type = :sql_server\nobfuscator.database_type = :postgres\n```\n\nIf using Postgres, use `pg_dump` to get a dump:\n\n```\npg_dump database | obfuscator \u003e obfuscated_dump.sql\n```\n\n## Types\n\nAvailable types include:\n- email\n- string\n- lorem\n- name\n- first_name\n- last_name\n- address\n- street_address\n- secondary_address\n- city\n- state\n- zip_code\n- phone\n- company\n- ipv4\n- ipv6\n- url\n- integer\n- fixed\n- null\n\nand `keep` to keep the same value.\n\n## Helping with creation of the \"obfuscator.cr\" script\n\nIf you don't want to type all those table names and column names into your obfuscator.cr script,\nyou can use triki to do some of that work for you. It can consume your database dump file and create a \"scaffold\" for the script.\nTo run triki in this mode, start with an \"empty\" scaffolder.cr script as follows:\n\n```crystal\nobfuscator = Triki.new\nobfuscator.scaffold(STDIN, STDOUT)\n```\n\nThen feed in your database dump:\n\n```\nmysqldump -c  --hex-blob -u user -ppassword database | scaffolder \u003e obfuscator_scaffold_snippet\npg_dump database | scaffolder \u003e obfuscator_scaffold_snippet\n```\n\nThe output will be a series of configuration statements of the form:\n\n```crystal\n  \"table_name\" =\u003e {\n    \"column1_name\" =\u003e :keep   # scaffold\n    \"column2_name\" =\u003e :keep   # scaffold\n    ... etc.\n```\n\nScaffolding also works if you have a partial configuration.  If your configuration is missing some tables or some columns, a call to 'scaffold' will pass through the configuration that exists and augment it with scaffolding for the missing tables or columns.\n\n## Speed\n\nThe main motivation to rewrite this from Ruby to Crystal was speed, here is an example obfuscating 16 tables and 15 columns in total.\n\n### MySQL dump 160MB (gzip'ed)\n\n#### Ruby\n\n```\nreal    1m56.980s\nuser    1m57.080s\nsys     0m2.660s\n```\n\n#### Crystal\n\n```\nreal    0m26.579s\nuser    0m28.220s\nsys     0m1.748s\n```\n\n### MySQL dump 1.4G\n\n#### Ruby\n\n```\nreal    1m52.974s\nuser    1m49.824s\nsys     0m4.560s\n```\n\n#### Crystal\n\n```\nreal    0m17.642s\nuser    0m17.952s\nsys     0m2.192s\n```\n\nThat's about 6.40x speedup compared to the Ruby version.\n\n## Note on Patches/Pull Requests\n\n* Fork the project.\n* Make your feature addition or bug fix.\n* Add tests for it. This is important so I don't break it in a future version unintentionally.\n* Commit, do not mess with version. (If you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)\n* Send me a pull request. Bonus points for topic branches.\n\n## Thanks\n\nForked from [https://github.com/cantino/my_obfuscate](https://github.com/cantino/my_obfuscate)\n\nThanks to all of the authors and contributors of the original Ruby gem\n\n## LICENSE\n\nThis work is provided under the MIT License.  See the included LICENSE file.\n\nThe included English word frequency list used for generating random text is provided under the Creative Commons – Attribution / ShareAlike 3.0 license by http://invokeit.wordpress.com/frequency-word-lists/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjosacar%2Ftriki","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjosacar%2Ftriki","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjosacar%2Ftriki/lists"}