{"id":23023203,"url":"https://github.com/kibitan/pgmasking","last_synced_at":"2025-07-19T17:10:03.541Z","repository":{"id":151835270,"uuid":"250517713","full_name":"kibitan/pgmasking","owner":"kibitan","description":"Command line tool for generating anonymized database for PostgreSQL (WIP)","archived":false,"fork":false,"pushed_at":"2024-09-17T21:13:36.000Z","size":34,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-09-18T02:08:18.945Z","etag":null,"topics":["command-line-tool","database","gdpr","golang","postgres","postgresql","rdbms"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kibitan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["kibitan"]}},"created_at":"2020-03-27T11:32:46.000Z","updated_at":"2024-09-17T21:13:39.000Z","dependencies_parsed_at":"2024-06-21T17:46:49.922Z","dependency_job_id":"9dbb7072-5fad-4229-86e8-136cc1d5a0fa","html_url":"https://github.com/kibitan/pgmasking","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kibitan%2Fpgmasking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kibitan%2Fpgmasking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kibitan%2Fpgmasking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kibitan%2Fpgmasking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kibitan","download_url":"https://codeload.github.com/kibitan/pgmasking/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229821827,"owners_count":18129428,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line-tool","database","gdpr","golang","postgres","postgresql","rdbms"],"created_at":"2024-12-15T13:12:48.327Z","updated_at":"2024-12-15T13:12:49.074Z","avatar_url":"https://github.com/kibitan.png","language":"Shell","funding_links":["https://github.com/sponsors/kibitan"],"categories":[],"sub_categories":[],"readme":"# pgMasKING🤴\n\n[![build](https://github.com/kibitan/pgmasking/workflows/build/badge.svg?branch=master)](https://github.com/kibitan/pgmasking/actions?query=workflow%3Abuild+branch%3Amaster)\n[![Acceptance Test Status](https://github.com/kibitan/pgmasking/workflows/Acceptance%20Test/badge.svg?branch=master)](https://github.com/kibitan/pgmasking/actions?query=workflow%3A%22Acceptance+Test%22+branch%3Amaster)\n[![Maintainability](https://api.codeclimate.com/v1/badges/d0a198f3633d329592af/maintainability)](https://codeclimate.com/github/kibitan/pgmasking/maintainability)\n[![codecov](https://codecov.io/gh/kibitan/pgmasking/branch/master/graph/badge.svg)](https://codecov.io/gh/kibitan/pgmasking)\n\n*This project is currently Work in Progress*\n\nThe command line tool for anonymizing PostgreSQL database records by parsing a SQL dump file and build a new SQL dump file with masking sensitive/credential data.\n\nfor MySQL: [MasKING](https://github.com/kibitan/masking)\n\n## Installation\n\n```bash\n(TBC)\n```\n\n## Requirement for development\n\n* Golang 1.22\n\n## Supported RDBMS\n\n* PostgreSQL: 12, 11, 10, 9.6, 9.5\n\n## Usage\n\n1. Setup configuration for anonymizing target tables/columns to `pgmasking.yml`\n\n    ```yaml\n      # table_name:\n      #   column_name: masked_value\n\n      users:\n        string: anonymized string\n        email: anonymized+%{n}@example.com # %{n} will be replaced with sequential number\n        integer: 12345\n        float: 123.45\n        boolean: true\n        null_column: null\n        date: 2018-08-24\n        time: 2018-08-24 15:54:06\n        binary_or_blob: !binary | # Binary Data Language-Independent Type for YAML™ Version 1.1: http://yaml.org/type/binary.html\n          R0lGODlhDAAMAIQAAP//9/X17unp5WZmZgAAAOfn515eXvPz7Y6OjuDg4J+fn5\n          OTk6enp56enmlpaWNjY6Ojo4SEhP/++f/++f/++f/++f/++f/++f/++f/++f/+\n          +f/++f/++f/++f/++f/++SH+Dk1hZGUgd2l0aCBHSU1QACwAAAAADAAMAAAFLC\n          AgjoEwnuNAFOhpEMTRiggcz4BNJHrv/zCFcLiwMWYNG84BwwEeECcgggoBADs=\n    ```\n\n    A value will be implicitly converted to a compatible type. If you prefer to explicitly convert, you could use a tag as defined in [YAML Version 1.1](http://yaml.org/spec/current.html#id2503753)\n\n    ```yaml\n      not-date: !!str 2002-04-28\n    ```\n\n    *NOTE: pgMasKING doesn't check actual schema's type from the dump. If you put incompatible value, it will cause an error during restoring to the database.*\n\n1. Dump database with anonymizing\n\n    (TBC)\n    pgMasKING work with `pg_dump` command. It doesn't (or only) work with `--column-inserts`/`--attribute-inserts`/`--rows-per-insert=n`(version 12~) options.\n\n    ```bash\n      pg_dump DATABASE_NAME | pgmasking \u003e anonymized_dump.sql\n    ```\n\n    or\n\n    ```bash\n      pg_dump DATABASE_NAME --column-inserts --rows-per-insert=100 | pgmasking \u003e anonymized_dump.sql\n    ```\n\n1. Restore from the anonymized dump file\n\n    ```bash\n      psql ANONYMIZED_DATABASE_NAME \u003c anonymized_dump.sql\n    ```\n\n    Tip: If you don't need to have an anonymized dump file, you can directly insert it from the stream. It can be faster because it has less IO interaction.\n\n      ```bash\n        pg_dump DATABASE_NAME | pgmasking | psql ANONYMIZED_DATABASE_NAME\n      ```\n\n### options\n\n(WIP)\n\n```bash\n$ pgmasking -h\nUsage: pgmasking [options]\n    -c, --config=FILE_PATH           specify config file. default: pgmasking.yml\n    -v, --version                    version\n```\n\n## Use case of anonymized (production) database\n\n[Read here](https://github.com/kibitan/masking#use-case-of-anonymized-production-database)\n\n## Development\n\n```bash\ngit clone git@github.com:kibitan/pgmasking.git\n```\n\n### boot\n\n```bash\n  go run .\n```\n\nor\n\n```bash\n  go build .\n  ./pgmasking\n```\n\nor\n\n```bash\n  go install .\n  pgmasking\n```\n\n### Run test\n\n```bash\n  go test -race -v ./...\n```\n\n### Lint\n\n```bash\n  golint ./... \u0026\u0026 go vet ./...\n```\n\n### Document\n\n(TBC)\n\n```bash\n  go get golang.org/x/tools/cmd/godoc\n  go doc pgmasking\n\n  godoc // boot http server\n  open http://localhost:6060\n```\n\n#### acceptance test\n\n```bash\n./acceptance/run_test.sh\n```\n\navailable option via environment variable:\n\n* `POSTGRES_HOST`: database host(default: `localhost`)\n* `POSTGRES_USER`: postgres user name(default: `postgres`}\n* `POSTGRES_PASSWORD`: password for user(default: `password`)\n* `POSTGRES_DBNAME`: database name(default: `pgmasking_acceptance`)\n\n##### with docker-compose\n\n```bash\ndocker-compose/acceptance_test.sh postgres12\n```\n\nThe docker-compose file names for other database versions, specify that file.\n\n* PostgreSQL 12: [`docker-compose/postgres12.yml`](./docker-compose/postgres12.yml)\n* PostgreSQL 11: [`docker-compose/postgres11.yml`](./docker-compose/postgres11.yml)\n* PostgreSQL 10: [`docker-compose/postgres10.yml`](./docker-compose/postgres10.yml)\n* PostgreSQL 9.6: [`docker-compose/postgres96.yml`](./docker-compose/postgres96.yml)\n* PostgreSQL 9.5: [`docker-compose/postgres95.yml`](./docker-compose/postgres95.yml)\n\n## Development with Docker\n\n```bash\ndocker build . -t pgmasking\necho \"sample stdout\" | docker run -i pgmasking\ndocker run pgmasking -v\n```\n\n### run test\n\n```bash\ndocker build . --target builder -t pgmasking-builder\ndocker run pgmasking-builder go test -v ./..\n\n# with mounting disk\ndocker run --mount src=`pwd`,target=/go/src/app,type=bind pgmasking-builder go test -v ./..\n```\n\n## Profiling\n\n(TBC)\n\n### Benchmark\n\n(TBC)\n\n## Design Concept\n\n[Read here](https://github.com/kibitan/masking#design-concept)\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at [https://github.com/kibitan/pgmasking](https://github.com/kibitan/pgmasking).\nThis project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n\n## Code of Conduct\n\nEveryone interacting in the pgMasKING project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/kibitan/masking/blob/master/CODE_OF_CONDUCT.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkibitan%2Fpgmasking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkibitan%2Fpgmasking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkibitan%2Fpgmasking/lists"}