{"id":19651279,"url":"https://github.com/embulk/embulk-filter-encrypt","last_synced_at":"2025-04-28T16:31:22.256Z","repository":{"id":49196306,"uuid":"52924215","full_name":"embulk/embulk-filter-encrypt","owner":"embulk","description":"Encrypt filter plugin for Embulk","archived":false,"fork":false,"pushed_at":"2021-06-24T04:16:43.000Z","size":179,"stargazers_count":6,"open_issues_count":1,"forks_count":1,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-14T19:54:39.269Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/embulk.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-03-02T01:41:48.000Z","updated_at":"2021-06-24T04:15:05.000Z","dependencies_parsed_at":"2022-07-26T01:02:14.580Z","dependency_job_id":null,"html_url":"https://github.com/embulk/embulk-filter-encrypt","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-filter-encrypt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-filter-encrypt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-filter-encrypt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-filter-encrypt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/embulk","download_url":"https://codeload.github.com/embulk/embulk-filter-encrypt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251345903,"owners_count":21574806,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T15:05:57.165Z","updated_at":"2025-04-28T16:31:21.710Z","avatar_url":"https://github.com/embulk.png","language":"Java","readme":"# Encrypt filter plugin for Embulk\n\nConverts columns using an encryption algorithm such as AES.\n\nEncrypted data is encoded using base64. For example, if you have following input records:\n\n    id,password,comment\n    1,super,a\n    2,secret,b\n\nYou can apply encryption to password column and get following outputs:\n\n    id,password,comment\n    1,ayxU9lMA1iASdHGy/eAlWw==,a\n    2,v8ffsUOfspaqZ1KI7tPz+A==,b\n\n## Overview\n\n* **Plugin type**: filter\n\n## Configuration\n\n- **algorithm**: encryption algorithm (see below) (enum, required)\n- **column_names**: names of string columns to encrypt (array of string, required)\n- **key_type**: encryption key (enum, optional, default: inline), can be either \"inline\" or \"s3\"\n- **key_hex**: encryption key (string, required if key_type is inline)\n- **iv_hex**: encryption initialization vector (string, required if mode of the algorithm is CBC and key_type is inline)\n- **output_encoding**: the encoding of encrypted value, can be either \"base64\" or \"hex\" (base16)\n- **aws_params**: AWS/S3 parameters (hash, required if key_type is s3)\n    - **region**: a valid AWS region\n    - **access_key**: a valid AWS access key\n    - **secret_key**: a valid AWS secret key\n    - **bucket**: a valid S3 bucket\n    - **path**: a valid S3 key (S3 file path)\n    \n## Algorithms\n\nAvailable algorithms are:\n\n* **AES-256-CBC** (recommended)\n* AES-192-CBC\n* AES-128-CBC\n* AES-256-ECB\n* AES-192-ECB\n* AES-128-ECB\n\nAES-256-CBC is the recommended algorithm. The other algorithms are prepared for compatibility with other components (see below \"Decrypting data\" section).\n\n## Generating key and iv\n\n### Using standard PBKDF2 Password-based Encryption algorithm\n\nPBKDF2 is a standard (PKCS #5) algorithm to generate key and iv from a password.\n\nTo generate it, you can use [genkey.rb](https://raw.githubusercontent.com/embulk/embulk-filter-encrypt/master/genkey.rb) script.\n\nYou save above text as \"genkey.rb\", and run it as following:\n\n    $ ruby genkey.rb AES-256-CBC \"my-pass-wo-rd\"\n\nIt shows key and iv as following:\n\n    key=D0867C9310D061F17ACD11EB30DE68265DCB79849BE5FB2BE157919D19BF2F42\n    iv =2A1D6BD59D2DB50A59364BAD3B9B6544\n\n### Using openssl EVP_BytesToKey algorithm\n\nYou can use `openssl` EVP_BytesToKey algorithm to generate key and iv from a password. If you use AES-256-CBC cipher algorithm, you type following command:\n\n    $ echo secret | openssl enc -aes-256-cbc -a -nosalt -p\n\nYou will be asked to enter password. Then it shows key and iv:\n\n    key=DAFFED346E29C5654F54133D1FC65CCB5930071ACEAF5B64A22A11406F467DC9\n    iv =C92D28D70B4440DA3F0F05577ECFEE54\n    6aEGvMrGx7tODkPF7x5Yog==\n\nYou can copy key and iv to key_hex and iv_hex parameters.\n\n## Decrypting data\n\n### openssl command\n\nYou can use openssl command as following:\n\n    $ echo \u003cencrypted value\u003e | openssl enc -d -base64 | openssl enc -aes-256-cbc -d -K \u003ckey\u003e -iv \u003civ\u003e\n\nFor example:\n\n    $ echo 6aEGvMrGx7tODkPF7x5Yog== | openssl enc -d -base64 | openssl enc -aes-256-cbc -d -K DAFFED346E29C5654F54133D1FC65CCB5930071ACEAF5B64A22A11406F467DC9 -iv C92D28D70B4440DA3F0F05577ECFEE54\n    secret\n\n### PostgreSQL\n\nYou can use PostgreSQL's `decrypt_iv` or `decrypt` function to decrypt values (provided as pgcrypto extension). If you use CBC,\n\n    decrypt_iv(decode(encrypted_column, 'base64'), decode('here_is_key_hex', 'hex'), decode('here_is_iv_hex', 'hex'), 'aes')\n\nIf you use ECB,\n\n    decrypt(decode(encrypted_column, 'base64'), decode('here_is_key_hex', 'hex'), 'aes')\n\n\u003c!-- This doesn't work. why?\n### MySQL\n\nYou can use MySQL's `AES_DECRYPT` function to decrypt values. If you use CBC,\n\n    AES_DECRYPT(FROM_BASE64(encrypted_column), unhex('here_is_key_hex'), unhex(here_is_iv_hex'))\n\nIf you use ECB,\n\n    AES_DECRYPT(FROM_BASE64(encrypted_column), unhex('here_is_key_hex'))\n--\u003e\n\n\u003c!-- not confirmed yet\n### Hive\n\nYou can use Hive's `aes_decrypt(input binary, key binary)` function (available since Hive 1.3.0) to decrypt values. But because Hive doesn't support CBC, you need to use AES-256-ECB, AES-192-ECB, or AES-128-ECB. Function call is:\n\n    aes_decrypt(unbase64(encrypted_column), unhex('here_is_key_hex'))\n--\u003e\n\n## Example\n\n* Inline key type\n\n```yaml\nfilters:\n  - type: encrypt\n    algorithm: AES-256-CBC\n    column_names: [password, ip]\n    key_hex: 098F6BCD4621D373CADE4E832627B4F60A9172716AE6428409885B8B829CCB05\n    iv_hex: C9DD4BB33B827EB1FBA1B16A0074D460\n    output_encoding: hex\n```\n\n* S3 key type\n\n```yaml\nfilters:\n  - type: encrypt\n    algorithm: AES-256-CBC\n    column_names: [password, ip]\n    output_encoding: hex\n    key_type: s3\n    aws_params:\n      region: us-east-2\n      access_key: XXXXXXXXXXXXXXXXXXXX\n      secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n      bucket: com.sample.keys\n      path: key.aes\n```\n\n## Build\n\n```\n$ ./gradlew gem  # -t to watch change of files and rebuild continuously\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembulk%2Fembulk-filter-encrypt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fembulk%2Fembulk-filter-encrypt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembulk%2Fembulk-filter-encrypt/lists"}