{"id":21355085,"url":"https://github.com/salrashid123/bq_aead_key","last_synced_at":"2025-03-16T05:23:45.656Z","repository":{"id":91309771,"uuid":"413401692","full_name":"salrashid123/bq_aead_key","owner":"salrashid123","description":"Importing and extracting external keys for BigQuery AEAD Tink KeySets","archived":false,"fork":false,"pushed_at":"2021-10-05T11:09:00.000Z","size":25,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-22T17:47:18.169Z","etag":null,"topics":["encryption","golang","google-bigquery","google-cloud-platform","tink-crypto"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salrashid123.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-04T11:58:15.000Z","updated_at":"2023-06-24T20:14:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"9dd08f91-8b15-4599-bf12-c1d1c6b4c3b3","html_url":"https://github.com/salrashid123/bq_aead_key","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fbq_aead_key","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fbq_aead_key/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fbq_aead_key/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fbq_aead_key/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salrashid123","download_url":"https://codeload.github.com/salrashid123/bq_aead_key/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243828709,"owners_count":20354545,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["encryption","golang","google-bigquery","google-cloud-platform","tink-crypto"],"created_at":"2024-11-22T04:15:46.114Z","updated_at":"2025-03-16T05:23:45.639Z","avatar_url":"https://github.com/salrashid123.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Importing and extracting external keys for BigQuery AEAD Tink KeySets\n\n\n[BigQuery AEAD encryption](https://cloud.google.com/bigquery/docs/reference/standard-sql/aead-encryption-concepts) functions uses [TINK Keysets](https://cloud.google.com/bigquery/docs/reference/standard-sql/aead-encryption-concepts#keysets).  \n\nWorks fine but all samples included there describe how to generate an encoded key using a BQ function itself: [KEYS.NEW_KEYSET('AEAD_AES_GCM_256')](https://cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#keysnew_keyset)\n\nHowever, what if you \n\n`a)` already have a _raw_ `AEAD_AES_GCM_256` that you want to use with BQ\n  or\n`b)` you've already generated a Key within BQ and want to extract the base `AEAD_AES_GCM_256` from a keyset and want to decrypt it on a sunday like as i'm writing this today.\n\nyou're probably wondering how to do that?\n\nwell\n\n- for `a)` you need to create a Tink Keyset from a raw AES key\n\n- for `b)` you need to extract an AES key from an existing TINK keyset\n\nthats what you can use this repo for..\n\n\u003e\u003e This is not supported by Google.  _caveat emptor_\n\n---\n\n### Importing to Tink\n\nIf the AES_GCM key is:\n\n```golang\n\t// 1. AES GCM Key\n\tsecret := \"change this password to a secret\"\n```\n\nthen import that into a tink keyset.  You can use the encoded key with Bigquery\n\n```log\n$ go run import_aes_gcm/main.go \n\nTink Keyset Encoded:  CMKIrNYJEmQKWAowdHlwZS5nb29nbGVhcGlzLmNvbS9nb29nbGUuY3J5cHRvLnRpbmsuQWVzR2NtS2V5EiIaIGNoYW5nZSB0aGlzIHBhc3N3b3JkIHRvIGEgc2VjcmV0GAEQARjCiKzWCSAB\nTink Keyset:\n {\n\t\"primaryKeyId\": 2596996162,\n\t\"key\": [\n\t\t{\n\t\t\t\"keyData\": {\n\t\t\t\t\"typeUrl\": \"type.googleapis.com/google.crypto.tink.AesGcmKey\",\n\t\t\t\t\"value\": \"GiBjaGFuZ2UgdGhpcyBwYXNzd29yZCB0byBhIHNlY3JldA==\",\n\t\t\t\t\"keyMaterialType\": \"SYMMETRIC\"\n\t\t\t},\n\t\t\t\"status\": \"ENABLED\",\n\t\t\t\"keyId\": 2596996162,\n\t\t\t\"outputPrefixType\": \"TINK\"\n\t\t}\n\t]\n}\n\nAttempt to decrypt with BQ output\nPlain text Decrypted from BQ: Greed\nPlain text Decrypted from BQ: Greed\nPlain text Decrypted from BQ: Greed\nReCreated Raw Key: change this password to a secret\n```\n\nuse the encoded key with the key of your choosing with [AEAD.ENCRYPT](https://cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#aeadencrypt) function\n\n```sql\n$ bq query --nouse_legacy_sql --parameter=keyset1::CMKIrNYJEmQKWAowdHlwZS5nb29nbGVhcGlzLmNvbS9nb29nbGUuY3J5cHRvLnRpbmsuQWVzR2NtS2V5EiIaIGNoYW5nZSB0aGlzIHBhc3N3b3JkIHRvIGEgc2VjcmV0GAEQARjCiKzWCSAB \\ '\nSELECT\n  title,AEAD.ENCRYPT(FROM_BASE64(@keyset1),title,\"\")\nFROM\n  bigquery-public-data.san_francisco_film_locations.film_locations\nWHERE\n  title = \"Greed\"\n'\n\n+-------+------------------------------------------------------+\n| title |                         f0_                          |\n+-------+------------------------------------------------------+\n| Greed | AZrLBEKaUq6kFMfPY7XzKcFxvSCJQ31WYqnJEPAzsHPhk6WQ0S4= |\n| Greed | AZrLBEL2mv3fZ05icMISrDXdN35gtAX54Z4zqDN0rDevfsfSFoY= |\n| Greed | AZrLBEKjpZf+H+JIFijHakbiHtrtY09GNPTrpeHL95CYZj+jk/8= |\n+-------+------------------------------------------------------+\n```\n\nthen decrypt the BQ output using [AEAD.DECRYPT](https://cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#aeaddecrypt_string) and the key you generated\n\n```sql\nbq query --nouse_legacy_sql --parameter=keyset1::CMKIrNYJEmQKWAowdHlwZS5nb29nbGVhcGlzLmNvbS9nb29nbGUuY3J5cHRvLnRpbmsuQWVzR2NtS2V5EiIaIGNoYW5nZSB0aGlzIHBhc3N3b3JkIHRvIGEgc2VjcmV0GAEQARjCiKzWCSAB \\ '\nSELECT\n  AEAD.DECRYPT_STRING(FROM_BASE64(@keyset1),FROM_BASE64(\"AZrLBEKaUq6kFMfPY7XzKcFxvSCJQ31WYqnJEPAzsHPhk6WQ0S4=\"),\"\"),\n  AEAD.DECRYPT_STRING(FROM_BASE64(@keyset1),FROM_BASE64(\"AZrLBEL2mv3fZ05icMISrDXdN35gtAX54Z4zqDN0rDevfsfSFoY=\"),\"\"),\n  AEAD.DECRYPT_STRING(FROM_BASE64(@keyset1),FROM_BASE64(\"AZrLBEKjpZf+H+JIFijHakbiHtrtY09GNPTrpeHL95CYZj+jk/8=\"),\"\"),    \n'\n\n+-------+-------+-------+\n|  f0_  |  f1_  |  f2_  |\n+-------+-------+-------+\n| Greed | Greed | Greed |\n+-------+-------+-------+\n```\n\nJust as verification you can even use golang to verify too using the keyset you just imported the raw key into\n\n```golang\n\n\tlog.Printf(\"Attempt to decrypt with BQ output\\n\")\n\n\tbqCiphers := []string{\"AZrLBEKaUq6kFMfPY7XzKcFxvSCJQ31WYqnJEPAzsHPhk6WQ0S4=\", \"AZrLBEL2mv3fZ05icMISrDXdN35gtAX54Z4zqDN0rDevfsfSFoY=\", \"AZrLBEKjpZf+H+JIFijHakbiHtrtY09GNPTrpeHL95CYZj+jk/8=\"}\n\n\tfor _, bc := range bqCiphers {\n\t\tbb, err := base64.StdEncoding.DecodeString(bc)\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\n\t\tdl, err := a.Decrypt(bb, []byte(\"\"))\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\t\tlog.Printf(\"Plain text Decrypted from BQ: %s\\n\", string(dl))\n\t}\n```\n\n## Exporting from TINK\n\nTo do the reverse (extract a key from tink, see the sample in this repo here which will unmarshall the encoded key.  Your key here for BQ is \n\n```golang\nconst (\n\tkeySetString = \"CMKIrNYJEmQKWAowdHlwZS5nb29nbGVhcGlzLmNvbS9nb29nbGUuY3J5cHRvLnRpbmsuQWVzR2NtS2V5EiIaIGNoYW5nZSB0aGlzIHBhc3N3b3JkIHRvIGEgc2VjcmV0GAEQARjCiKzWCSAB\"\n)\n```\n\n\nwhich is actually the same raw key from above: `change this password to a secret`\n\n```log\n$ go run export_aes_gcm/main.go \nTink Keyset:\n {\n\t\"primaryKeyId\": 2596996162,\n\t\"key\": [\n\t\t{\n\t\t\t\"keyData\": {\n\t\t\t\t\"typeUrl\": \"type.googleapis.com/google.crypto.tink.AesGcmKey\",\n\t\t\t\t\"value\": \"GiBjaGFuZ2UgdGhpcyBwYXNzd29yZCB0byBhIHNlY3JldA==\",\n\t\t\t\t\"keyMaterialType\": \"SYMMETRIC\"\n\t\t\t},\n\t\t\t\"status\": \"ENABLED\",\n\t\t\t\"keyId\": 2596996162,\n\t\t\t\"outputPrefixType\": \"TINK\"\n\t\t}\n\t]\n}\n\nEncrypted Data: AZrLBELglGoDvAXMqIT+N7J8Pu6VPBHwM2Hp5Z6tS0Z1Py9szPk=\nPlain text: Greed\nExtracted Raw Key: change this password to a secret\n```\n\n---\n\nreferences\n\n\nfor more refernces, see [https://github.com/salrashid123/tink_samples](https://github.com/salrashid123/tink_samples)\n\n\nYou can also use [tinkkey](https://developers.google.com/tink/install-tinkey) to manage a keyset\n\n```bash\n$ ./tinkey list-keyset --in-format=json --in keyset.json\nprimary_key_id: 2596996162\nkey_info {\n  type_url: \"type.googleapis.com/google.crypto.tink.AesGcmKey\"\n  status: ENABLED\n  key_id: 2596996162\n  output_prefix_type: TINK\n}\n\n$ ./tinkey rotate-keyset --in-format=json --in keyset.json  --key-template AES256_GCM --out-format=json --out keyset2.json\n\n$ ./tinkey list-keyset --in-format=json --in keyset2.json\nprimary_key_id: 2130552249\nkey_info {\n  type_url: \"type.googleapis.com/google.crypto.tink.AesGcmKey\"\n  status: ENABLED\n  key_id: 2596996162\n  output_prefix_type: TINK\n}\nkey_info {\n  type_url: \"type.googleapis.com/google.crypto.tink.AesGcmKey\"\n  status: ENABLED\n  key_id: 2130552249\n  output_prefix_type: TINK\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalrashid123%2Fbq_aead_key","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalrashid123%2Fbq_aead_key","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalrashid123%2Fbq_aead_key/lists"}