Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dwyl/phoenix-ecto-encryption-example
🔐 A detailed example for how to encrypt data in an Elixir (Phoenix v1.7) App before inserting into a database using Ecto Types
https://github.com/dwyl/phoenix-ecto-encryption-example
aes aes-256 aes-gcm argon2 custom database ecto elixir encryption learn phoenix tutorial types
Last synced: about 1 month ago
JSON representation
🔐 A detailed example for how to encrypt data in an Elixir (Phoenix v1.7) App before inserting into a database using Ecto Types
- Host: GitHub
- URL: https://github.com/dwyl/phoenix-ecto-encryption-example
- Owner: dwyl
- Created: 2018-01-24T21:59:28.000Z (almost 7 years ago)
- Default Branch: main
- Last Pushed: 2024-09-17T09:30:06.000Z (3 months ago)
- Last Synced: 2024-09-17T12:06:09.405Z (3 months ago)
- Topics: aes, aes-256, aes-gcm, argon2, custom, database, ecto, elixir, encryption, learn, phoenix, tutorial, types
- Language: Elixir
- Homepage:
- Size: 430 KB
- Stars: 273
- Watchers: 18
- Forks: 20
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- freaking_awesome_elixir - Elixir - A comprehensive example/tutorial showing people how to use Ecto Types to transparently encrypt/decrypt data in a Phoenix 1.4 app. (Examples and funny stuff)
- fucking-awesome-elixir - phoenix-ecto-encryption-example - A comprehensive example/tutorial showing people how to use Ecto Types to transparently encrypt/decrypt data in a Phoenix 1.4 app. (Examples and funny stuff)
- awesome-elixir - phoenix-ecto-encryption-example - A comprehensive example/tutorial showing people how to use Ecto Types to transparently encrypt/decrypt data in a Phoenix 1.4 app. (Examples and funny stuff)
README
# Phoenix Ecto Encryption Example
[](https://github.com/dwyl/phoenix-ecto-encryption-example/actions)
[](https://codecov.io/github/dwyl/phoenix-ecto-encryption-example?branch=master)
[](https://hex.pm/packages/phoenix)
[](https://hexdocs.pm/fields/api-reference.html)
[](https://github.com/dwyl/phoenix-ecto-encryption-example/issues)
[](https://hits.dwyl.com/dwyl/phoenix-ecto-encryption-example)
💡 **Note**: we wrote this example/tutorial
to _understand_ how to do field-level encryption
from **first principals**.
Once we solved the problem,
we built a library to streamline it:
[**`fields`**](https://github.com/dwyl/fields).
We still recommend going through this example,
but if you just want
to get on with building your **`Phoenix` App**,
use
[**`fields`**](https://github.com/dwyl/fields).
- [Phoenix Ecto Encryption Example](#phoenix-ecto-encryption-example)
- [Why?](#why)
- [What?](#what)
- [Technical Overview](#technical-overview)
- [OWASP Cryptographic Rules?](#owasp-cryptographic-rules)
- [Who?](#who)
- [Prerequisites?](#prerequisites)
- [Crypto Knowledge?](#crypto-knowledge)
- [Time Requirement?](#time-requirement)
- [How?](#how)
- [1. Create the `encryption` App](#1-create-the-encryption-app)
- [2. Create the `user` Schema (_Database Table_)](#2-create-the-user-schema-database-table)
- [3. Define The 6 Functions](#3-define-the-6-functions)
- [3.1 Encrypt](#31-encrypt)
- [Test the `encrypt/1` Function](#test-the-encrypt1-function)
- [3.2 Decrypt](#32-decrypt)
- [Test the `decrypt/1` Function](#test-the-decrypt1-function)
- [3.3 Key rotation](#33-key-rotation)
- [3.4 `ENCRYPTION_KEYS` Environment Variable](#34--encryption_keys-environment-variable)
- [Test the `get_key/0` and `get_key/1` Functions?](#test-the-get_key0-and-get_key1-functions)
- [4. Hash _Email Address_](#4-hash-email-address)
- [4.1 Generate the `SECRET_KEY_BASE`](#41-generate-the-secret_key_base)
- [5. _Create_ and use `HashField` Custom Ecto Type](#5-create-and-use-hashfield-custom-ecto-type)
- [`type/0`](#type0)
- [`cast/1`](#cast1)
- [`dump/1`](#dump1)
- [`load/1`](#load1)
- [`embed_as/1`](#embed_as1)
- [`equal?/2`](#equal2)
- [6. Create and user Hash _Password_ Custom Ecto type](#6-create-and-user-hash-password-custom-ecto-type)
- [Add the `argon2` Dependency](#add-the-argon2-dependency)
- [6.1 Define the `hash_password/1` Function](#61-define-the-hash_password1-function)
- [6.1.1 Test the `hash_password/1` Function?](#611-test-the-hash_password1-function)
- [6.2 _Verify_ Password](#62-verify-password)
- [Test for `verify_password/2`](#test-for-verify_password2)
- [7. _Create_ and use `EncryptedField` Custom Ecto Type](#7-create-and-use-encryptedfield-custom-ecto-type)
- [`type/0`](#type0-1)
- [`cast/1`](#cast1-1)
- [`dump/1`](#dump1-1)
- [`load/1`](#load1-1)
- [`embed_as/1`](#embed_as1-1)
- [`equal?/2`](#equal2-1)
- [8. Ensure All Tests Pass](#8-ensure-all-tests-pass)
- [Conclusion](#conclusion)
- [How To Generate AES Encryption Keys?](#how-to-generate-aes-encryption-keys)
- [Useful Links, FAQ \& Background Reading](#useful-links-faq--background-reading)
- [Understanding Advanced Encryption Standard (AES)](#understanding-advanced-encryption-standard-aes)
- [Running a Single Test](#running-a-single-test)
- [Ecto Validation Error format](#ecto-validation-error-format)
- [Stuck / Need Help?](#stuck--need-help)
- [Credits](#credits)
# Why?
**Encrypting User/Personal data** stored by your Web App is ***essential***
for security/privacy.
> If your app offers any **personalised content** or interaction
that depends on "**login**", it is **_storing_ personal data**
(_by definition_).
You might be tempted to think that
data is "safe" in a database, but it's _not_.
There is an entire ("dark") army/industry of people
([_cybercriminals_](https://en.wikipedia.org/wiki/Cybercrime))
who target websites/apps attempting to "steal" data by compromising databases.
All the time you spend _building_ your app,
they spend trying to "_break_" apps like yours.
Don't let the people using your app
be the victims of identity theft,
protect their personal data!
(_it's both the "**right**" **thing to do** and the
[**law**](https://github.com/dwyl/learn-security/#gdpr) ..._)
# What?
This example/tutorial is intended as a _comprehensive_ answer
to the question:
> ["_**How to Encrypt/Decrypt Sensitive Data** in `Elixir` Apps
**Before** Inserting (Saving) it into the
Database?_"](https://github.com/dwyl/learn-elixir/issues/80)
## Technical Overview
We are _not_ "re-inventing encryption"
or using our "own algorithm"
_everyone_ knows that's a "_**bad** idea_":
https://security.stackexchange.com/questions/18197/why-shouldnt-we-roll-our-own
We are _following_ a **_battle-tested_ industry-standard** approach
and applying it to our Elixir/Phoenix App.
We are using:
+ **Advanced Encryption Standard** (**AES**) to encrypt sensitive data.
+ **Galois/Counter Mode**
for _symmetric_ key cryptographic block ciphers:
https://en.wikipedia.org/wiki/Galois/Counter_Mode
recommended by many security and cryptography practitioners including
[Matthew Green](https://blog.cryptographyengineering.com/),
[Niels Ferguson](https://en.wikipedia.org/wiki/Niels_Ferguson)
and [Bruce Schneier](https://www.schneier.com/blog/about/)
+ "Under the hood" we are using Erlang's
[crypto](https://erlang.org/doc/man/crypto.html) library
_specifically_ AES with **256 bit keys**
(_the same as AWS or Google's KMS service_)
see: https://erlang.org/doc/man/crypto.html#block_encrypt-4
+ Password "hashing" using the **Argon2**
key derivation function (KDF): https://en.wikipedia.org/wiki/Argon2
_specifically_ the Elixir implementation of `argon2`
written by David Whitlock: https://github.com/riverrun/argon2_elixir
which in turn uses the **C** "_reference implementation_"
as a "Git Submodule".
> `¯\_(ツ)_/¯...?` Don't be "put off" if any of these terms/algorithms
are _unfamiliar_ to you;
this example is "***step-by-step***"
and we are _happy_ to answer/clarify
_any_ (_relevant and specific_) questions you have!
### OWASP Cryptographic Rules?
This example/tutorial follows
the Open Web Application Security Project (**OWASP**)
Cryptographic and Password rules:
+ [x] Use "***strong approved Authenticated Encryption***"
based on an ***AES algorithm***.
+ [x] Use GCM mode of operation for symmetric key cryptographic block ciphers.
+ [x] Keys used for encryption must be rotated at least annually.
+ [x] Only use approved public algorithm **SHA-256** or better for hashing.
+ [x] **Argon2** is the winner of the password hashing competition
and should be your ***first choice*** for **_new_ applications**.
See:
+ [https://www.owasp.org/index.php/Cryptographic_Storage_Cheat_Sheet](https://www.owasp.org/index.php/Cryptographic_Storage_Cheat_Sheet#Rule_-_Use_strong_approved_Authenticated_Encryption)
+ https://www.owasp.org/index.php/Password_Storage_Cheat_Sheet
# Who?
This example/tutorial is for _any_ developer
(_or technical decision maker / "application architect"_)
who takes personal data protection _seriously_
and wants a robust/reliable and "transparent" way
of _encrypting data_ `before` storing it,
and _decrypting_ when it is queried.
## Prerequisites?
+ Basic **`Elixir`** syntax knowledge: https://github.com/dwyl/learn-elixir
+ Familiarity with the **`Phoenix`** framework: https://github.com/dwyl/learn-phoenix-framework
+ Basic understanding of **`Ecto`**
(_the module used to interface with databases in elixir/phoenix_)
> If you are totally `new` to (_or "rusty" on_)
Elixir, Phoenix or Ecto,
we recommend going through our **Phoenix Chat Example**
(_Beginner's Tutorial_) _first_:
https://github.com/dwyl/phoenix-chat-example
### Crypto Knowledge?
You will _not_ need any "advanced" mathematical knowledge;
we are _not_ "inventing" our own encryption or
going into the "internals" of any cyphers/algorithms/schemes.
**You do _not_ need to _understand_
how the encryption/hashing algorithms _work_,**
but it is _useful_ to **know the _difference_** between
[encryption](https://en.wikipedia.org/wiki/Encryption)
vs.
[hashing](https://en.wikipedia.org/wiki/Hash_function)
and
[plaintext](https://en.wikipedia.org/wiki/Plaintext)
vs.
[ciphertext](https://en.wikipedia.org/wiki/Ciphertext).
The fact that the example/tutorial follows _all_ OWASP crypto/hashing rules
(_see:_
["OWASP Cryptographic Rules?"](https://github.com/dwyl/phoenix-ecto-encryption-example#owasp-cryptographic-rules)
_section above_),
should be "enough" for _most_ people who just want to focus
on building their app and don't want to
["_go down the rabbit hole_"](https://youtu.be/6IDT3MpSCKI?t=1m2s).
_However_ ... We have included 30+ links in the
["Useful Links"](https://github.com/dwyl/phoenix-ecto-encryption-example#useful-links-faq--background-reading)
section at the _end_ of this readme.
The list includes several common questions (_and **answers**_)
so if you are _curious_, you can [learn](https://youtu.be/hOZnP4dZYK0).
> _**Note**: in the @dwyl Library we have_
https://www.schneier.com/books/applied_cryptography
_So, if you're **really curious** let us know!_
### Time Requirement?
Simply _reading_ ("_skimming_") through this example will
only take **15 minutes**.
_Following_ the examples on your computer (_to fully understand it_)
will take around **1 hour**
(_including reading a few of the links_).
> _**Invest** the time **up-front** to **avoid** on the **embarrassment**
and [**fines**](https://www.itgovernance.co.uk/dpa-and-gdpr-penalties)
of a data breach_.
# How?
These are "step-by-step" instructions,
don't skip any step(s).
## 1. Create the `encryption` App
In your Terminal,
***create*** a `new` Phoenix application called "encryption":
```sh
mix phx.new encryption
```
When you see `Fetch and install dependencies? [Yn]`,
type `y` and press the `[Enter]` key
to download and install the dependencies.
You should see following in your terminal:
```sh
* running mix deps.get
* running mix deps.compile
* running cd assets && npm install && node node_modules/webpack/bin/webpack.js --mode development
We are almost there! The following steps are missing:
$ cd encryption
Then configure your database in config/dev.exs and run:
$ mix ecto.create
Start your Phoenix app with:
$ mix phx.server
You can also run your app inside IEx (Interactive Elixir) as:
$ iex -S mix phx.server
```
Follow the _first_ instruction
**change** into the `encryption` directory:
```sh
cd encryption
```
Next ***create*** the database for the App using the command:
```sh
mix ecto.create
```
You should see the following output:
```sh
Compiling 13 files (.ex)
Generated encryption app
The database for Encryption.Repo has been created
```
## 2. Create the `user` Schema (_Database Table_)
In our _example_ `user` database table,
we are going to store 3 (_primary_) pieces of data.
+ `name`: the person's name (_encrypted_)
+ `email`: their email address (_encrypted_)
+ `password_hash`: the hashed password (_so the person can login_)
In _addition_ to the 3 "_primary_" fields,
we need _**one** more field_ to store "metadata":
+ `email_hash`: so we can check ("lookup")
if an email address is in the database
_without_ having to _decrypt_ the email(s) stored in the DB.
Create the `user` schema using the following generator command:
```sh
mix phx.gen.schema User users email:binary email_hash:binary name:binary password_hash:binary
```
The _reason_ we are creating the encrypted/hashed fields as `:binary`
is that the _data_ stored in them will be _encrypted_
and `:binary` is the _most efficient_ Ecto/SQL data type
for storing encrypted data;
storing it as a `String` would take up more bytes
for the _same_ data.
i.e. _wasteful_ without any _benefit_ to security or performance.
see:
https://dba.stackexchange.com/questions/56934/what-is-the-best-way-to-store-a-lot-of-user-encrypted-data
and: https://elixir-lang.org/getting-started/binaries-strings-and-char-lists.html
Next we need to update our newly created migration file. Open
`priv/repo/migrations/{timestamp}_create_users.exs`.
> Your migration file will
have a slightly different name to ours as migration files are named with a
timestamp when they are created but it will be in the same location.
Update the file ***from***:
```elixir
defmodule Encryption.Repo.Migrations.CreateUsers do
use Ecto.Migration
def change do
create table(:users) do
add(:email, :binary)
add(:email_hash, :binary)
add(:name, :binary)
add(:password_hash, :binary)
timestamps()
end
end
end
```
**To**
```elixir
defmodule Encryption.Repo.Migrations.CreateUsers do
use Ecto.Migration
def change do
create table(:users) do
add(:email, :binary)
add(:email_hash, :binary)
add(:name, :binary)
add(:password_hash, :binary)
timestamps()
end
create(unique_index(:users, [:email_hash]))
end
end
```
The newly added line ensures that we will never be allowed to enter duplicate
`email_hash` values into our database.
Run the "migration" task to create the tables in the Database:
```sh
mix ecto.migrate
```
Running the `mix ecto.migrate` command will create the
`users` table in your `encryption_dev` database.
You can _view_ this (_empty_) table in a PostgreSQL GUI. Here is a screenshot
from **pgAdmin**:
## 3. Define The 6 Functions
We need 6 functions for encrypting, decrypting, hashing and verifying
the data we will be storing:
1. **Encrypt** - to encrypt any personal data we want to store in the database.
2. **Decrypt** - decrypt any data that needs to be viewed.
3. **Get Key** - get the _latest_ encryption/decryption key
(_or a **specific** older key where data was encrypted with a different key_)
4. **Hash Email** (_deterministic & **fast**_) - so that we can "lookup"
an email without "decrypting".
The hash of an email address should _always_ be the ***same***.
5. **Hash Password** (_pseudorandom & **slow**_) - the output of the hash
should _always_ be ***different*** and relatively **slow** to compute.
6. **Verify Password** - check a password against the stored `password_hash`
to confirm that the person "logging-in" has the _correct_ password.
The next 6 sections of the example/tutorial will walk through
the creation of (_and testing_) these functions.
> _**Note**: If you have **any questions** on these functions_,
***please ask***:
[github.com/dwyl/**phoenix-ecto-encryption-example/issues**](https://github.com/nelsonic/phoenix-ecto-encryption-example/issues)
### 3.1 Encrypt
Create a file called `lib/encryption/aes.ex` and copy-paste (_or hand-write_)
the following code:
```elixir
defmodule Encryption.AES do
@aad "AES256GCM" # Use AES 256 Bit Keys for Encryption.
def encrypt(plaintext) do
iv = :crypto.strong_rand_bytes(16) # create random Initialisation Vector
key = get_key() # get the *latest* key in the list of encryption keys
{ciphertext, tag} =
:crypto.crypto_one_time_aead(:aes_256_gcm, key, iv, to_string(plaintext), @aad, true)
iv <> tag <> ciphertext # "return" iv with the cipher tag & ciphertext
end
defp get_key do # this is a "dummy function" we will update it in step 3.3
<<109, 182, 30, 109, 203, 207, 35, 144, 228, 164, 106, 244, 38, 242,
106, 19, 58, 59, 238, 69, 2, 20, 34, 252, 122, 232, 110, 145, 54,
241, 65, 16>> # return a random 32 Byte / 128 bit binary to use as key.
end
end
```
The `encrypt/1` function for encrypting `plaintext` into `ciphertext`
is quite simple; (_the "body" is only 4 lines_).
Let's "step through" these lines one at a time:
+ `encrypt/1` accepts one argument; the `plaintext` to be encrypted.
+ First we create a "**strong**" _random_
[***initialization vector***](https://en.wikipedia.org/wiki/Initialization_vector)
(IV) of **16 bytes** (***128 bits***)
using the Erlang's crypto library `strong_rand_bytes` function:
https://erlang.org/doc/man/crypto.html#strong_rand_bytes-1
The "IV" ensures that each time a string/block of text/data is encrypted,
the `ciphertext` is _different_.
> Having **different** `ciphertext` each time `plaintext` is encrypted
is _essential_ for
["semantic security"](https://en.wikipedia.org/wiki/Semantic_security)
whereby repeated use of the _same encryption key and algorithm_
does not allow an "attacker" to infer relationships
between segments of the encrypted message.
> [Cryptanalysis](https://en.wikipedia.org/wiki/Cryptanalysis) techniques
are _well_ "beyond scope" for this example/tutorial,
but we _highly_ encourage to check-out the "Background Reading" links
at the end and read up on the subject for deeper understanding.
+ Next we use the `get_key/0` function
to retrieve the _latest_ encryption key
so we can use it to `encrypt` the `plaintext`
(_the "real"_ `get_key/0` _is defined below in section 3.3_).
+ Then we use the Erlang `block_encrypt` function
to encrypt the `plaintext`.
Using `:aes_gcm` ("_Advanced Encryption Standard Galois Counter Mode_"):
+ `@aad` is a "module attribute" (_Elixir's equivalent of a "constant"_)
is defined in `aes.ex` as `@aad "AES256GCM"`
this simply defines the encryption mode we are using which,
if you break down the code into 3 parts:
+ AES = Advanced Encryption Standard.
+ 256 = "256 Bit Key"
+ GCM = "Galois Counter Mode"
+ Finally we "return" the `iv` with the `ciphertag` & `ciphertext`,
this is what we store in the database.
Including the IV and ciphertag is _essential_ for allowing decryption,
without these two pieces of data, we would not be able to "reverse" the process.
> _**Note**: in addition to this_ `encrypt/1` _function,
we have defined an_ `encrypt/2` _"sister" function which accepts
a **specific** (encryption)_ `key_id` _so that we can use the desired
encryption key for encrypting a block of text.
For the purposes of this example/tutorial,
it's **not strictly necessary**,
but it is included for "completeness"_.
#### Test the `encrypt/1` Function
Create a file called `test/lib/aes_test.exs` and _copy-paste_
the following code into it:
```elixir
defmodule Encryption.AESTest do
use ExUnit.Case
alias Encryption.AES
test ".encrypt includes the random IV in the value" do
<> = AES.encrypt("hello")
assert String.length(iv) != 0
assert String.length(ciphertext) != 0
assert is_binary(ciphertext)
end
test ".encrypt does not produce the same ciphertext twice" do
assert AES.encrypt("hello") != AES.encrypt("hello")
end
end
```
Run these two tests by running the following command:
```sh
mix test test/lib/aes_test.exs
```
> The full function definitions for AES `encrypt/1` & `encrypt/2` are in:
[`lib/encryption/aes.ex`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/lib/encryption/aes.ex)
> And tests are in:
[`test/lib/aes_test.exs`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/test/lib/aes_test.exs)
### 3.2 Decrypt
The `decrypt` function _reverses_ the work done by `encrypt`;
it accepts a "blob" of `ciphertext` (_which as you may recall_),
has the IV and cypher tag prepended to it, and returns the original `plaintext`.
In the `lib/encryption/aes.ex` file, copy-paste (_or hand-write_)
the following `decrypt/1` function definition:
```elixir
def decrypt(ciphertext) do
<> =
ciphertext
:crypto.crypto_one_time_aead(:aes_256_gcm, get_key(), iv, ciphertext, @aad, tag, false)
end
```
The fist step (line) is to "split" the IV from the `ciphertext`
using Elixir's binary pattern matching.
> If you are unfamiliar with Elixir binary pattern matching syntax:
`<>`
read the following guide:
https://elixir-lang.org/getting-started/binaries-strings-and-char-lists.html
The
`:crypto.crypto_one_time_aead(:aes_256_gcm, get_key(key_id), iv, ciphertext, @aad, tag, false)`
line is the _very similar_ to the `encrypt` function.
The `ciphertext` is decrypted using
[`block_decrypt/4`](https://erlang.org/doc/man/crypto.html#block_decrypt-4)
passing in the following parameters:
+ `:aes_256_gcm` = encyrption algorithm
+ `get_key(key_id)` = get the encryption key used to `encrypt` the `plaintext`
+ `iv` = the original Initialisation Vector used to `encrypt` the `plaintext`
+ `{@aad, ciphertext, tag}` = a Tuple with the encryption "mode",
`ciphertext` and the `tag` that was originally used to encrypt the `ciphertext`.
Finally return _just_ the original `plaintext`.
> _**Note**: as above with the_ `encrypt/2` _function,
we have defined an_ `decrypt/2` _"sister" function which accepts
a **specific** (encryption)_ `key_id` _so that we can use the desired
encryption key for decrypting the_ `ciphertext`.
_For the purposes of this example/tutorial,
it's **not strictly necessary**,
but it is included for "completeness"_.
#### Test the `decrypt/1` Function
In the `test/lib/aes_test.exs` add the following test:
```elixir
test "decrypt/1 ciphertext that was encrypted with default key" do
plaintext = "hello" |> AES.encrypt |> AES.decrypt()
assert plaintext == "hello"
end
```
Re-run the tests `mix test test/lib/aes_test.exs` and confirm they pass.
> The full `encrypt` & `decrypt` function definitions with `@doc` comments
are in:
[`lib/encryption/aes.ex`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/lib/encryption/aes.ex)
> And tests are in:
[`test/lib/aes_test.exs`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/test/lib/aes_test.exs)
### 3.3 Key rotation
Key rotation is a "**best practice**"
that **limits** the amount of data an "attacker" can decrypt if the database were ever "compromised"
_(provided we keep the encryption keys safe that is!)_ A really good guide to this is: https://cloud.google.com/kms/docs/key-rotation.
For this reason we want to 'store' a `key_id`. The `key_id` indicates which **encryption key** was used to encrypt the data. Besides the IV and ciphertag, the key_id is also _essential_ for allowing decryption, so we change the encrypt/1 function to preserve the key_id as well
```elixir
defmodule Encryption.AES do
@aad "AES256GCM" # Use AES 256 Bit Keys for Encryption.
def encrypt(plaintext) do
iv = :crypto.strong_rand_bytes(16)
# get latest key
key = get_key()
# get latest ID;
key_id = get_key_id()
# {ciphertext, tag} = :crypto.block_encrypt(:aes_gcm, key, iv, {@aad, plaintext, 16})
{ciphertext, tag} = :crypto.block_encrypt(:aes_gcm, key, iv, {@aad, to_string(plaintext), 16})
iv <> tag <> <> <> ciphertext
end
defp get_key do
get_key_id() |> get_key
end
defp get_key(key_id) do
encryption_keys() |> Enum.at(key_id)
end
defp get_key_id do
Enum.count(encryption_keys()) - 1
end
defp encryption_keys do
Application.get_env(:encryption, Encryption.AES)[:keys]
end
end
```
> _For the **complete** file containing these functions see_:
[`lib/encryption/aes.ex`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/lib/encryption/aes.ex)
For this example/demo we are using **two** encryption keys which are kept as an application environment variable. The values of the encryptions keys are associated with the key Encryption.AES. During the encryption we are by default always using the latest (most recent) encryption key (get_key/0) and the corresponding key_id is fetched by get_key_id/0 which becomes part of the ciphertext.
With decrypting we now pattern match the associated key_id from the ciphertext in order to be able to decrypt with the correct encryption key.
```elixir
def decrypt(ciphertext) do
<> =
ciphertext
:crypto.block_decrypt(:aes_gcm, get_key(key_id), iv, {@aad, ciphertext, tag})
end
```
So we defined the `get_key` _twice_ in `lib/encryption/aes.ex`
as per Erlang/Elixir standard,
once for each ["arity"](https://en.wikipedia.org/wiki/Arity)
or number of "arguments".
In the first case `get_key/0` _assumes_ you want the _latest_ Encryption Key.
The second case `get_key/1` lets you supply the `key_id` to be "looked up":
Both versions of `get_key` use encryption_keys/0 function to call the `Application.get_env` function:
`Application.get_env(:encryption, Encryption.AES)[:keys]` _specifically_.
For this to work we need to define the keys as an Environment Variable
and make it available to our App in `config.exs`.
### 3.4 `ENCRYPTION_KEYS` Environment Variable
In order for our `get_key/0` and `get_key/1` functions to _work_,
it needs to be able to "read" the encryption keys.
We need to "export" an Environment Variable
containing a (_comma-separated_) list of (_one or more_)
encryption key(s).
_Copy-paste_ (_and run_) the following command in your terminal:
```elixir
echo "export ENCRYPTION_KEYS='nMdayQpR0aoasLaq1g94FLba+A+wB44JLko47sVQXMg=,L+ZVX8iheoqgqb22mUpATmMDsvVGtafoAeb0KN5uWf0='" >> .env && echo ".env" >> .gitignore
```
> For _now_, copy paste this command exactly as it is.
> When you are deploying your own App,
> generate your own AES encryption key(s)
> see:
[How To Generate AES Encryption Keys?](https://github.com/dwyl/phoenix-ecto-encryption-example#how-to-generate-aes-encryption-keys)
> section below for how to do this.
> _**Note**: there are **two** encryption keys separated by a comma.
This is to **demonstrate** that it's **possible** to use **multiple keys**._
> _We prefer to store our Encryption Keys as
**Environment Variables** this is consistent with the "12 Factor App"
best practice:_ https://en.wikipedia.org/wiki/Twelve-Factor_App_methodology
Update the `config/config.exs` to load the environment variables from the `.env`
file into the application. Add the following code your config file just above
`import_config "#{Mix.env()}.exs"`:
```elixir
# run shell command to "source .env" to load the environment variables.
try do # wrap in "try do"
File.stream!("./.env") # in case .env file does not exist.
|> Stream.map(&String.trim_trailing/1) # remove excess whitespace
|> Enum.each(fn line -> line # loop through each line
|> String.replace("export ", "") # remove "export" from line
|> String.split("=", parts: 2) # split on *first* "=" (equals sign)
|> Enum.reduce(fn(value, key) -> # stackoverflow.com/q/33055834/1148249
System.put_env(key, value) # set each environment variable
end)
end)
rescue
_ -> IO.puts "no .env file found!"
end
# Set the Encryption Keys as an "Application Variable" accessible in aes.ex
config :encryption, Encryption.AES,
keys: System.get_env("ENCRYPTION_KEYS") # get the ENCRYPTION_KEYS env variable
|> String.replace("'", "") # remove single-quotes around key list in .env
|> String.split(",") # split the CSV list of keys
|> Enum.map(fn key -> :base64.decode(key) end) # decode the key.
```
#### Test the `get_key/0` and `get_key/1` Functions?
Given that `get_key/0` and `get_key/1` are _both_ `defp` (_i.e. "private"_)
they are not "exported" with the AES module and therefore cannot be _invoked_
outside of the AES module.
The `get_key/0` and `get_key/1` are _invoked_ by `encrypt/1` and `decrypt/1`
and thus provided these (public) latter functions
are tested adequately, the "private" functions will be too.
Re-run the tests `mix test test/lib/aes_test.exs` and confirm they _still_ pass.
We also define a test in order to verify the working of key rotation. We add a new encryption key and assert (and make sure) that an encrypted value with an older encryption key will still be decrypted correctly.
```elixir
test "can still decrypt the value after adding a new encryption key" do
encrypted_value = "hello" |> AES.encrypt()
original_keys = Application.get_env(:encryption, Encryption.AES)[:keys]
# add a new key
Application.put_env(:encryption, Encryption.AES,
keys: original_keys ++ [:crypto.strong_rand_bytes(32)]
)
assert "hello" == encrypted_value |> AES.decrypt()
# rollback to the original keys
Application.put_env(:encryption, Encryption.AES, keys: original_keys)
end
```
> The full `encrypt` & `decrypt` function definitions with `@doc` comments
are in:
[`lib/encryption/aes.ex`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/lib/encryption/aes.ex)
> And tests are in:
[`test/lib/aes_test.exs`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/test/lib/aes_test.exs)
## 4. Hash _Email Address_
The idea behind _hashing_ email addresses is to
allow us to perform a _lookup_ (_in the database_) to check if the
email has _already_ been registered/used for app/system.
Imagine that `[email protected]` has previously used your app.
The `SHA256` hash (_encoded as [base64](https://hexdocs.pm/elixir/Base.html)_)
is: `"bbYebcvPI5DkpGr0JvJqEzo77kUCFCL8euhukTbxQRA="`
`try` it for _yourself_ in `iex`:
```elixir
iex(1)> email = "[email protected]"
"[email protected]"
iex(2)> email_hash = :crypto.hash(:sha256, email) |> Base.encode64
"bbYebcvPI5DkpGr0JvJqEzo77kUCFCL8euhukTbxQRA="
```
If we store the `email_hash` in the database,
when `Alex` wants to _log-in_ to the App/System,
we simply perform a "lookup" in the `users` table:
```elixir
hash = :crypto.hash(:sha256, email) |> Base.encode64
query = "SELECT * FROM users WHERE email_hash = $1"
user = Ecto.Adapters.SQL.query!(Encryption.Repo, query, [hash])
```
> _**Note**: there's a "**built-in**" Ecto_
[`get_by`](https://hexdocs.pm/ecto/Ecto.Repo.html#c:get_by/3) _function
to perform this type of_
``"SELECT ... WHERE field = value"`` _query **effortlessly**_
### 4.1 Generate the `SECRET_KEY_BASE`
All Phoenix apps have a `secret_key_base` for sessions.
see: https://hexdocs.pm/plug/1.13.6/Plug.Session.COOKIE.html
Run the following command to generate a new phoenix secret key:
```sh
mix phx.gen.secret
```
_copy-paste_ the _output_ (64bit `String`)
into your `.env` file after the "equals sign" on the line for `SECRET_KEY_BASE`:
```yml
export SECRET_KEY_BASE={YourSecreteKeyBaseGeneratedUsing-mix_phx.gen.secret}
```
Your `.env` file should look _similar_ to:
[`.env_sample`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/.env_sample)
Load the secret key into your environment by typing into your terminal:
```sh
source .env
```
> _**Note**: We are using an_ `.env` _file,
but if you are using a "Cloud Platform" to deploy your app,
you could consider using their "Key Management Service"
for managing encryption keys. eg_:
+ Heroku:
https://github.com/dwyl/learn-environment-variables#environment-variables-on-heroku
+ AWS: https://aws.amazon.com/kms/
+ Google Cloud: https://cloud.google.com/kms/
We now need to update our config files again. Open your `config.exs` file and
change the the following:
***from***
```elixir
secret_key_base: "3PXN/6k6qoxqQjWFskGew4r74yp7oJ1UNF6wjvJSHjC5Y5LLIrDpWxrJ84UBphJn",
# your secret_key_base will be different but that is fine.
```
**To**
```elixir
secret_key_base: System.get_env("SECRET_KEY_BASE"),
```
As mentioned above, all Phoenix applications come with a `secret_key_base`.
Instead of using this default one, we have told our application to use the new
one that we added to our `.env` file.
Now we need to edit our `config/test.exs` file. Change the following:
***from***
```elixir
config :encryption, EncryptionWeb.Endpoint,
http: [port: 4001],
server: false
```
**To**
```elixir
config :encryption, EncryptionWeb.Endpoint,
http: [port: 4001],
server: false,
secret_key_base: System.get_env("SECRET_KEY_BASE")
```
By adding the previous code block we will now have a `secret_key_base` which
we will be able to use for testing.
## 5. _Create_ and use `HashField` Custom Ecto Type
When we first created the Ecto Schema for our "user", in
[Step 2](https://github.com/dwyl/phoenix-ecto-encryption-example#2-create-the-user-schema-database-table)
(_above_)
This created the
[`lib/encryption/user.ex`](https://raw.githubusercontent.com/dwyl/phoenix-ecto-encryption-example/36da851f30670967dd3493f055fb9f7b2649c188/lib/encryption/user.ex)
file with the following schema:
```elixir
schema "users" do
field :email, :binary
field :email_hash, :binary
field :name, :binary
field :password_hash, :binary
timestamps()
end
```
The _default_ Ecto field types (`:binary`) are a good start.
But we can do _so much_ better if we define _custom_ Ecto Types!
Ecto Custom Types are a way of automatically "_pre-processing_" data
before inserting it into (_and reading from_) a database.
Examples of "pre-processing" include:
+ Custom Validation e.g: phone number or address format.
+ Encrypting / Decrypting
+ Hashing
A custom type expects [6 callback functions](https://hexdocs.pm/ecto/Ecto.Type.html#callbacks)
to be implemented in the file:
+ [`type/0`](https://hexdocs.pm/ecto/Ecto.Type.html#c:type/0) - define
the Ecto Type we want Ecto to use to _store_ the data
for our Custom Type. e.g: `:integer` or `:binary`
+ [`cast/1`](https://hexdocs.pm/ecto/Ecto.Type.html#c:cast/1) - "typecasts" (_converts_)
the given data to the desired type e.g: Integer to String.
+ [`dump/1`](https://hexdocs.pm/ecto/Ecto.Type.html#c:dump/1) - performs the "processing"
on the raw data before it get's "dumped" into the Ecto Native Type.
+ [`load/1`](https://hexdocs.pm/ecto/Ecto.Type.html#c:load/1) - called when
loading data from the database and receive an Ecto native type.
+ [`embed_as/1`](https://hexdocs.pm/ecto/Ecto.Type.html#c:embed_as/1) - the return value
(`:self` or `:dump`) determines how the type is treated inside embeds (not used here).
+ [`equal?/2`](https://hexdocs.pm/ecto/Ecto.Type.html#c:equal?/2) - invoked to determine
if changing a type's field value changes the corresponding database record.
Create a file called `lib/encryption/hash_field.ex` and add the following:
```elixir
defmodule Encryption.HashField do
@behaviour Ecto.Type
def type, do: :binary
def cast(value) do
{:ok, to_string(value)}
end
def dump(value) do
{:ok, hash(value)}
end
def load(value) do
{:ok, value}
end
def embed_as(_), do: :self
def equal?(value1, value2), do: value1 == value2
def hash(value) do
:crypto.hash(:sha256, value <> get_salt(value))
end
# Get/use Phoenix secret_key_base as "salt" for one-way hashing Email address
# use the *value* to create a *unique* "salt" for each value that is hashed:
defp get_salt(value) do
secret_key_base =
Application.get_env(:encryption, EncryptionWeb.Endpoint)[:secret_key_base]
:crypto.hash(:sha256, value <> secret_key_base)
end
end
```
Let's step through each of these
### `type/0`
The best data type for storing encrypted data is `:binary`
(_it uses **half** the "space" of a `:string` for the **same** ciphertext_).
### `cast/1`
Cast any data type `to_string` before encrypting it.
(_the encrypted data "ciphertext" will be of_ `:binary` _type_)
### `dump/1`
The `hash/1` function use Erlang's `crypto` library
[`hash/2`](https://erlang.org/doc/man/crypto.html#hash-2) function.
+ First we tell the `hash/2` function that we want to use `:sha256`
"SHA 256" is the most widely used/recommended hash; it's both fast and "secure".
+ We then hash the `value` passed in to the `hash/1` function (_we defined_)
and _concatenate_ it with "salt" using the `get_salt/1` function
which retrieves the `secret_key_base` environment variable
and computes a ***unique*** "salt" using the value.
We use the `SHA256` one-way hash for _speed_.
We "salt" the email address so that
the hash has _some_ level of "obfuscation",
in case the DB is ever "compromised"
the "attacker" still has to "compute"
a ["rainbow table"](https://en.wikipedia.org/wiki/Rainbow_table) from _scratch_.
### `load/1`
Return the hash value as it is _read_ from the database.
### `embed_as/1`
This callback is only of importance when the type is part of an [embed](https://hexdocs.pm/ecto/Ecto.Changeset.html#module-associations-embeds-and-on-replace). It's not used here,
but required for modules adopting the `Ecto.Type` behaviour as of Ecto 3.2.
### `equal?/2`
This callback is invoked when we cast changes into a changeset and want to
determine whether the database record needs to be updated. We use a simple
equality comparison (`==`) to compare the current value to the requested
update. If both values are equal, there's no need to update the record.
> _**Note**: Don't forget to export your_ `SECRET_KEY_BASE`
_environment variable_ (_see instructions above_)
> The full file containing these two functions is:
[`lib/encryption/hash_field.ex`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/lib/encryption/hash_field.ex)
> And the tests for the functions are:
[`test/lib/hash_field_test.exs`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/test/lib/hash_field_test.exs)
_First_ add the `alias` for `HashField` near the top
of the `lib/encryption/user.ex` file. e.g:
```elixir
alias Encryption.HashField
```
_Next_, in the `lib/encryption/user.ex` file,
***update*** the lines for `email_hash` in the users schema
***from***:
```elixir
schema "users" do
field :email, :binary
field :email_hash, :binary
field :name, :binary
field :password_hash, :binary
timestamps()
end
```
**To**:
```elixir
schema "users" do
field :email, :binary
field :email_hash, HashField
field :name, :binary
field :password_hash, :binary
timestamps()
end
```
```Elixir
def changeset(%User{} = user, attrs \\ %{}) do
user
|> cast(attrs, [:name, :email])
|> validate_required([:email])
|> add_email_hash
|> unique_constraint(:email_hash)
end
defp add_email_hash(changeset) do
if Map.has_key?(changeset.changes, :email) do
changeset |> put_change(:email_hash, changeset.changes.email)
else
changeset
end
end
```
We should _test_ this new functionality. Create the file
`test/lib/user_test.exs` and add the following:
```elixir
defmodule Encryption.UserTest do
use Encryption.DataCase
alias Encryption.User
@valid_attrs %{
name: "Max",
email: "[email protected]",
password: "NoCarbsBeforeMarbs"
}
@invalid_attrs %{}
describe "Verify correct working of hashing" do
setup do
user = Repo.insert!(User.changeset(%User{}, @valid_attrs))
{:ok, user: user, email: @valid_attrs.email}
end
test "inserting a user sets the :email_hash field", %{user: user} do
assert user.email_hash == user.email
end
test ":email_hash field is the encrypted hash of the email", %{user: user} do
user_from_db = User |> Repo.one()
assert user_from_db.email_hash == Encryption.HashField.hash(user.email)
end
end
end
```
For the _full_ user tests please see:
[`test/user/user_test.exs`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/test/user/user_test.exs)
## 6. Create and user Hash _Password_ Custom Ecto type
When hashing **passwords**, we want to use the **_strongest_ hashing algorithm**
and we also want the hashed value (_or "digest"_) to be ***different***
each time the _same_ `plaintext` is hashed
(_unlike when hashing the email address
where we want a deterministic digest_).
Using `argon2` makes "cracking" a password
(_in the event of the database being "compromised")
far less likely_ as it uses _both_ a CPU-bound "work-factor"
_and_ a "Memory-hard" algorithm which will _significantly_
"slow down" the attacker.
### Add the `argon2` Dependency
In order to use `argon2` we must add it to our `mix.exs` file:
in the `defp deps do` (_dependencies_) section, add the following line:
```elixir
{:argon2_elixir, "~> 1.3"}, # securely hashing & verifying passwords
```
You will need to run `mix deps.get` to install the dependency.
### 6.1 Define the `hash_password/1` Function
Create a file called `lib/encryption/password_field.ex` in your project.
The first function we need is `hash_password/1`:
```elixir
defmodule Encryption.PasswordField do
def hash_password(value) do
Argon2.Base.hash_password(to_string(value),
Argon2.Base.gen_salt(), [{:argon2_type, 2}])
end
end
```
`hash_password/1` accepts a password to be hashed and invokes
[`Argon2.Base.hash_password/3`](https://hexdocs.pm/argon2_elixir/Argon2.Base.html#hash_password/3)
passing in 3 arguments:
+ `value` - the value (_password_) to be hashed.
+ [`Argon2.Base.gen_salt/1`](https://hexdocs.pm/argon2_elixir/Argon2.Base.html#gen_salt/1) - the salt used to initialise the hash function
note: "behind the scenes" just
[`:crypto.strong_rand_bytes(16)`](https://github.com/riverrun/argon2_elixir/blob/d283a4f316a2a26e61f032a826ff992a480018c2/lib/argon2.ex#L76)
as we saw before in the `encrypt` function; again,
**128 bits** is considered "secure" as a hash salt or initialization vector.
+ `[{:argon2_type, 2}]` - this corresponds to `argon2id` see:
+ https://github.com/riverrun/argon2_elixir/issues/17
+ https://crypto.stackexchange.com/questions/48935/why-use-argon2i-or-argon2d-if-argon2id-exists
#### 6.1.1 Test the `hash_password/1` Function?
In order to _test_ the `PasswordField.hash_password/1` function
we use the `Argon2.verify_pass` function to _verify_ a password hash.
Create a file called `test/lib/password_field_test.exs`
and _copy-paste_ (_or hand-type_) the following test:
```elixir
defmodule Encryption.PasswordFieldTest do
use ExUnit.Case
alias Encryption.PasswordField, as: Field
test ".verify_password checks the password against the Argon2id Hash" do
password = "EverythingisAwesome"
hash = Field.hash_password(password)
verified = Argon2.verify_pass(password, hash)
assert verified
end
end
```
Run the test using the command:
```sh
mix test test/lib/password_field_test.exs
```
The test should _pass_;
if _not_, please re-trace the steps.
### 6.2 _Verify_ Password
The _corresponding_ function to _check_ (_or "verify"_)
the password is `verify_password/2`.
We need to supply both the `password` and `stored_hash`
(_the hash that was previously stored in the database
when the person registered or updated their password_)
It then runs `Argon2.verify_pass` which does the checking.
```elixir
def verify_password(password, stored_hash) do
Argon2.verify_pass(password, stored_hash)
end
```
`hash_password/1` and `verify_password/2` functions are defined in:
[`lib/encryption/password_field.ex`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/lib/encryption/password_field.ex)
##### Test for `verify_password/2`
To test that our `verify_password/2` function works as _expected_,
open the file: `test/lib/password_field_test.exs`
and add the following code to it:
```elixir
test ".verify_password fails if password does NOT match hash" do
password = "EverythingisAwesome"
hash = Field.hash_password(password)
verified = Field.verify_password("LordBusiness", hash)
assert !verified
end
```
Run the tests: `mix test test/lib/password_field_test.exs`
and confirm they pass.
If you get stuck, see:
[`test/lib/password_field_test.exs`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/test/lib/password_field_test.exs)
Define the other Ecto.Type behaviour functions:
```Elixir
defmodule Encryption.PasswordField do
@behaviour Ecto.Type
def type, do: :binary
def cast(value) do
{:ok, to_string(value)}
end
def dump(value) do
{:ok, hash_password(value)}
end
def load(value) do
{:ok, value}
end
def embed_as(_), do: :self
def equal?(value1, value2), do: value1 == value2
def hash_password(value) do
Argon2.Base.hash_password(to_string(value),
Argon2.Base.gen_salt(), [{:argon2_type, 2}])
end
def verify_password(password, stored_hash) do
Argon2.verify_pass(password, stored_hash)
end
end
```
```elixir
alias Encryption.{HashField, PasswordField, User}
```
Update the lines for `:email` and `:name` in the schema
***from***:
```elixir
schema "users" do
field :email, :binary
field :email_hash, HashField
field :name, :binary
field :password_hash, :binary
timestamps()
end
```
**To**:
```elixir
schema "users" do
field :email, :binary
field :email_hash, HashField
field :name, :binary
field :password_hash, PasswordField
timestamps()
end
```
## 7. _Create_ and use `EncryptedField` Custom Ecto Type
Create a file called `lib/encryption/encrypted_field.ex` and add the following:
```elixir
defmodule Encryption.EncryptedField do
alias Encryption.AES # alias our AES encrypt & decrypt functions (3.1 & 3.2)
@behaviour Ecto.Type # Check this module conforms to Ecto.type behavior.
def type, do: :binary # :binary is the data type ecto uses internally
# cast/1 simply calls to_string on the value and returns a "success" tuple
def cast(value) do
{:ok, to_string(value)}
end
# dump/1 is called when the field value is about to be written to the database
def dump(value) do
ciphertext = value |> to_string |> AES.encrypt
{:ok, ciphertext} # ciphertext is :binary data
end
# load/1 is called when the field is loaded from the database
def load(value) do
{:ok, AES.decrypt(value)} # decrypted data is :string type.
end
# embed_as/1 dictates how the type behaves when embedded (:self or :dump)
def embed_as(_), do: :self # preserve the type's higher level representation
# equal?/2 is called to determine if two field values are semantically equal
def equal?(value1, value2), do: value1 == value2
end
```
Let's step through each of these
### `type/0`
The best data type for storing encrypted data is `:binary`
(_it uses **half** the "space" of a `:string` for the **same** ciphertext_).
### `cast/1`
Cast any data type `to_string` before encrypting it.
(_the encrypted data "ciphertext" will be of_ `:binary` _type_)
### `dump/1`
Calls the `AES.encrypt/1` function we defined in section 3.1 (_above_)
so data is _encrypted_ 'automatically' before we insert into the database.
### `load/1`
Calls the `AES.decrypt/1` function so data is 'automatically' _decrypted_ when it is _read_
from the database.
> _**Note**: the_ `load/2` _function is **not required**
for Ecto Type compliance.
Further reading_: https://hexdocs.pm/ecto/Ecto.Type.html
### `embed_as/1`
This callback is only of importance when the type is part of an
[embed](https://hexdocs.pm/ecto/Ecto.Changeset.html#module-associations-embeds-and-on-replace).
It's not used here,
but required for modules adopting the `Ecto.Type` behaviour as of Ecto 3.2.
### `equal?/2`
This callback is invoked when we cast changes into a changeset and want to
determine whether the database record needs to be updated. We use a simple
equality comparison (`==`) to compare the current value to the requested
update. If both values are equal, there's no need to update the record.
_Your_ `encrypted_field.ex` Custom Ecto Type should look like this:
[`lib/encryption/encrypted_field.ex`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/lib/encryption/encrypted_field.ex)
`try` to write the **tests** for the callback functions,
if you get "stuck", take a look at:
[`test/lib/encrypted_field_test.exs`](https://github.com/dwyl/phoenix-ecto-encryption-example/blob/master/test/lib/encrypted_field_test.exs)
Now that we have defined a Custom Ecto Type `EncryptedField`,
we can _use_ the Type in our User Schema.
Add the following line to "alias" the Type and a User
in the `lib/encryption/user.ex` file:
```elixir
alias Encryption.{HashField, PasswordField, EncryptedField, User}
```
Update the lines for `:email` and `:name` in the schema
***from***:
```elixir
schema "users" do
field :email, :binary
field :email_hash, HashField
field :name, :binary
field :password_hash, PasswordField
timestamps()
end
```
**To**:
```elixir
schema "users" do
field :email, EncryptedField
field :email_hash, HashField
field :name, EncryptedField
field :password_hash, PasswordField
timestamps()
end
```
## 8. Ensure All Tests Pass
Typically we will create `git commit` (_if we don't already have one_)
for the "known state" where the tests were passing
(_before starting the refactor_).
The commit _before_ refactoring the example is:
https://github.com/dwyl/phoenix-ecto-encryption-example/tree/3659399ec32ca4f07f45d0552b9cf25c359a2456
The corresponding Travis-CI build for this commit is:
https://travis-ci.org/dwyl/phoenix-ecto-encryption-example/jobs/379887597#L833
> _**Note**: if you are_ `new` _to Travis-CI see_:
[https://github.com/dwyl/**learn-travis**](https://github.com/dwyl/learn-travis)
# Conclusion
We have gone through how to create custom Ecto Types
in order to define our own functions for handling
(_transforming_) specific types of data.
Our hope is that you have _understood_ the flow.
We plan to extend this tutorial include User Interface
please "star" the repo if you would find that useful.
## How To Generate AES Encryption Keys?
Encryption keys should be the appropriate length (in bits)
as required by the chosen algorithm.
> An **AES 128-bit** key can be expressed
as a hexadecimal string with 32 characters.
It will require **24 characters** in **base64**.
> An **AES 256-bit** key can be expressed
as a hexadecimal string with 64 characters.
It will require **44 characters** in **base64**.
see: https://security.stackexchange.com/a/45334/117318
Open `iex` in your Terminal and paste the following line (_then press enter_)
```elixir
:crypto.strong_rand_bytes(32) |> :base64.encode
```
You should see terminal output similar to the following:
We generated 3 keys for demonstration purposes:
+ "h6pUk0ZccS0pYsibHZZ4Cd+PRO339rMA7sMz7FnmcGs="
+ "nMd/yQpR0aoasLaq1g94FL/a+A+wB44JLko47sVQXMg="
+ "L+ZVX8iheoqgqb22mUpATmMDsvVGt/foAe/0KN5uWf0="
These two Erlang functions are described in:
+ https://erlang.org/doc/man/crypto.html#strong_rand_bytes-1
+ https://erlang.org/doc/man/base64.html#encode-1
Base64 encoding the bytes generated by `strong_rand_bytes`
will make the output human-readable
(_whereas bytes are less user-friendly_).
# Useful Links, FAQ & Background Reading
+ Bits and Bytes: https://web.stanford.edu/class/cs101/bits-bytes.html
+ Thinking in Ecto - Schemas and Changesets:
https://cultofmetatron.io/2017/04/22/thinking-in-ecto---schemas-and-changesets/
+ Initialization Vector Length:
https://stackoverflow.com/questions/4608489/how-to-pick-an-appropriate-iv-initialization-vector-for-aes-ctr-nopadding (128 bits is 16 bytes).
+ What is the effect of the different AES key lengths?
https://crypto.stackexchange.com/questions/3615/what-is-the-effect-of-the-different-aes-key-lengths
+ How is decryption done in AES CTR mode?: https://crypto.stackexchange.com/questions/34918/how-is-decryption-done-in-aes-ctr-mode
+ Block Cipher Counter (CTR) Mode:
https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_.28CTR.29
+ Is AES-256 _weaker_ than 192 and 128 bit versions?
https://crypto.stackexchange.com/questions/5118/is-aes-256-weaker-than-192-and-128-bit-versions
+ What are the practical differences between 256-bit, 192-bit, and 128-bit AES encryption?
https://crypto.stackexchange.com/questions/20/what-are-the-practical-differences-between-256-bit-192-bit-and-128-bit-aes-enc
+ **How to Choose** an **Authenticated Encryption mode**
(_by Matthew Green cryptography professor at Johns Hopkins University_):
https://blog.cryptographyengineering.com/2012/05/19/how-to-choose-authenticated-encryption
+ How to choose an AES encryption mode (CBC ECB CTR OCB CFB)? (v. long answers, but good comparison!)
https://stackoverflow.com/questions/1220751/how-to-choose-an-aes-encryption-mode-cbc-ecb-ctr-ocb-cfb
+ AES GCM vs CTR+HMAC tradeoffs:
https://crypto.stackexchange.com/questions/14747/gcm-vs-ctrhmac-tradeoffs
+ Galois/Counter Mode for symmetric key cryptographic block ciphers:
https://en.wikipedia.org/wiki/Galois/Counter_Mode
+ What is the difference between CBC and GCM mode?
https://crypto.stackexchange.com/questions/2310/what-is-the-difference-between-cbc-and-gcm-mode
+ Ciphertext and tag size and IV transmission with AES in GCM mode:
https://crypto.stackexchange.com/questions/26783/ciphertext-and-tag-size-and-iv-transmission-with-aes-in-gcm-mode
+ How long (in letters) are encryption keys for AES?
https://security.stackexchange.com/questions/45318/how-long-in-letters-are-encryption-keys-for-aes
+ Why we can't implement AES 512 key size?
https://crypto.stackexchange.com/questions/20253/why-we-cant-implement-aes-512-key-size
+ Generate random alphanumeric string (_used for AES keys_)
https://stackoverflow.com/questions/12788799/how-to-generate-a-random-alphanumeric-string-with-erlang
+ Singular or Plural controller names?: https://stackoverflow.com/questions/35882394/phoenix-controllers-singular-or-plural
+ What's the purpose of key-rotation?
https://crypto.stackexchange.com/questions/41796/whats-the-purpose-of-key-rotation
+ Postgres Data Type for storing `bcrypt` hashed passwords: https://stackoverflow.com/questions/33944199/bcrypt-and-postgresql-what-data-type-should-be-used >> `bytea` (_byte_)
+ Do security experts recommend bcrypt? https://security.stackexchange.com/questions/4781/do-any-security-experts-recommend-bcrypt-for-password-storage/6415#6415
+ Hacker News discussion thread "***Don't use `bcrypt`***":
https://news.ycombinator.com/item?id=3724560
+ Storing Passwords in a Highly Parallelized World:
https://hynek.me/articles/storing-passwords
+ Password hashing security of argon2 versus bcrypt/PBKDF2?
https://crypto.stackexchange.com/questions/30785/password-hashing-security-of-argon2-versus-bcrypt-pbkdf2
+ The memory-hard Argon2 password hash function (ietf proposal):
https://tools.ietf.org/id/draft-irtf-cfrg-argon2-03.html
unlikely to be a "standard" any time soon...
+ Erlang Dirty Scheduler Overhead:
https://medium.com/@jlouis666/erlang-dirty-scheduler-overhead-6e1219dcc7
+ Erlang Scheduler Details and Why They Matter:
https://news.ycombinator.com/item?id=11064763
+ Why use argon2i or argon2d if argon2id exists?
https://crypto.stackexchange.com/questions/48935/why-use-argon2i-or-argon2d-if-argon2id-exists
+ Good explanation of _Custom_ Ecto Types:
https://medium.com/acutario/ecto-custom-types-a-practical-case-with-enumerize-rails-gem-b5496c2912ac
+ Consider using ETS to store encryption/decryption keys:
https://elixir-lang.org/getting-started/mix-otp/ets.html &
https://elixirschool.com/en/lessons/specifics/ets
## Understanding Advanced Encryption Standard (AES)
If you prefer to _read_,
Ryo Nakao wrote an excellent post on
understanding how AES encryption works:
https://nakabonne.dev/posts/understanding-how-aes-encryption-works/
If you have the bandwidth and prefer a video,
Computerphile (YouTube channel)
has an great explaination:
[](https://www.youtube.com/watch?v=O4xNJsjtN6E)
## Running a Single Test
To run a _single_ test (_e.g: while debugging_), use the following syntax:
```sh
mix test test/user/user_test.exs:9
```
For more detail, please see: https://hexdocs.pm/phoenix/testing.html
## Ecto Validation Error format
When Ecto `changeset` validation fails,
for example if there is a "unique" constraint on email address
(_so that people cannot re-register with the same email address twice_),
Ecto returns the `changeset` with an `errors` key:
```elixir
#Ecto.Changeset<
action: :insert,
changes: %{
email: <<224, 124, 228, 125, 105, 102, 38, 170, 15, 199, 228, 198, 245, 189,
82, 193, 164, 14, 182, 8, 189, 19, 231, 49, 80, 223, 84, 143, 232, 92, 96,
156, 100, 4, 7, 162, 26, 2, 121, 32, 187, 65, 254, 50, 253, 101, 202>>,
email_hash: <<21, 173, 0, 16, 69, 67, 184, 120, 1, 57, 56, 254, 167, 254,
154, 78, 221, 136, 159, 193, 162, 130, 220, 43, 126, 49, 176, 236, 140,
131, 133, 130>>,
key_id: 1,
name: <<2, 215, 188, 71, 109, 131, 60, 147, 219, 168, 106, 157, 224, 120,
49, 224, 225, 181, 245, 237, 23, 68, 102, 133, 85, 62, 22, 166, 105, 51,
239, 198, 107, 247, 32>>,
password_hash: <<132, 220, 9, 85, 60, 135, 183, 155, 214, 215, 156, 180,
205, 103, 189, 137, 81, 201, 37, 214, 154, 204, 185, 253, 144, 74, 222,
80, 158, 33, 173, 254>>
},
errors: [email_hash: {"has already been taken", []}],
data: #Encryption.User<>,
valid?: false
>
```
The `errors` part is:
```elixir
[email_hash: {"has already been taken", []}]
```
A `tuple` _wrapped_ in a `keyword list`.
Why this construct? A changeset can have multiple errors, so they're stored as a keyword list, where the _key_ is the field, and the _value_ is the error tuple.
The first item in the tuple is the error message, and the second is another keyword list, with additional information that we would use when mapping over the errors in order to make them more user-friendly (though here, it's empty).
See the Ecto docs for [`add_error/4`](https://hexdocs.pm/ecto/Ecto.Changeset.html#add_error/4) and [`traverse_errors/2`](https://hexdocs.pm/ecto/Ecto.Changeset.html#traverse_errors/2) for more information.
So to _access_ the error message `"has already been taken"`
we need some pattern-matching and list popping:
```elixir
{:error, changeset} = Repo.insert User.changeset(%User{}, @valid_attrs)
{:ok, message} = Keyword.fetch(changeset.errors, :email_hash)
msg = List.first(Tuple.to_list(message))
assert "has already been taken" == msg
```
To see this in _action_ run:
```sh
mix test test/user/user_test.exs:40
```
# Stuck / Need Help?
If _you_ get "stuck", please open an issue on GitHub:
https://github.com/nelsonic/phoenix-ecto-encryption-example/issues
describing the issue you are facing with as much detail as you can.
# Credits
Inspiration/credit/thanks for this example goes to **Daniel Berkompas**
[@danielberkompas](https://github.com/danielberkompas)
for his post:
https://blog.danielberkompas.com/2015/07/03/encrypting-data-with-ecto
Daniel's post is for
[Phoenix `v0.14.0`](https://github.com/danielberkompas/danielberkompas.github.io/blob/c6eb249e5019e782e891bfeb591bc75f084fd97c/_posts/2015-07-03-encrypting-data-with-ecto.md)
which is quite "old" now ...
therefore a few changes/updates are required.
e.g: There are no more "**Models**" in Phoenix 1.3 or Ecto callbacks.
_Also_ his post only includes the "sample code"
and is _not_ a _complete_ example
and does _not_ _explain_ the functions & Custom Ecto Types.
Which means anyone following the post needs
to _manually_ copy-paste the code ...
and "figure out" the "gaps" themselves to make it work.
We _prefer_ to include the _complete_ "**end state**"
of any tutorial (_not just "samples"_)
so that _anyone_ can `git clone` and _`run`_
the code locally to _fully understand_ it.
Still, props to Daniel for his post, a _good intro_ to the topic!