{"id":13468581,"url":"https://github.com/incuna/django-pgcrypto-fields","last_synced_at":"2025-05-15T07:03:30.026Z","repository":{"id":22256513,"uuid":"25590252","full_name":"incuna/django-pgcrypto-fields","owner":"incuna","description":"Transparent field level encryption for Django using the pgcrypto postgresql extension.","archived":false,"fork":false,"pushed_at":"2024-12-07T02:46:42.000Z","size":256,"stargazers_count":233,"open_issues_count":67,"forks_count":52,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-05-15T07:02:13.477Z","etag":null,"topics":["django","pgcrypto","psql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/incuna.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2014-10-22T15:51:55.000Z","updated_at":"2025-05-15T05:51:27.000Z","dependencies_parsed_at":"2023-10-05T04:13:59.094Z","dependency_job_id":"999a3fcf-a739-4a0f-80d8-c54008e17f26","html_url":"https://github.com/incuna/django-pgcrypto-fields","commit_stats":{"total_commits":286,"total_committers":11,"mean_commits":26.0,"dds":0.5629370629370629,"last_synced_commit":"68d60e173e2291e513f84f227fc40cc4c29c9559"},"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/incuna%2Fdjango-pgcrypto-fields","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/incuna%2Fdjango-pgcrypto-fields/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/incuna%2Fdjango-pgcrypto-fields/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/incuna%2Fdjango-pgcrypto-fields/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/incuna","download_url":"https://codeload.github.com/incuna/django-pgcrypto-fields/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254291961,"owners_count":22046424,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["django","pgcrypto","psql"],"created_at":"2024-07-31T15:01:14.053Z","updated_at":"2025-05-15T07:03:29.996Z","avatar_url":"https://github.com/incuna.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# django-pgcrypto-fields\n\n[![Latest Release](https://img.shields.io/pypi/v/django-pgcrypto-fields.svg)](https://pypi.org/pypi/django-pgcrypto-fields/) [![Python Versions](https://img.shields.io/pypi/pyversions/django-pgcrypto-fields.svg)](https://pypi.org/pypi/django-pgcrypto-fields/) [![Build Status](https://travis-ci.org/incuna/django-pgcrypto-fields.svg?branch=master)](https://travis-ci.org/incuna/django-pgcrypto-fields?branch=master) [![Requirements Status](https://requires.io/github/incuna/django-pgcrypto-fields/requirements.svg?branch=master)](https://requires.io/github/incuna/django-pgcrypto-fields/requirements/?branch=master) [![Updates](https://pyup.io/repos/github/incuna/django-pgcrypto-fields/shield.svg)](https://pyup.io/repos/github/incuna/django-pgcrypto-fields/) [![Coverage Status](https://coveralls.io/repos/github/incuna/django-pgcrypto-fields/badge.svg?branch=master)](https://coveralls.io/github/incuna/django-pgcrypto-fields?branch=master)\n\n`django-pgcrypto-fields` is a `Django` extension which relies upon `pgcrypto` to\nencrypt and decrypt data for fields.\n\n## Requirements\n\n - postgres with `pgcrypto`\n - Supports Django 2.2.x, 3.0.x, 3.1.x and 3.2.x\n - Compatible with Python 3 only\n \n Last version of this library that supports `Django` 1.8.x, 1.9.x, 1.10.x\n was `django-pgcrypto-fields` 2.2.0.\n \n Last version of this library that supports `Django` 2.0.x and 2.1.x was\n was `django-pgcrypto-fields` 2.5.2.\n \n\n## Installation\n\n### Install package \n\n```bash\npip install django-pgcrypto-fields\n```\n\n### Django settings\n\nOur library support different crypto keys for multiple databases by \ndefining the keys in your `DATABASES` settings.\n\nIn `settings.py`:\n```python\nimport os\nBASEDIR = os.path.dirname(os.path.dirname(__file__))\nPUBLIC_PGP_KEY_PATH = os.path.abspath(os.path.join(BASEDIR, 'public.key'))\nPRIVATE_PGP_KEY_PATH = os.path.abspath(os.path.join(BASEDIR, 'private.key'))\n\n# Used by PGPPublicKeyField used by default if not specified by the db\nPUBLIC_PGP_KEY = open(PUBLIC_PGP_KEY_PATH).read()\nPRIVATE_PGP_KEY = open(PRIVATE_PGP_KEY_PATH).read()\n\n# Used by TextHMACField and PGPSymmetricKeyField if not specified by the db\nPGCRYPTO_KEY='ultrasecret'\n\nDIFF_PUBLIC_PGP_KEY_PATH = os.path.abspath(\n    os.path.join(BASEDIR, 'tests/keys/public_diff.key')\n)\nDIFF_PRIVATE_PGP_KEY_PATH = os.path.abspath(\n    os.path.join(BASEDIR, 'tests/keys/private_diff.key')\n)\n\n# And add 'pgcrypto' to `INSTALLED_APPS` to create the extension for\n# pgcrypto (in a migration).\nINSTALLED_APPS = (\n    'pgcrypto',\n    # Other installed apps\n)\n\nDATABASES = {\n    # This db will use the default keys above\n    'default': {\n        'ENGINE': 'django.db.backends.postgresql_psycopg2',\n        'NAME': 'pgcryto_fields',\n        'USER': 'pgcryto_fields',\n        'PASSWORD': 'xxxx',\n        'HOST': 'psql.test.com',\n        'PORT': 5432,\n        'OPTIONS': {\n            'sslmode': 'require',\n        }\n    },\n    'diff_keys': {\n        'ENGINE': 'django.db.backends.postgresql_psycopg2',\n        'NAME': 'pgcryto_fields_diff',\n        'USER': 'pgcryto_fields_diff',\n        'PASSWORD': 'xxxx',\n        'HOST': 'psqldiff.test.com',\n        'PORT': 5432,\n        'OPTIONS': {\n            'sslmode': 'require',\n        },\n        'PGCRYPTO_KEY': 'djangorocks',\n        'PUBLIC_PGP_KEY': open(DIFF_PUBLIC_PGP_KEY_PATH, 'r').read(),\n        'PRIVATE_PGP_KEY': open(DIFF_PRIVATE_PGP_KEY_PATH, 'r').read(),\n    },\n}\n```\n\n### Generate GPG keys if using Public Key Encryption\n\nThe public key is going to encrypt the message and the private key will be\nneeded to decrypt the content. The following commands have been taken from the\n[pgcrypto documentation](http://www.postgresql.org/docs/devel/static/pgcrypto.html)\n(see Generating PGP Keys with GnuPG).\n\nGenerating a public and a private key (The preferred key type is \"DSA and Elgamal\".):\n\n```bash\n$ gpg --gen-key\n$ gpg --list-secret-keys\n\n/home/bob/.gnupg/secring.gpg\n---------------------------\nsec   2048R/21 2014-10-23\nuid                  Test Key \u003cexample@example.com\u003e\nssb   2048R/42 2014-10-23\n\n\n$ gpg -a --export 42 \u003e public.key\n$ gpg -a --export-secret-keys 21 \u003e private.key\n```\n\n#### Limitations\n\nThis library currently does not support Public Key Encryption private keys that are password protected yet. See Issue #89 to help implement it.\n\n### Upgrading to 2.4.0 from previous versions\n\nThe 2.4.0 version of this library received a large rewrite in order to support \nauto-decryption when getting encrypted field data as well as the ability to filter \non encrypted fields without using the old PGPCrypto aggregate functions available\nin previous versions.\n\nThe following items in this library have been removed and therefore references in \nyour application to these items need to be removed as well:\n\n* `managers.PGPManager`\n* `admin.PGPAdmin`\n* `aggregates.*`\n\n## Fields\n\n`django-pgcrypto-fields` has 3 kinds of fields:\n  - Hash based fields\n  - Public Key (PGP) fields\n  - Symmetric fields\n\n#### Hash Based Fields\n\nSupported hash based fields are:\n - `TextDigestField`\n - `TextHMACField`\n\n`TextDigestField` is hashed in the database using the `digest` pgcrypto function \nusing the `sha512` algorithm.\n\n`TextHMACField` is hashed in the database using the `hmac` pgcrypto function \nusing a key and the `sha512` algorithm. This is similar to the digest version however\nthe hash can only be recalculated knowing the key. This prevents someone from altering \nthe data and also changing the hash to match.\n\n#### Public Key Encryption Fields\n\nSupported PGP public key fields are:\n - `CharPGPPublicKeyField`\n - `EmailPGPPublicKeyField`\n - `TextPGPPublicKeyField`\n - `DatePGPPublicKeyField`\n - `DateTimePGPPublicKeyField`\n - `TimePGPPublicKeyField`\n - `IntegerPGPPublicKeyField`\n - `BigIntegerPGPPublicKeyField`\n - `DecimalPGPPublicKeyField`\n - `FloatPGPPublicKeyField`\n - `BooleanPGPPublicKeyField`\n\nPublic key encryption creates a token generated with a public key to\nencrypt the data and a private key to decrypt it.\n\nPublic and private keys can be set in settings with `PUBLIC_PGP_KEY` and\n`PRIVATE_PGP_KEY`.\n\n#### Symmetric Key Encryption Fields\n\nSupported PGP symmetric key fields are:\n - `CharPGPSymmetricKeyField`\n - `EmailPGPSymmetricKeyField`\n - `TextPGPSymmetricKeyField`\n - `DatePGPSymmetricKeyField`\n - `DateTimePGPSymmetricKeyField`\n - `TimePGPSymmetricKeyField`\n - `IntegerPGPSymmetricKeyField`\n - `BigIntegerPGPSymmetricKeyField`\n - `DecimalPGPSymmetricKeyField`\n - `FloatPGPSymmetricKeyField`\n - `BooleanPGPSymmetricKeyField`\n\n\nEncrypt and decrypt the data with `settings.PGCRYPTO_KEY` which acts like a password.\n\n### Django Model Field Equivalents \n\n| Django Field    | Public Key Field            | Symmetric Key Field            |\n|-----------------|-----------------------------|--------------------------------|\n| `CharField`     | `CharPGPPublicKeyField`     | `CharPGPSymmetricKeyField`     |\n| `EmailField`    | `EmailPGPPublicKeyField`    | `EmailPGPSymmetricKeyField`    |\n| `TextField`     | `TextPGPPublicKeyField`     | `TextPGPSymmetricKeyField`     |\n| `DateField`     | `DatePGPPublicKeyField`     | `DatePGPSymmetricKeyField`     |\n| `DateTimeField` | `DateTimePGPPublicKeyField` | `DateTimePGPSymmetricKeyField` |\n| `TimeField`     | `TimePGPPublicKeyField`     | `TimePGPSymmetricKeyField`     |\n| `IntegerField`  | `IntegerPGPPublicKeyField`  | `IntegerPGPSymmetricKeyField`  |\n| `BigIntegerField`  | `BigIntegerPGPPublicKeyField`  | `BigIntegerPGPSymmetricKeyField`  |\n| `DecimalField`  | `DecimalPGPPublicKeyField`  | `DecimalPGPSymmetricKeyField`  |\n| `FloatField`    | `FloatPGPPublicKeyField`    | `FloatPGPSymmetricKeyField`    |\n| `BooleanField`  | `BooleanPGPPublicKeyField`  | `BooleanPGPSymmetricKeyField`  |\n\n**Other Django model fields are not currently supported. Pull requests are welcomed.**\n\n### Usage\n\n#### Model Definition\n\n```python\nfrom django.db import models\n\nfrom pgcrypto import fields\n\nclass MyModel(models.Model):\n    digest_field = fields.TextDigestField()\n    digest_with_original_field = fields.TextDigestField(original='pgp_sym_field')\n    hmac_field = fields.TextHMACField()\n    hmac_with_original_field = fields.TextHMACField(original='pgp_sym_field')\n\n    email_pgp_pub_field = fields.EmailPGPPublicKeyField()\n    integer_pgp_pub_field = fields.IntegerPGPPublicKeyField()\n    pgp_pub_field = fields.TextPGPPublicKeyField()\n    date_pgp_pub_field = fields.DatePGPPublicKeyField()\n    datetime_pgp_pub_field = fields.DateTimePGPPublicKeyField()\n    time_pgp_pub_field = fields.TimePGPPublicKeyField()\n    decimal_pgp_pub_field = fields.DecimalPGPPublicKeyField()\n    float_pgp_pub_field = fields.FloatPGPPublicKeyField()\n    boolean_pgp_pub_field = fields.BooleanPGPPublicKeyField()\n    \n    email_pgp_sym_field = fields.EmailPGPSymmetricKeyField()\n    integer_pgp_sym_field = fields.IntegerPGPSymmetricKeyField()\n    pgp_sym_field = fields.TextPGPSymmetricKeyField()\n    date_pgp_sym_field = fields.DatePGPSymmetricKeyField()\n    datetime_pgp_sym_field = fields.DateTimePGPSymmetricKeyField()\n    time_pgp_sym_field = fields.TimePGPSymmetricKeyField()\n    decimal_pgp_sym_field = fields.DecimalPGPSymmetricKeyField()\n    float_pgp_sym_field = fields.FloatPGPSymmetricKeyField()\n    boolean_pgp_sym_field = fields.BooleanPGPSymmetricKeyField()\n```\n\n#### Encrypting\n\nData is automatically encrypted when inserted into the database.\n\nExample:\n```\n\u003e\u003e\u003e MyModel.objects.create(value='Value to be encrypted...')\n```\n\nHash fields can have hashes auto updated if you use the `original` attribute. This\nattribute allows you to indicate another field name to base the hash value on.\n\n```python\nfrom django.db import models\n\nfrom pgcrypto import fields\n\nclass User(models.Model):\n    first_name = fields.TextPGPSymmetricKeyField(max_length=20, verbose_name='First Name')\n    first_name_hashed = fields.TextHMACField(original='first_name') \n```\n\nIn the above example, if you specify the optional original attribute it would \ntake the unencrypted value from the first_name model field as the input value \nto create the hash. If you did not specify an original attribute, the field \nwould work as it does now and would remain backwards compatible.\n\n##### PGP fields\n\nWhen accessing the field name attribute on a model instance we are getting the\ndecrypted value.\n\nExample:\n```\n\u003e\u003e\u003e # When using a PGP public key based encryption\n\u003e\u003e\u003e my_model = MyModel.objects.get()\n\u003e\u003e\u003e my_model.value\n'Value decrypted'\n```\n\nFiltering encrypted values is now handled automatically as of 2.4.0. And `aggregate`\nmethods are not longer supported and have been removed from the library.\n\nAlso, auto-decryption is support for `select_related()` models.\n\n```python\nfrom django.db import models\n\nfrom pgcrypto import fields\n\n\nclass EncryptedFKModel(models.Model):\n    fk_pgp_sym_field = fields.TextPGPSymmetricKeyField(blank=True, null=True)\n\n\nclass EncryptedModel(models.Model):\n    pgp_sym_field = fields.TextPGPSymmetricKeyField(blank=True, null=True)\n    fk_model = models.ForeignKey(\n        EncryptedFKModel, blank=True, null=True, on_delete=models.CASCADE\n    )\n```\n\nExample:\n```\n\u003e\u003e\u003e import EncryptedModel\n\u003e\u003e\u003e my_model = EncryptedModel.objects.get().select_releated('fk_model')\n\u003e\u003e\u003e my_model.pgp_sym_field\n'Value decrypted'\n\u003e\u003e\u003e my_model.fk_model.fk_pgp_sym_field\n'Value decrypted'\n```\n\n##### Hash fields\n\nTo filter hash based values we need to compare hashes. This is achieved by using\na `__hash_of` lookup.\n\nExample:\n```\n\u003e\u003e\u003e my_model = MyModel.objects.filter(digest_field__hash_of='value')\n[\u003cMyModel: MyModel object\u003e]\n\u003e\u003e\u003e my_model = MyModel.objects.filter(hmac_field__hash_of='value')\n[\u003cMyModel: MyModel object\u003e]\n\n```\n\n## Limitations\n\n### Unique Indexes\n\nIt is usually not possible to index a `bytea` column in the database as the value in the index exceeds the the pgsql's maximum length allowed for an index (8192 bytes). One solution is to create a digest message of the value that you want unique and apply the unique constraint to the digest.\n\nYou can use the hash field ability to auto-create digest on the value of another field in the same model using the `original` argument. In the example below, a digest is created for unencrypted value that is in the `name` field when the model is saved or updated. A unique constraint exists on the name_digest so no two digests are allowed.  Note well that bulk updates do NOT cause hashes to be updated.\n\n```python\nfrom django.db import models\nfrom pgcrypto import fields\n\nclass Product(models.Model):\n    name_digest = fields.TextDigestField(original='name')\n    name = fields.TextPGPSymmetricKeyField()\n\n    class Meta:\n        constraints = [\n            models.UniqueConstraint(\n                fields=['name_digest', ],\n                name='name_digest_unique'\n            )\n       ]\n```\n\n### `.distinct('encrypted_field_name')`\n\nDue to a missing feature in the Django ORM, using `distinct()` on an encrypted field\ndoes not work for Django 2.0.x and lower.\n\nThe normal distinct works on Django 2.1.x and higher:\n\n```python\nitems = EncryptedFKModel.objects.filter(\n    pgp_sym_field__startswith='P'\n).only(\n    'id', 'pgp_sym_field', 'fk_model__fk_pgp_sym_field'\n).distinct(\n    'pgp_sym_field'\n)\n```\n\nWorkaround for Django 2.0.x and lower:\n\n```python\nfrom django.db import models\n\nitems = EncryptedFKModel.objects.filter(\n    pgp_sym_field__startswith='P'\n).annotate(\n    _distinct=models.F('pgp_sym_field')\n).only(\n    'id', 'pgp_sym_field', 'fk_model__fk_pgp_sym_field'\n).distinct(\n    '_distinct'\n)\n```\n\nThis works because the annotated field is auto-decrypted by Django as a `F` field and that \nfield is used in the `distinct()`.\n\n### Migrating existing fields into PGCrypto Fields\n\nMigrating existing fields into PGCrypto Fields is not performed by this library.  You will need to migrate the data \nin a forwards migration or other means. The only migration that is supported except to create/activate the pgcrypto \nextension in Postgres.\n\nMigrating data is complicated as there might be a few things to consider such as:\n\n* the shape of the data\n* validations/constrains done on the table/model/form and anywhere else\n\nThe library has no way of doing all these guesses or to make all these decisions.\n\nIf you need to migrate data from unencrypted fields to encrypted fields, three ways to solve it:\n\n1. When there's no data in the db it should be possible to start from scratch by recreating the db\n1. When there's no data in the table it should be possible to recreate the table\n1. When there's data or if the project is shared it should be possible to do it in a non destructive way\n\n**Option 1: No data is in the db**\n\n1. Drop the database\n1. Squash the migrations\n1. Recreate the db\n\n**Option 2: No data in the table**\n\n1. Create a migration to drop the table\n1. Create a new migration for the table with the encrypted field\n1. Optionally squash the migration\n\n**Option 3: Migrating in a non-destructive way**\n\nThe goal here is to be able to use to legacy field if something goes wrong.\n\nPart 1:\n\n1. Create new field\n1. When data is saved write both to legacy and new field\n1. Create a data migration to cast data from legacy field to new field\n1. check existing data from legacy and new field are the same if possible\n\nPart 2:\n\n1. Rename the fields and drop legacy fields\n1. Update the code to use only the new field\n\n## Common Errors\n\n### `psycopg2.errors.UndefinedFunction: function pgp_sym_encrypt(numeric, unknown) does not exist`\n\nThis commonly means you do not have the `pgcrypto` extension installed in Postgres.  Run the migration available in this library or install it manually in pgsql console.\n\n\n## Security Limitations\n\nTaken direction from the PostgreSQL documentation:\n\nhttps://www.postgresql.org/docs/9.6/static/pgcrypto.html#AEN187024\n\nAll pgcrypto functions run inside the database server. That means that all the \ndata and passwords move between pgcrypto and client applications in clear text. Thus you must:\n\n1. Connect locally or use SSL connections.\n1. Trust both system and database administrator.\n\nIf you cannot, then better do crypto inside client application.\n\nThe implementation does not resist side-channel attacks. For example, the time \nrequired for a pgcrypto decryption function to complete varies among ciphertexts of \na given size.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fincuna%2Fdjango-pgcrypto-fields","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fincuna%2Fdjango-pgcrypto-fields","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fincuna%2Fdjango-pgcrypto-fields/lists"}