{"id":21040122,"url":"https://github.com/sz3/pog","last_synced_at":"2025-05-15T16:33:08.476Z","repository":{"id":56403703,"uuid":"226614103","full_name":"sz3/pog","owner":"sz3","description":"Encryption/backup utility","archived":false,"fork":false,"pushed_at":"2024-06-29T05:14:24.000Z","size":1187,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-21T14:06:00.516Z","etag":null,"topics":["backup","cli","encryption","libsodium","x25519","xsalsa20poly1305"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sz3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-08T04:27:46.000Z","updated_at":"2024-09-20T15:59:25.000Z","dependencies_parsed_at":"2023-02-13T17:55:26.660Z","dependency_job_id":null,"html_url":"https://github.com/sz3/pog","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sz3%2Fpog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sz3%2Fpog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sz3%2Fpog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sz3%2Fpog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sz3","download_url":"https://codeload.github.com/sz3/pog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254377400,"owners_count":22061132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","cli","encryption","libsodium","x25519","xsalsa20poly1305"],"created_at":"2024-11-19T13:44:54.795Z","updated_at":"2025-05-15T16:33:08.191Z","avatar_url":"https://github.com/sz3.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://github.com/sz3/pog/workflows/ci/badge.svg)](https://github.com/sz3/pog/actions?query=workflow%3Aci)\n[![Coverage Status](https://coveralls.io/repos/github/sz3/pog/badge.svg?branch=master)](https://coveralls.io/github/sz3/pog?branch=master)\n[![PyPI Version](https://img.shields.io/pypi/v/pogcli.svg)](https://pypi.python.org/pypi/pogcli)\n[![Supported Python versions](https://img.shields.io/pypi/pyversions/pogcli.svg)](https://pypi.python.org/pypi/pogcli)\n\n## Pretty Ok Gncryption\n\n* File encryption and backups!\n* Uses `nacl.SecretBox` and `zstandard`!\n* Chunks up large files!\n* Can be used to generate encrypted archives locally, or as a backup tool that pushes to cloud storage providers. (`s3`, `b2`, ...tbd)\n* There is a GUI: [pogui](https://github.com/sz3/pogui).\n\nThis tool is built around symmetric encryption -- specifically libsodium's `crypto_secretbox`, which is `XSalsa20+Poly1305`. Doing asymmetric PGP-like things is not in the cards -- but there is an experimental use case using asymmetric crypto that will likely be supported.\n\n* Still in beta!\n* Don't rely on this to keep your government leaks secret!\n\n## Installation\n\nUse `pip`:\n```\npip install pogcli\n```\n\nor from source,\n```\npython setup.py build\npython setup.py install\n```\n\n## Usage\n\n### Credentials\n* Pog does not manage cloud storage credentials -- it asks that you configure your environment with API keys before use.\n\t* To validate s3 credentials:\n\t\t* `awscli ls \u003cbucket_name\u003e`\n\t* To validate b2 credentials:\n\t\t* `b2 ls \u003cbucket_name\u003e`\n\n### Using a password or keyfiles\n1. symmetric keyfile\n\t* any file can be used as a keyfile.\n\t* the contents of the keyfile will be hashed, and that hash will become the cryptographic key\n\t* cryptographic randomness (ex: 1024 bytes from /dev/urandom) is recommended\n2. asymmetric keyfiles\n\t* the `pog-create-keypair` script will generate an \"encrypt\" and \"decrypt\" keypair.\n\t* encrypt is used for creating archives\n\t* decrypt is used for extracting them\n3. Password entry\n\t* if no keyfiles are specified, Pog supports password entry for creating or reading archives\n\n### Creating cloud archives and backups\n\n* Consider an S3 backup:\n\n```\npog /home/user/my_file.txt --keyfile=/home/user/secret.keyfile --save-to=s3://my-bucket --store-absolute-paths\n```\n\nThis does a few things:\n1. `my_file.txt` is encrypted with `secret.keyfile`. If the file is sufficiently large, it is split into multiple pieces during encryption.\n2. The encrypted contents (\"blob\") of `my_file.txt` is saved to the s3 bucket `my-bucket`, under the top-level `data/` subdirectory.\n3. An encrypted \"manifest\" file is created, named according to the time the archive was created. This manifest file acts as an index from filenames (`/home/user/my_file`) to one or more encrypted blobs.\n   a. The `--store-absolute-paths` flag tells the manifest to resolve ambiguous paths with the absolute path (`/home/user/my_file`) instead of the relative path (`my_file`). This can be useful to have when extracting archives or backups.\n4. The manifest file is also saved to `my-bucket` in s3.\n\n----\n\n* Here is another example, with a series of directories:\n\n```\npog /opt/games /opt/apps /opt/music --encryption-keyfile=secret.encrypt --save-to=s3://my-bucket,b2://my-b2-bucket\n```\n\n* This will recursively go through those 3 directories, gathering up all files and saving the encrypted blobs to both s3 and b2.\n\nThe command line help (`pog -h`) shows other useful examples.\n\n### Creating local archives\n\n* It is also possible to use Pog to encrypt a single file.\n\n```\npog /home/myfile.original \u003e outputs.txt\n```\n\n* and to decrypt:\n\n```\npog --decrypt $(cat outputs.txt) \u003e myfile.copy\n```\n\n### Reading archives and backups\n\nFor a given manifest file (`2020-01-23T12:34:56.012345.mfn`), we can download and extract the archive like so:\n\n```\npog --decrypt s3:/my-bucket/2020-01-23T12:34:56.012345.mfn --keyfile=/home/user/secret.keyfile\n```\n\n* The `--decrypt` flag should be specified for read+decrypt -- the default behavior is to write+encrypt.\n* If a `--decryption-keyfile` is provided, `--decrypt` is assumed.\n* If a local manifest file is provided, it is assumed that the data blobs are already downloaded into the working directory.\n\n## Algorithm\n\n* files are compressed with `zstandard`, and split (\"chunked\") into blobs. The default chunk size is 50MB.\n\n* blob contents are encrypted with `crypto_secretbox`. The key is 256 bits, independent *per-blob*, and stored in the blob header.\n\n* the blob header is encrypted in one of 3 ways:\n\t* `crypto_secretbox` with key=sha256(argon2.ID with `time_cost=8, memory_cost=102400, parallelism=8, hash_len=32`)\n\t\t* this is what is used when you get a password prompt\n\t* `crypto_secretbox` with key=sha256(keyfile contents)\n\t\t* this is what the `--keyfile` option does\n\t* `crypto_sealedbox` with an X25519 key pair\n\t\t* this is what `--decryption-keyfile` and `--encryption-keyfile` do\n\t\t* an X25519 key pair can be generated with `pog-create-keypair`.\n\n* the file-\u003eblob relationship is stored in an encrypted manifest file (`.mfn`), which also stores file metadata -- e.g. last modified time.\n\t* the `.mfn` can be thought of as the dictionary for the archive.\n\t* blobs *can* be decrypted without the manifest, *IF* the blob order is correct. However, only the file contents are stored in the blobs. The original file name and file metadata will not survive the trip.\n\n* blobs are named by urlsafe base64(sha256(sha256(secret) + sha256(content)). The \"secret\" is derived from the encryption key.\n\t* the goal is to pseudo-randomize the names of the blobs, while still keeping them consistent for backups run with the same key.\n\t* we want to \"leak\" the content hash only to the extent it's necessary to save work on successive backups (e.g. \"I don't need to reupload blob X, it already exists\")\n\t* because we use the content hash for this purpose, we can achieve some amount of file de-duplication.\n\n## Disclaimer\n\nI'm a not a cryptographer, just an engineer with internet access.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsz3%2Fpog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsz3%2Fpog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsz3%2Fpog/lists"}