{"id":16931926,"url":"https://github.com/thenickdude/snap-to-s3","last_synced_at":"2025-03-17T07:32:09.551Z","repository":{"id":21251398,"uuid":"91047664","full_name":"thenickdude/snap-to-s3","owner":"thenickdude","description":"Upload EBS volume snapshots to Amazon S3/Glacier","archived":false,"fork":false,"pushed_at":"2023-06-11T06:02:07.000Z","size":307,"stargazers_count":115,"open_issues_count":3,"forks_count":23,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-13T21:18:03.287Z","etag":null,"topics":["aws","aws-ebs-snapshot","aws-glacier","aws-s3"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thenickdude.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"License.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-05-12T03:31:43.000Z","updated_at":"2024-12-28T22:30:45.000Z","dependencies_parsed_at":"2023-10-21T00:30:31.716Z","dependency_job_id":null,"html_url":"https://github.com/thenickdude/snap-to-s3","commit_stats":{"total_commits":31,"total_committers":1,"mean_commits":31.0,"dds":0.0,"last_synced_commit":"6f1d9f29ba3e28f9add68fee1de4717d7a9fa98c"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thenickdude%2Fsnap-to-s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thenickdude%2Fsnap-to-s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thenickdude%2Fsnap-to-s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thenickdude%2Fsnap-to-s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thenickdude","download_url":"https://codeload.github.com/thenickdude/snap-to-s3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243852425,"owners_count":20358270,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-ebs-snapshot","aws-glacier","aws-s3"],"created_at":"2024-10-13T20:45:01.333Z","updated_at":"2025-03-17T07:32:09.254Z","avatar_url":"https://github.com/thenickdude.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# snap-to-s3\n\nThis tool will turn AWS EBS volume snapshots into temporary EBS volumes, tar them up, compress \nthem with LZ4/ZSTD, and upload them to Amazon S3 for you. You can also opt to create an image of the \nentire volume by using `dd`, instead of using `tar`.\n\nOnce stored on S3, you could add an S3 Lifecycle Rule to the S3 bucket to automatically [migrate\nthe snapshots into Glacier](http://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html#before-deciding-to-archive-objects).\n\n## Requirements and installation\n\nThis tool is only intended to run on Linux, and has only been tested on Ubuntu 22.04,\nAmazon Linux 2017.03 and Amazon Linux 2 2017.12.\n\nThis tool must be run on an EC2 instance, and can only operate on snapshots within the same\nregion as the instance.\n\nThis is a Node.js application, so if you don't have it installed already, install node (at least\nversion 12 LTS or newer) and npm:\n\n```bash\n# Ubuntu 22.04\n# To use the Ubuntu-bundled Node JS:\nsudo apt-get install -y nodejs npm\n\n# Or to get a newer version:\ncurl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -\nsudo apt-get install -y nodejs\n\n# Amazon Linux 2\ncurl -fsSL https://rpm.nodesource.com/setup_lts.x | sudo bash -\n```\n\nBy default, the \"lz4\" command-line compression tool will be used to compress the tars, so make sure you \nhave it available:\n\n```bash\n# Ubuntu 22.04\nsudo apt-get install liblz4-tool\n\n# Amazon Linux 2\nsudo yum install lz4\n# We'll also need git for installation:\nsudo yum install git\n```\n\nIf you want to use zstd compression instead (with `--compression zstd`), make sure you have the \n\"zstd\" package installed.\n\nIf your instance attaches its drives using NVMe (i.e. disks have names like /dev/nvme1n1), you \nalso need to have an NVMe tool installed (`ebsnvme-id` provided by the package `ec2-util` on \nAmazon Linux, or `nvme` provided by the package `nvme-cli` on other Linuxes). \n\nNow you can fetch and install snap-to-s3 from NPM:\n\n```bash\nsudo npm install -g snap-to-s3\n```\n\nOr if you download `snap-to-s3` from [its GitHub repository](https://github.com/thenickdude/snap-to-s3), \nyou can install that version instead from the repository root:\n\n```bash\nnpm install # Fetch dependencies\nnpm link    # Link this installation to your $PATH\n```\n\nNow it'll be on your $PATH, so you can run it like so:\n\n```bash\nsudo snap-to-s3 --help\n```\n\nIn order to mount and unmount volumes, and read all files for backup, `snap-to-s3` will need\nto be run as root or with `sudo`.\n\n### Instance metadata service\nThis tool requires access to the [instance metadata service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)\nat `http://169.254.169.254:80/`, so ensure that your instance does not have a firewall policy \nthat blocks access to this.\n\n### Credentials / IAM policy\n\nThis tool needs to create volumes from snapshots, perform uploads to S3, attach and detach \nvolumes to/from instances, delete volumes, and add and delete tags. For snapshot validation, it \nalso needs to read objects from S3.\n\nYou can grant these permissions by attaching an [IAM Role](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html)\nto your instance with [the following policy attached](iam-policy.json). Don't forget to update \nthe bucket names in that policy with the actual name of the S3 bucket you'll be uploading to. \n`snap-to-s3` will then be able to use that policy automatically with no further configuration.\n\nIf you're not using an IAM Instance Role to give permissions to snap-to-s3, you can grant\nthose permissions to an IAM user and provide an AWS Access Key ID / Secret Access Key pair \nfor that user instead, \n[follow these instructions](http://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html).\n\n## Disclaimer and warnings\n\nThis tool works for me and for my use-case, and I'm happy if it works for you too, but you \nmight be doing something that I didn't expect. There is a definite potential for data-loss \nif something goes wrong. \n\nI haven't tested it at all with `LVM` or `mdadm`, and it's likely to fail horribly. Only use \nwith disks with regular partitions on them (or disks with no partition table, i.e. only one \nnormal formatted filesystem).\n\nIt's better that the temporary EBS volumes created from the snapshots don't automatically get \nmounted read-write by your instance's `/etc/fstab`, since this might cause changes to the \nfiles on the volume during upload. For `dd` backups, this is much more critical.\n\n`snap-to-s3` will tag temporary EBS volumes it creates from snapshots with a tag called\n`snap-to-s3` (this is configurable). Accordingly, it will assume that any EBS volume with this\ntag is one of its volumes that it can do whatever it likes with (including deleting it).\n\nThe fidelity of the backup depends on how well `tar` is able to preserve your files.\n`snap-to-s3` calls your system's `tar` using the default options. If you have some unusual \nfiles (odd extended file attributes, long path names, special characters in filenames) you \nmay find that a tar backup isn't perfect. You can use the `--dd` option instead to just image \nthe entire volume, but note that this will include data from deleted  files in the \"free\" \nportion of the drive, and so will increase the backup size considerably for non-full volumes.\n\nWarnings from `tar` will be printed to the screen, but otherwise ignored by `snap-to-s3` \nunless `tar` returns a non-zero exit code. In practice, the only warnings I've seen have come \nfrom snapshots taken of a running operating system's root disk, where `tar` will note that it\nis ignoring unix socket files like `/var/spool/postfix/public/flush` (this is a good thing).\n\nNote that snapshots will not be deleted for you even after copying them to S3, so you have\nthe opportunity to verify the snapshot was transferred correctly before removing it\nyourself. Upload validation can be performed by `snap-to-s3` using the `--validate` option, \nor you could do it yourself manually.\n\nFor a manual validation, you could use the `--keep-temp-volumes` option to retain the\ntemporary volume after migration, and run \n`find . -type f -exec md5sum {} \\; | sort -k 2 | md5sum` in that directory to compute a \nsignature for the files in the volume. Then in a different directory, you can download and \nuntar the snapshot you just uploaded to S3 (e.g. using the [AWS CLI](https://aws.amazon.com/cli/) \nlike  `aws s3 cp \"s3://backups.example.com/vol-xxx/snap-xxx.tar.lz4\" - | lz4 -d | tar -x`), \ncompute the same signature over the files you unpacked, and ensure the signatures match.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD \nTO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN \nNO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL \nDAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER \nIN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN \nCONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.\n\n## Usage\n\n### Migrating snapshots to S3\nA typical migration command looks like this:\n \n```bash\nsudo snap-to-s3 --migrate --all --bucket backups.example.com\n```\n\nThis will search for all snapshots in the current region which have a tag called \"snap-to-s3\" \nwith the value \"migrate\" (you need to tag these yourself beforehand). The snapshots will be\nturned into temporary EBS volumes and attached to your instance. Each partition of those\nvolumes will be separately tar'd and compressed with lz4 before being uploaded to the bucket \nthat you've specified. The temporary EBS volume will then be detached and deleted.\nFinally, the snapshot will be tagged with \"migrated\". \n\nThe resulting S3 objects will have locations like:\n\n```\ns3://backups.example.com/vol-xxx/2017-01-01T00:00:00+00:00 snap-xxx.1.tar.lz4\ns3://backups.example.com/vol-xxx/2017-01-01T00:00:00+00:00 snap-xxx.2.tar.lz4\ns3://backups.example.com/vol-xxx/2017-05-01T00:00:00+00:00 snap-yyy.tar.lz4\n```\n\nMetadata is added to the files on S3 with details of the snapshot that it was created from,\nand tags that were applied to the snapshot are copied over (with some substitutions for \nillegal characters).\n\nNote that you need to create the bucket beforehand (it's not created for you), and you should\ncreate it in the same region as your snapshots in order to eliminate AWS's inter-region\ntransfer fees.\n\n### Validating uploaded snapshots\nIf you want to make sure that the snapshot was uploaded to S3 correctly, you can use the \n\"--validate\" option. This option can either be added at the same time as you perform your\n--migrate:\n\n```bash\nsudo snap-to-s3 --migrate --validate --all --bucket backups.example.com\n```\n\nOr it can be done in a separate invocation after migration using just the \n\"--validate\" option. \n\n```bash\nsudo snap-to-s3 --validate --all --bucket backups.example.com\n```\n\nIf you want to validate it using a separate invocation, you can speed up validation massively \nby providing the \"--keep-temp-volumes\" option when you perform the --migrate.\n\nThe previously-uploaded tar will be downloaded from S3, unpacked, and MD5 \nhashes will be computed of all of the files in it. This uses a streaming approach, so \nno extra disk space is needed for temporary files. For dd images, a hash of the entire\nraw volume is taken instead.\n\nAt the same time, the snapshot being verified will be turned into a temporary EBS volume, \nmounted, and MD5 hashes will be computed of all of its files.\n\nFinally, when both processes are complete, you'll be told if there are any files missing\nfrom the S3 copy of the snapshot, and if any of the file hashes differ. If validation was\nsuccessful, the snapshot will be tagged with the value \"validated\", and the temporary EBS\nvolume will be detached and deleted.\n\nNote that the tar validation only compares the hashes of the content of regular files. Special \nfiles like symlinks are not checked at all, and neither are attributes like file permissions.\n\n### Restoring snapshots from S3\n\n`snap-to-s3` doesn't perform snapshot restorations itself, but you can do this with the\n[AWS CLI](https://aws.amazon.com/cli/). \n\nTo restore a tar, check the metadata on the archive on S3 to find the \n\"x-amz-meta-uncompressed-size\" header, this will give you a hint about how large of an EBS \nvolume you'll need to create to hold the volume (you'll need somewhat more space than this in \norder to hold filesystem metadata). Create a volume of that size, attach it to the instance, \ncreate a filesystem on it with `mkfs`, mount it somewhere useful, and enter that directory. Now \nyou can download and extract the tar from S3 like so:\n\n```bash\naws s3 cp \"s3://backups.example.com/vol-xxx/2017-01-01 snap-xxx.tar.lz4\" - | lz4 -d | sudo tar -x\n```\n\nIf you're restoring an image that was created with dd, create and attach an EBS volume at least \nas large as the \"x-amz-meta-snapshot-volumesize\" field indicates. If you attached it at\n`/dev/xvdf` (for example), then you could restore the snapshot like so:\n\n```bash\naws s3 cp \"s3://backups.example.com/vol-xxx/2017-01-01 snap-xxx.img.lz4\" - | lz4 -d | sudo dd bs=1M of=/dev/xvdf\n```\n\n### Analyzing a Cost and Usage report\n\n`snap-to-s3` can examine an Amazon Cost and Usage report to show you a per-volume and \nper-snapshot breakdown of your EBS snapshot charges, which you can use to identify snapshots \nsuitable for migrating to S3/Glacier.\n\nFrom your Amazon billing dashboard, go to the Reports section, and create a new report. Choose\n\"daily\" for the time period, tick the \"Include Resource IDs\" box, choose GZip compression,\nand select an S3 bucket to store the report in. Around 24 hours later, you should have a .csv.gz \nreport in that bucket to analyze. Download it to your instance.\n\nYou can pass that file to `snap-to-s3` using any of these styles:\n\n```bash\nsnap-to-s3 --analyze costreport-1.csv \n\nsnap-to-s3 --analyze costreport-1.csv.gz \n\naws s3 cp \"s3://cost-reports.example.com/20170501-20170601/x-x-x-x-x/costreport-1.csv.gz\" - | snap-to-s3 --analyze\n```\n\n`snap-to-s3` will summarize the billing data in the report, then combine it with information about\nyour current volumes and snapshots (DescribeVolumes and DescribeSnapshots).\n\nHere's an example output. The effective size of each snapshot is shown next to it, this is the\namount of data in the snapshot that differs from the previous snapshot, which is the size you \nare billed for: \n\n```\nRegion us-west-2 ($166.16/month for 24 snapshots)\nvol-xxx (500GB, MySQL Slave DB): 3016 GB total, $151/month for 16 snapshots, average snapshot change 32%\n  snap-xxx  2016-11-01  448.7 GB\n  snap-xxx  2016-12-01  261.7 GB (52%)\n  snap-xxx  2017-01-01  301.5 GB (60%)\n  snap-xxx  2017-02-01  275.4 GB (55%)\n  snap-xxx  2017-03-01  250.5 GB (50%)\n  snap-xxx  2017-04-01  279.3 GB (56%)\n  snap-xxx  2017-05-01  320.6 GB (64%)\n  snap-xxx  2017-05-17  218.1 GB (44%)\n  snap-xxx  2017-05-18  90.8 GB (18%)\n  snap-xxx  2017-05-19  85.2 GB (17%)\n  snap-xxx  2017-05-20  89.4 GB (18%)\n  snap-xxx  2017-05-21  93.2 GB (19%)\n  snap-xxx  2017-05-22  92.6 GB (19%)\n  snap-xxx  2017-05-23  82.8 GB (17%)\n  snap-xxx  2017-05-24  87.1 GB (17%)\n  snap-xxx  2017-05-25  39.5 GB (7.9%)\n\nvol-xxx (20GB, deleted): 5 GB total, $0.247/month for 6 snapshots, average snapshot change 2.1%\n  snap-xxx  2015-04-27   2.4 GB\n  snap-xxx  2015-05-04   0.1 GB (0.43%)\n  snap-xxx  2015-06-01   0.1 GB (0.56%)\n  snap-xxx  2015-06-29   0.1 GB (0.62%)\n  snap-xxx  2016-04-25   2.2 GB (11%)\n  snap-xxx  2016-09-08   0.0 GB (0.069%)\n```\n\nIn this case, the older snapshots of the first volume change a lot, so the delta encoding scheme\nof EBS snapshots isn't saving us very much. These snapshots are a great candidate to move to S3\nor Glacier. Whereas the second set of snapshots change by nearly nothing, so S3/Glacier will be\nmore expensive.\n\n### All options\nHere's the full options list:\n\n```\nMigrate snapshots to S3\n\n  --migrate                    Migrate EBS snapshots to S3\n  --all                        Migrate all snapshots whose tag is set to \"migrate\"\n  --one                        ... or migrate any one snapshot whose tag is set to \"migrate\"\n  --snapshots SnapshotId ...   ... or provide an explicit list of snapshots to migrate (tags are ignored)\n  --upload-streams num         Number of simultaneous streams to send to S3 (increases memory usage, \n                               default: 1)\n  --compression name           Compression type to use (lz4 or zstd, default: lz4)\n  --compression-level level    Compression level (1-9 for lz4, 1-19 for zstd, default: 1)\n  --dd                         Use dd to create a raw image of the entire volume, instead of tarring up the\n                               files of each partition\n  --sse mode                   Enables server-side encryption, valid modes are AES256 and aws:kms\n  --sse-kms-key-id id          KMS key ID to use for aws:kms encryption, if not using the S3 master KMS key\n  --gpg-recipient keyname      Encrypt the image for the given GPG recipient (add multiple times for\n                               multiple recipients)\n\nValidate uploaded snapshots\n\n  --validate                   Validate uploaded snapshots from S3 against the original EBS snapshots (can\n                               be combined with --migrate)\n  --gpg-session-key key        See readme for details\n  --all                        Validate all snapshots whose tag is set to \"migrated\"\n  --one                        ... or validate any one snapshot whose tag is set to \"migrated\"\n  --snapshots SnapshotId ...   ... or provide an explicit list of snapshots to validate (tags are ignored)\n\nAnalyze AWS Cost and Usage reports\n\n  --analyze filename   Analyze an AWS Cost and Usage report to find opportunities for savings\n\nCommon options\n\n  --help                 Show this page\n\n  --tag name             Name of tag you have used to mark snapshots for migration, and to mark\n                         created EBS temporary volumes (default: snap-to-s3)\n  --bucket name          S3 bucket to upload to (required)\n  --mount-point path     Temporary volumes will be mounted here, created if it doesn't already exist\n                         (default: /mnt)\n  --keep-temp-volumes    Don't delete temporary volumes after we're done with them\n  --volume-type type     Volume type to use for temporary EBS volumes (e.g. standard or gp2), by default\n                         standard volumes will be used for volumes 1TiB and smaller, gp2 will be used for\n                         larger volumes\n```\n\n## Performance\n\nThe snapshot migration rate that `snap-to-s3` achieves seems to be largely limited by how fast\nEC2 will turn a snapshot into a completely-readable volume (i.e. the rate they can copy blocks\nfrom their own private S3 snapshot storage into EBS). For me a simple `dd` from an EBS volume \n(freshly created from a snapshot) to `/dev/zero` averages a transfer rate of 3.5MiB/s, which \ngives an expected 24 hours to upload a 300GiB snapshot with `snap-to-s3`.\n\nIf your migration rate is being limited by this, you'll notice it as a high iowait percentage \nin \"top\" and low \"volume idle\" percentages in the EC2 console. In this situation you can \nincrease your effective snapshot upload rate by running multiple instances of `snap-to-s3` at \nthe same time to upload multiple snapshots in parallel. This allows you to scale your upload \nrate nearly linearly with the number of snapshots being uploaded, until other limits are \nreached like network speed and CPU usage.\n\n## Resource usage\n\nThe compression consumes the largest portion of the CPU time. If you're on a t2-series instance, \nyou'll probably want to use the least amount of compression (`--compression-level 1`), which is\nthe default. Faster instances can afford to use up to level 9. If you're using compression \nlevels beyond about 1 or 2 with lz4, consider switching to zstd compression using \n`--compression zstd` instead, as it gives much better compression ratios at the same speed\nat these levels.\n\nAs an example, using zstd at compression level 9, on a c5.large, the 2 vCPUs are completely \nmaxed out with 9 parallel upload processes (i.e. 4.5 processes per vCPU). This measurement \nshould scale to the other instances in the c5 family.\n\nNode will consume the most memory. Of that memory, the S3 upload process will use at least \n`part_size * num_upload_streams` bytes to buffer the upload. \n\nThe part size is set automatically based on the uncompressed size of the data being uploaded, \nit is approximately `uncompressed_size / 9000`, and is always at least 5MB.\n \nThe number of upload streams defaults to 1. If your snapshot volume can be read very quickly\n(e.g. if you are using EBS Snapshot Fast Restore), this may be a bottleneck, and you may\nwant to bump this up using the `--upload-streams` option.\n\nHere's the minimum amount of memory that would be consumed with various volume sizes (for \n100% full volumes):\n\n| Volume size | Memory with streams = 1 | Memory with streams = 4 |\n| ----------- | ----------------------- | ----------------------- |\n| 1GB         | 5MB                     | 20MB                    |\n| 40GB        | 5MB                     | 20MB                    |\n| 100GB       | 11MB                    | 46MB                    |\n| 200GB       | 23MB                    | 91MB                    |\n| 400GB       | 46MB                    | 180MB                   |\n| 800GB       | 91MB                    | 360MB                   |\n| 1600GB      | 180MB                   | 730MB                   |\n| 3200GB      | 360MB                   | 1500MB                  |\n\nOn top of this, inefficiencies in Node's memory management (especially the garbage collector) \nwill likely require a factor more memory, and there's a fixed-size overhead of around 100MB.\nTest it out with your snapshots/Node version if memory is tight.\n\n## Cost analysis\n\n### Storage costs\n\nAt the time of writing and in the region I use, EBS snapshots were charged at $0.05/GB-month, \nS3 Standard at $0.023/GB-month, S3 Infrequent-Access at $0.0125/GB-month, and Glacier at \n$0.004/GB-month, so migrating snapshots to S3 or Glacier could potentially save you on your \nmonthly storage bill.\n\nHowever, keep in mind that EBS snapshots are incremental; if you have two snapshots of the\nsame volume, the second snapshot will only require enough storage space to hold the blocks \nthat changed since the previous snapshot. In contrast, snapshots pushed to S3 or Glacier with \nthis tool are full backups, not incremental.\n\nThis means that the cost difference between EBS snapshots and S3/Glacier will depend on how \nmuch your successive snapshots differ.\n\nIf you have many snapshots of the same volume (i.e., in the limit as the number of snapshots\nreaches infinity), and your volume changes by more than 46% between successive snapshots, S3 \nStandard is cheaper than EBS snapshots. For S3 Infrequent Access, the breakeven point is at\n25%.\n\nFor Glacier, the breakeven point comes much sooner, with volumes changing more than 8% between \nsnaps being cheaper to store on Glacier. \n\nIf LZ4 achieves a 2:1 compression ratio on your data, this breakeven point is correspondingly\nhalved (i.e. Glacier would break-even at 4% change between snapshots).\n\nFor a smaller number of snapshots, the breakeven point is reached earlier. Let's say \nthat you are slowly outgrowing your volume sizes, so you create a new volume each year, and \nyou want to retain one snapshot per month (12 snapshots for the lifetime of the volume). \nHere's the price of S3 and Glacier as a percentage of the cost of EBS snapshots.\n\n| Change between snapshots | S3 Std  | S3 IA | Glacier |\n| ------------------------ | ------- | ----- | ------- |\n| 0%                       | 550%    | 300%  | 96%     |\n| 1%                       | 500%    | 270%  | 86%     |\n| 2%                       | 450%    | 250%  | 79%     |\n| 3%                       | 420%    | 230%  | 72%     |\n| 4%                       | 380%    | 210%  | 67%     |\n| 5%                       | 360%    | 190%  | 62%     |\n| 10%                      | 260%    | 140%  | 46%     |\n| 15%                      | 210%    | 110%  | 36%     |\n| 20%                      | 170%    | 94%   | 30%     |\n| 30%                      | 130%    | 70%   | 22%     |\n| 40%                      | 102%    | 56%   | 18%     |\n| 50%                      | 85%     | 46%   | 15%     |\n| 75%                      | 60%     | 32%   | 10%     |\n| 100%                     | 46%     | 25%   | 8.0%    |\n\nNotice that if you only have 12 snapshots of a given volume, Glacier is always cheaper than\nEBS snapshots, no matter how much each snapshot changes compared to the previous one.\n\nHere's the same situation if you achieve a 2:1 compression ratio using LZ4:\n\n| Change between snapshots | S3 Std 2:1 | S3 IA 2:1 | Glacier 2:1 |\n| ------------------------ | ---------- | --------- | ----------- |\n| 0%                       | 280%       | 150%      | 48%         |\n| 1%                       | 250%       | 140%      | 43%         |\n| 2%                       | 230%       | 120%      | 39%         |\n| 3%                       | 210%       | 110%      | 36%         |\n| 4%                       | 190%       | 104%      | 33%         |\n| 5%                       | 180%       | 97%       | 31%         |\n| 10%                      | 130%       | 71%       | 23%         |\n| 15%                      | 104%       | 57%       | 18%         |\n| 20%                      | 86%        | 47%       | 15%         |\n| 30%                      | 64%        | 35%       | 11%         |\n| 40%                      | 51%        | 28%       | 8.9%        |\n| 50%                      | 42%        | 23%       | 7.4%        |\n| 75%                      | 30%        | 16%       | 5.2%        |\n| 100%                     | 23%        | 12.5%     | 4.0%        |\n\nIf you want to play with the parameters that generated this table you can do so using [this\nspreadsheet on Google Docs](https://docs.google.com/spreadsheets/d/1NH0XS5_HSuHJ8K5WOgSPYRFVW0OKFR1j1NgPLi9ig7E/edit?usp=sharing).\nYou'll need to download it or copy it to your own account to be able to edit the fields.\n\nS3 Infrequent Access archives have their storage costs charged based on a minimum 30-day\nlifetime, even if you delete them or migrate them to Glacier before then.\n\nGlacier archives have their storage costs charged based on a minimum 90-day lifetime, even if \nyou delete them sooner.\n\nDon't forget that restoring from Glacier takes longer and costs much more than from EBS or S3. \nIt's mostly suitable for archival backups. \n\n### Migration costs\n\nThere are several costs involved in using `snap-to-s3`. Here are some of the costs that I \nconsider significant in my use-case. Your use-case may vary:\n\nPushing snapshots to S3 will require the use of an EC2 instance for some hours (prices vary,\nespecially depending on how many snapshots you upload at once). You'll pay EBS storage and I/O \ncosts for the temporary volumes created from the snapshots while they are uploading. Uploads \nto S3 are made in at most 9000 parts, which requires 9000 S3 PutObject calls (so $0.045 per\nvolume if charged at $0.005/1000 PUTs).\n\nAvoid uploading to a S3 bucket in a different region, since it will incur inter-region transfer\ncosts for both upload and download.\n\nThere are other per-request costs that should only become significant if you are migrating\nthousands of snapshots. Please read the relevant Amazon documentation for details.\n\n## If something goes wrong\n### Stuck volumes\nOccasionally, EC2 fails to properly attach a volume to the instance, and it gets stuck in the\n\"attaching\" state. You'll get this error message on the command line:\n\n```\n[snap-xxx] An error occurred, tagging snapshot with \"migrate\" so it can be retried later\n[snap-xxx] Timed out waiting for vol-xxx to appear in /dev\n```\n\nOn the EC2 web console, use the \"force detach\" option on the volume, then reattach it on a \ndifferent mount-point and re-run snap-to-s3. \n\nIf you end up \"poisoning\" too many mount-points with this problem, you may need to stop and\nstart your instance in order to clear them.\n\n### Killed process\nIf Ctrl+C is pressed while an upload is in progress to S3 (sending a SIGINT), the \nmultipart-upload to S3 is cleanly aborted.\n\nIf the process gets SIGINT at some other time, the snapshot that was currently being uploaded \nwill likely still have its tag set to \"migrating\", which will prevent it from being migrated \nagain when calling `snap-to-s3 --migrate --all`. \n\nYou can either manually change that snapshot tag in the EC2 web console to \"migrate\" \nbefore trying again, or you can explicitly pass the snapshot id to the `--snapshots` argument \nwhich will ignore the \"migrating\" tag for you.\n\nIf `snap-to-s3` receives a `SIGHUP` from your SSH session dropping, it will be killed. \nConsider running snap-to-s3 in a `screen`/`tmux` session, or with \n[nohup](https://en.wikipedia.org/wiki/Nohup).\n\n### Incomplete uploads\nIf something really weird goes wrong, (e.g. process receives SIGKILL due to out of memory\ncondition) a half-completed multipart upload might be left on S3, which will continue to \nincur storage charges. These incomplete uploads do not appear in your bucket as objects, or \nin the S3 web console. \n\nYou can use \n[abort-incomplete-multipart](https://github.com/thenickdude/abort-incomplete-multipart) to \nremove those leftovers. Another option is to add a lifecycle policy to the S3 bucket which\nautomatically deletes incomplete multipart uploads after X days (where X is comfortably longer\nthan the longest snapshot upload time you expect with snap-to-s3).\n\n## Encrypting snapshots with GPG\n\nSnapshots can be additionally encrypted using `gpg2`'s asymmetric encryption before they are uploaded to S3. By allowing\nthe corresponding private key to be stored offsite (perhaps in a hardware device such as a YubiKey), this allows \nencrypted snapshots to be stored on S3 that cannot be decrypted even if the AWS root account is compromised.\n\nYou need to have gpg2 installed (typically provided by the `gnupg2` package), and the public keys that you will encrypt\nthe snapshot to need to be loaded into GPG and marked as trusted (e.g. with `gpg2 --import` or `gpg2 --recv-keys`, \nthen `gpg2 --edit-key`). Then when you call `snap-to-s3 --migrate`, you can add `--gpg-recipient` arguments (one per \npublic key) that name the public keys that GPG will encrypt the snapshot to, for example:\n\n```bash\nsnap-to-s3 --migrate --gpg-recipient SnapToS3Example --bucket backups.example.com --snapshots snap-xxx\n```\n\nHowever, the encryption poses a challenge if you want to use the `--validate` command, because the snapshot needs to be \ndecrypted in order to perform validation, and this requires the corresponding private key to be available to `snap-to-s3`.\n\nThere are two main strategies for dealing with this:\n\n### Validation where the private key is available to snap-to-s3\n\nLet's imagine that you were okay with storing both the public and private keys for the snapshot on the `snap-to-s3` \ninstance (after carefully considering the security implications). If you don't already have one, you could generate \nsuch a key like this:\n\n```\n# gpg2 --full-gen-key\n\ngpg (GnuPG) 2.1.11; Copyright (C) 2016 Free Software Foundation, Inc.\n\nPlease select what kind of key you want: 1 RSA and RSA (default)\nWhat keysize do you want? (2048) 2048\nKey is valid for? (0) 10y\n\nReal name: SnapToS3Example\nEmail address: n.sherlock@gmail.com\n\nYou selected this USER-ID:\n    \"SnapToS3Example \u003cn.sherlock@gmail.com\u003e\"\n    \ngpg: Please enter a passphrase to protect your private key\n...\n\ngpg: key 142906AF marked as ultimately trusted\n\ngpg: checking the trustdb\ngpg: marginals needed: 3  completes needed: 1  trust model: PGP\ngpg: depth: 0  valid:   3  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 3u\ngpg: next trustdb check due at 2029-01-18\npub   rsa2048/142906AF 2019-01-21 [S] [expires: 2029-01-18]\n      Key fingerprint = 06C9 ADB1 E792 4C6F 5CF0  5473 98FC 45D4 1429 06AF\nuid         [ultimate] SnapToS3Example \u003cn.sherlock@gmail.com\u003e\nsub   rsa2048/84B62A1E 2019-01-21 [] [expires: 2029-01-18]\n```\n\nThen you can use snap-to-s3 to encrypt a snapshot to that key like so:\n\n```bash\nsnap-to-s3 --migrate --validate --gpg-recipient SnapToS3Example --bucket backups.example.com --snapshots snap-xxx\n```\n\nHowever, in order for the `--validate` to succeed, the private key needs to have its passphrase unlocked. `snap-to-s3` \nis not able to accept keyboard input while uploading, so gpg will fail when it tries to prompt for a passphrase.\nYou can make this work by \"presetting\" your passphrase into the GPG agent instead (using the `gnupg-agent` package).\n\nPassphrase presetting is disabled in the agent's default configuration. Edit `~/.gnupg/gpg-agent.conf` (you'll probably \nneed to create this file) to add:\n\n```\nallow-preset-passphrase\n```\n\nRestart the agent to make it reload this config by running:\n\n```bash\ngpg-connect-agent reloadagent /bye\n```\n\nTo preset the passphrase for a key, we must first find out the key's \"keygrip\". To do this run:\n\n```bash\n# gpg2 --list-keys --with-keygrip\n\n/root/.gnupg/pubring.kbx\n------------------------\npub   rsa2048/142906AF 2019-01-21 [SC] [expires: 2029-01-18]\n      Keygrip = 22C54011D7772B7B23F2FBD0D69AAE74873BEAD0\nuid         [ultimate] SnapToS3Example \u003cn.sherlock@gmail.com\u003e\nsub   rsa2048/84B62A1E 2019-01-21 [E] [expires: 2029-01-18]\n      Keygrip = 04A78677528EE326B8AFEE81C4823B47F30DECDC\n```\n\nThe keygrip we need is the one for the \"[E]\" (encryption) subkey. Now use that keygrip to run:\n\n```bash\n/usr/lib/gnupg2/gpg-preset-passphrase --preset 04A78677528EE326B8AFEE81C4823B47F30DECDC\n``` \n\nThe utility will then wait for you to enter your passphrase and press enter. Note that your passphrase will be echoed \nout to the terminal in the clear (this utility doesn't replace it with stars to hide it).\n\nNow `snap-to-s3` should be able to successfully validate the upload of snapshots that use this key (until the system or \nGPG agent is restarted).\n\n### Validation where the private key never leaves your local machine\n\nYou may not want your private key to ever touch Amazon, or to ever be exported from your local hardware key storage \ndevice. This means that `snap-to-s3` will not be able to use that key to decrypt its uploaded archives for validation, \nso `--validate` will fail.\n\nOne way of solving this is to generate an additional keypair just for `snap-to-s3` to use. Cache the passphrase for\nsnap-to-s3's key using the instructions in the previous section. When migrating your snapshots,\nadd both your original keypair and the `snap-to-s3` keypair as recipients with `--gpg-recipient` (e.g. \n`--gpg-recipient SnapToS3Example --gpg-recipient \"Nicholas Sherlock\"`). Then once you're \nsatisfied that your snapshots have uploaded correctly using `--validate`, you can securely destroy the snap-to-s3 \nkeypair, leaving your original keypair as the only surviving key that can decrypt the snapshots. If you're not keen on \nthat solution, read on: \n\nWhen GPG encrypts an archive, it generates a random session key, which is used to encrypt the archive using symmetric\nencryption, then that session key is encrypted with the public key of the recipient and stored into the archive. \nFor decryption, the process is reversed: first the session key is decrypted with the corresponding private key. That's\nthe part that `snap-to-s3` cannot perform, since it doesn't have access to the private key.\n\nHowever, you can do that part of the process yourself manually on your local machine. First, use `snap-to-s3` to encrypt\nand upload your snapshot using `--migrate --keep-temp-volumes --gpg-recipient YourPublicKey` (no `--validate`). \nThen, on your local machine where your private key resides, you can decrypt the session key from the uploaded file like \nso:\n\n```bash \naws s3 cp \"s3://backups.example.com/vol-xxx/2019-01-21T00:16:59+00:00 snap-xxx.tar.lz4.gpg\" - \\\n\t| head -c 524288 \\\n\t| gpg2 --decrypt --show-session-key \\\n\t\u003e /dev/null\n\t\ngpg: encrypted with 4096-bit RSA key, ID 45BE6A42B05996C3, created 2018-08-08\n      \"Nicholas Sherlock \u003cn.sherlock@gmail.com\u003e\"\ngpg: session key: '9:D41D8DB13C9CA64F9E24C697973599B6B1E71BEFE8C7BAB41AC8FD97C4F14143'\ngpg: block_filter 0x00007f8fc1100020: read error (size=11277,a-\u003esize=11277)\ngpg: WARNING: encrypted message has been manipulated!\ngpg: block_filter: pending bytes!\n```\n\nThis command fetches the first 512kB of the archive, which is large enough that it should contain the encrypted session \nkey packet (since that appears at the start of the archive), then pipes that to GPG to decrypt and print out the session \nkey. GPG will prompt you for your passphrase (or your hardware key device) for decryption. (The decrypted archive output \nthat would normally be sent to stdout is piped to `/dev/null` to be discarded.) You can ignore the error messages, as \nthey are merely being triggered by the file being truncated by `head`. \n\nBack on your EC2 instance, you can now give that key to `snap-to-s3` for validation:\n\n```bash\nsnap-to-s3 --validate --bucket backups.example.com --snapshots snap-xxx --gpg-session-key \"9:D41D8DB13C9CA64F9E24C697973599B6B1E71BEFE8C7BAB41AC8FD97C4F14143\"\n```\n\nThis key allows `snap-to-s3` to decrypt the archive for verification, without having to give it access to your private \nGPG key.\n\nNote that because you can only supply a single session key to `snap-to-s3`, you can only validate a single snapshot at\na time using this method.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthenickdude%2Fsnap-to-s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthenickdude%2Fsnap-to-s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthenickdude%2Fsnap-to-s3/lists"}