{"id":21972814,"url":"https://github.com/agustinsrg/encrypted-storage","last_synced_at":"2025-03-22T23:14:53.083Z","repository":{"id":181297203,"uuid":"666535904","full_name":"AgustinSRG/encrypted-storage","owner":"AgustinSRG","description":"Collection of tools to create an encrypted storage (golang)","archived":false,"fork":false,"pushed_at":"2025-02-23T15:53:58.000Z","size":19,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-23T16:31:55.326Z","etag":null,"topics":["encryption","golang","library","storage"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/AgustinSRG/encrypted-storage","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AgustinSRG.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-14T19:20:08.000Z","updated_at":"2025-02-23T15:54:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"257c3490-463d-4c2c-815c-d84b64fddeab","html_url":"https://github.com/AgustinSRG/encrypted-storage","commit_stats":{"total_commits":8,"total_committers":1,"mean_commits":8.0,"dds":0.0,"last_synced_commit":"1fe0cbadc469983df00cecac538a3ea21a26647e"},"previous_names":["agustinsrg/encrypted-storage"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AgustinSRG%2Fencrypted-storage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AgustinSRG%2Fencrypted-storage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AgustinSRG%2Fencrypted-storage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AgustinSRG%2Fencrypted-storage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AgustinSRG","download_url":"https://codeload.github.com/AgustinSRG/encrypted-storage/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245031516,"owners_count":20549926,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["encryption","golang","library","storage"],"created_at":"2024-11-29T15:21:25.644Z","updated_at":"2025-03-22T23:14:53.066Z","avatar_url":"https://github.com/AgustinSRG.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Encrypted storage tools\n\nThis library implements a collection of tools to create an encrypted storage:\n\n- Functions to encrypt and decrypt, using `AES-256`, with the option to compress the data using `ZLIB`.\n- Read and write streams to create and read encrypted files in chunks.\n- Read and write streams to pack multiple small encrypted files into a single container file.\n\n[Documentation](https://pkg.go.dev/github.com/AgustinSRG/encrypted-storage)\n\n## Installation\n\nIn order to add it to your project, use\n\n```sh\ngo get github.com/AgustinSRG/encrypted-storage\n```\n\n## File encryption\n\nYou can encrypt a buffer of data using `EncryptFileContents`, with a key of 64 bytes.\n\nYou can then decrypt it using `DecryptFileContents` and the same key.\n\n[Example](./file_encrypt_test.go)\n\n### Details\n\nThe encrypted data returned by `EncryptFileContents` and accepted by `DecryptFileContents` is binary-encoded, with the following structure:\n\n| Starting byte | Size (bytes) | Value name   | Description                                                                                                    |\n| ------------- | ------------ | ------------ | -------------------------------------------------------------------------------------------------------------- |\n| `0`           | `2`          | Algorithm ID | Identifier of the algorithm, stored as a **Big Endian unsigned integer**                                       |\n| `2`           | `H`          | Header       | Header containing any parameters required by the encryption algorithm. The size depends on the algorithm used. |\n| `2 + H`       | `N`          | Body         | Body containing the raw encrypted data. The size depends on the initial unencrypted data and algorithm used.   |\n\nThe system is flexible enough to allow multiple encryption algorithms. Currently, there are 2 supported ones:\n\n- `AES256_ZIP`: ID = `1`, Uses ZLIB ([RFC 1950](https://datatracker.ietf.org/doc/html/rfc1950)) to compress the data, and then uses AES with a key of 256 bits to encrypt the data, CBC as the mode of operation and an IV of 128 bits. This algorithm uses a header of 20 bytes, containing the following fields:\n\n| Starting byte | Size (bytes) | Value name                | Description                                                        |\n| ------------- | ------------ | ------------------------- | ------------------------------------------------------------------ |\n| `0`           | `4`          | Compressed plaintext size | Size of the compressed plaintext, in bytes, used to remove padding |\n| `4`           | `16`         | IV                        | Initialization vector for AES_256_CBC algorithm                    |\n\n- `AES256_FLAT`: ID = `2`, Uses AES with a key of 256 bits to encrypt the data, CBC as the mode of operation and an IV of 128 bits. This algorithm uses a header of 20 bytes, containing the following fields:\n\n| Starting byte | Size (bytes) | Value name     | Description                                             |\n| ------------- | ------------ | -------------- | ------------------------------------------------------- |\n| `0`           | `4`          | Plaintext size | Size of the plaintext, in bytes, used to remove padding |\n| `4`           | `16`         | IV             | Initialization vector for AES_256_CBC algorithm         |\n\n## Block-Encrypted Files\n\nBlock encrypted files are used to encrypt an arbitrarily large file, splitting it's contents in blocks (or chunks) with a set max size. Each block is then encrypted using the file encryption method detailed above.\n\nFor creating / writing files:\n\n- You can create a file using `CreateFileBlockEncryptWriteStream`, a function that returns a new instance of `FileBlockEncryptWriteStream`.\n- After it's creation, you must call `FileBlockEncryptWriteStream.Initialize` to set the file size, the block size and the encryption key.\n- Once it is initialized, you may call `FileBlockEncryptWriteStream.Write` to write data into the file. When the data reached a block limit, that block is encrypted and stored into the file.\n- After you wrote all the data, you must call `FileBlockEncryptWriteStream.Close` to close the file.\n\nFor reading files:\n\n- You can open a file calling `CreateFileBlockEncryptReadStream`, a function that returns an instance of `FileBlockEncryptReadStream`\n- After it's opened, you may call `FileBlockEncryptReadStream.FileSize`, `FileBlockEncryptReadStream.BlockSize` or `FileBlockEncryptReadStream.BlockCount` to retrieve the parameters of the file.\n- You may call `FileBlockEncryptReadStream.Read` to decrypt and read the data.\n- You can call `FileBlockEncryptReadStream.Seek` to change the cursor position. You may also call `FileBlockEncryptReadStream.Cursor` to retrieve the cursor position if needed.\n- After you are done, you must call `FileBlockEncryptReadStream.Close` to close the file.\n\n[Example](./file_block_encrypt_test.go)\n\n### Details\n\nThey are binary files consisting of 3 contiguous sections: The header, the chunk index and the encrypted chunks.\n\nThe header contains the following fields:\n\n| Starting byte | Size (bytes) | Value name       | Description                                                                      |\n| ------------- | ------------ | ---------------- | -------------------------------------------------------------------------------- |\n| `0`           | `8`          | File size        | Size of the original file, in bytes, stored as a **Big Endian unsigned integer** |\n| `8`           | `8`          | Chunk size limit | Max size of a chunk, in bytes, stored as a **Big Endian unsigned integer**       |\n\nAfter the header, the chunk index is stored. **For each chunk** the file was split into, the chunk index will store a metadata entry, withe the following fields:\n\n| Starting byte | Size (bytes) | Value name    | Description                                                              |\n| ------------- | ------------ | ------------- | ------------------------------------------------------------------------ |\n| `0`           | `8`          | Chunk pointer | Starting byte of the chunk, stored as a **Big Endian unsigned integer**  |\n| `8`           | `8`          | Chunk size    | Size of the chunk, in bytes, stored as a **Big Endian unsigned integer** |\n\nAfter the chunk index, the encrypted chunks are stored following the same structure described above.\n\nThis chunked structure allows to randomly access any point in the file as a low cost, since you don't need to decrypt the entire file, only the corresponding chunks.\n\n## Multi-File Pack\n\nMulti-file pack container files are used to store multiple small files inside a single container.\n\nFor creating / writing files:\n\n- You can create a file by calling `CreateMultiFilePackWriteStream`, a function that returns an instance of `MultiFilePackWriteStream`\n- You must call `MultiFilePackWriteStream.Initialize`, setting the number of files you want to store\n- You may call `MultiFilePackWriteStream.PutFile` for each file you want to store, in order.\n- After all files are written, you must call `MultiFilePackWriteStream.Close` to close the file.\n\nFor reading files:\n\n- You can open a file by calling `CreateMultiFilePackReadStream`, a function that returns an instance of `MultiFilePackReadStream`\n- You may call `MultiFilePackReadStream.FileCount` to retrieve the number of stored files.\n- You may call `MultiFilePackReadStream.GetFile` to read a file, by its index.\n- After you are done, you must call `MultiFilePackReadStream.Close` to close the file.\n\n### Details\n\nThey are binary files consisting of 3 contiguous sections: The header, the file table and the encrypted files.\n\nThe header contains the following fields:\n\n| Starting byte | Size (bytes) | Value name | Description                                                                      |\n| ------------- | ------------ | ---------- | -------------------------------------------------------------------------------- |\n| `0`           | `8`          | File count | Number of files stored by the asset, stored as a **Big Endian unsigned integer** |\n\nAfter the header, a file table is stored. **For each file** stored by the asset, a metadata entry is stored, with the following fields:\n\n| Starting byte | Size (bytes) | Value name        | Description                                                                           |\n| ------------- | ------------ | ----------------- | ------------------------------------------------------------------------------------- |\n| `0`           | `8`          | File data pointer | Starting byte of the file encrypted data, stored as a **Big Endian unsigned integer** |\n| `8`           | `8`          | File size         | Size of the encrypted file, in bytes, stored as a **Big Endian unsigned integer**     |\n\nAfter the file table, each file is stored following the same structure described above.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagustinsrg%2Fencrypted-storage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fagustinsrg%2Fencrypted-storage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagustinsrg%2Fencrypted-storage/lists"}