{"id":48446165,"url":"https://github.com/michaelgiuliano/s3-crypt-vault","last_synced_at":"2026-04-06T18:01:32.933Z","repository":{"id":344185646,"uuid":"1178123963","full_name":"michaelgiuliano/s3-crypt-vault","owner":"michaelgiuliano","description":"A secure client-side encrypted storage system for AWS S3.","archived":false,"fork":false,"pushed_at":"2026-04-05T15:38:39.000Z","size":38,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-05T17:18:15.504Z","etag":null,"topics":["aws","cli","cloud-security","cryptography","cybersecurity","devops","encryption","python","s3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michaelgiuliano.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-10T17:58:31.000Z","updated_at":"2026-04-05T15:36:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/michaelgiuliano/s3-crypt-vault","commit_stats":null,"previous_names":["michaelgiuliano/s3-crypt-vault"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/michaelgiuliano/s3-crypt-vault","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelgiuliano%2Fs3-crypt-vault","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelgiuliano%2Fs3-crypt-vault/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelgiuliano%2Fs3-crypt-vault/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelgiuliano%2Fs3-crypt-vault/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michaelgiuliano","download_url":"https://codeload.github.com/michaelgiuliano/s3-crypt-vault/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelgiuliano%2Fs3-crypt-vault/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31483380,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-06T17:22:55.647Z","status":"ssl_error","status_checked_at":"2026-04-06T17:22:54.741Z","response_time":112,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cli","cloud-security","cryptography","cybersecurity","devops","encryption","python","s3"],"created_at":"2026-04-06T18:01:31.420Z","updated_at":"2026-04-06T18:01:32.924Z","avatar_url":"https://github.com/michaelgiuliano.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# S3 Crypt Vault\n![Python](https://img.shields.io/badge/python-3.12-blue)\n![License](https://img.shields.io/badge/license-MIT-green)\n![CI](https://github.com/michaelgiuliano/s3-crypt-vault/actions/workflows/ci.yml/badge.svg)\n\n\u003e**A secure client-side encrypted storage system for AWS S3.**\n\n\n## Project Overview\n\nThis project implements a **secure storage system** based on client-side encryption and zero-knowledge principles.\n\nFiles are encrypted locally using **AES-256-GCM** with an envelope encryption model before being transmitted to AWS S3. Even if the S3 bucket is compromised, the stored data remains unreadable without the local encryption key.\n\nThe cloud provider never has access to plaintext data, ensuring that encryption and decryption always occur on the client side.\n\nThe system exposes two interfaces: a **CLI tool** for direct local usage and a **REST API** for programmatic access.\n\n\n## Architecture\n\nThe system is composed of multiple layers, each with a clear responsibility:\n\n- **CLI Layer (Typer)**  \n  Handles user interaction and command execution.\n\n- **API Layer (FastAPI)**\n  Exposes encrypted storage operations over HTTP.\n\n- **Vault Layer (`CryptVault`)**  \n  Orchestrates encryption and storage operations. Accepts injected dependencies — decoupled from both CLI and API concerns.\n\n- **Encryption Layer**  \n  Implements client-side encryption using AES-256-GCM and envelope encryption.\n\n- **Storage Layer (`S3Client`)**  \n  Handles communication with AWS S3 (or LocalStack in development).\n\n- **Error Handling Layer**  \n  Provides domain-specific exceptions mapped consistently to HTTP responses in the API layer.\n\n### Data Flow\n\n```\nUser Input (CLI or HTTP)\n ↓\nCLI / API Layer\n ↓\nVault (orchestration)\n ↓\nEncryption (client-side)\n ↓\nS3 Client\n ↓\nAWS S3\n```\n\nEncryption always happens locally before any data is transmitted to the cloud.\n\n\n## Security Model\n\nThis project follows a zero-knowledge architecture:\n\n### What it protects against\n\n- Compromise of the S3 bucket\n- Unauthorized access to stored objects\n- Data leakage from cloud provider\n\n### What it does NOT protect against\n\n- Loss or theft of the encryption key\n- Compromise of the local machine\n- Weak user key management practices\n\n### Key Responsibility\n\nThe user is fully responsible for securely storing the encryption key.\nIf the key is lost, the data cannot be recovered.\n\n\n## Features\n\n- **Client-side encryption** using *AES-256-GCM* with envelope encryption and password-based key derivation (`scrypt`).\n- **Zero-knowledge design** – the encryption key never leaves the local environment.\n- **REST API** (FastAPI) for programmatic access to encrypted storage operations.\n- Secure **AWS integration** using environment-based credentials.\n- Local development support with **LocalStack**.\n- Installable CLI tool (**`s3vault`**) for interacting with encrypted storage.\n- **Integration testing** using `pytest`, including full API test coverage via FastAPI `TestClient`.\n- **CI pipeline** with GitHub Actions.\n\n\n## Technical Stack\n\n- **Language**: Python 3.12+\n- **Cloud Provider**: AWS S3\n- **API Framework**: FastAPI + Uvicorn\n- **Infrastructure Emulation**: LocalStack + Docker Compose\n- **Automation**: GitHub Actions (CI)\n- **CLI Framework**: Typer\n- **Testing**: Pytest\n\n\n## Prerequisites\n\nBefore using the vault on AWS you must have:\n\n- AWS credentials with S3 permissions\n- An S3 bucket to store encrypted files\n- A local encryption key file (`master.key`)\n\nGenerate a key:\n```bash\ns3vault init-key\n```\n\nCreate a bucket:\n```bash\ns3vault create-bucket\n```\n\n\n## Installation\n\nClone the repository:\n```bash\ngit clone https://github.com/michaelgiuliano/s3-crypt-vault.git\ncd s3-crypt-vault\n```\n\nInstall in editable mode:\n```bash\npip install -e .\n```\n\nAfter installation the CLI command becomes available:\n```bash\ns3vault\n```\n\n### Dependency Management\n\nThis project uses `pip-tools` for reproducible dependency management.\n\nTo update dependencies:\n```bash\npip-compile requirements.in\npip-compile dev-requirements.in\n```\n\n\n## Environment Configuration\n\nCreate a `.env` file based on `.env.example`.\n\n```\nAWS_ACCESS_KEY_ID=your_access_key_here\nAWS_SECRET_ACCESS_KEY=your_secret_key_here\nAWS_REGION=eu-north-1\n\nS3_BUCKET_NAME=your-bucket-name-here\n\nKEY_PATH=master.key\n\nUSE_LOCALSTACK=true\n```\n\nWhen `USE_LOCALSTACK=true`, both the CLI and API connect to a local S3 emulator instead of AWS.\n\n\n## Local Development (LocalStack)\n\nStart the local S3 environment:\n\n```bash\ndocker-compose up -d\n```\n\nLocalStack emulates the entire AWS S3 workflow locally without creating real cloud resources.\n\n\n## CLI Usage\n\u003e Note: Upload and download operations require a password for encryption/decryption.\n\n### Setup\n\n```bash\n# Generate a new encryption key:\ns3vault init-key\n\n# Create the configured S3 bucket:\ns3vault create-bucket\n\n# List available buckets:\ns3vault list-buckets\n```\n\n### Operations\n\n```bash\n# Upload a file (encrypted locally before upload):\ns3vault upload secret.txt\n\n# List encrypted files stored in the configured bucket:\ns3vault list-files\n\n# Download and decrypt a file:\ns3vault download secret.txt.enc decrypted.txt\n```\n\n\n## API Usage\n\nStart the API server:\n```bash\nuvicorn app.api.main:app --reload\n```\n\nInteractive documentation is available at `http://localhost:8000/docs`.\n\n### Endpoints\n\n#### Health check\n```bash\ncurl http://localhost:8000/health\n```\n```json\n{\"status\": \"ok\"}\n```\n\n#### List encrypted files\n```bash\ncurl http://localhost:8000/files\n```\n```json\n{\"files\": [\"secret.txt.enc\"]}\n```\n\n#### Upload a file\n```bash\ncurl -X POST http://localhost:8000/files \\\n  -F \"file=@secret.txt\" \\\n  -F \"password=your-password\"\n```\n```json\n{\"object_key\": \"secret.txt.enc\"}\n```\n\nReturns `409` if the file already exists.\n\n#### Download and decrypt a file\n```bash\ncurl -X POST http://localhost:8000/files/secret.txt.enc/download \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"password\": \"your-password\"}' \\\n  --output decrypted.txt\n```\n\nReturns `400` if the password is wrong, `404` if the file does not exist.\n\n\n## Encryption Architecture\n\nAll files are **encrypted locally** before being uploaded to S3.\n\nThe project uses **AES-256-GCM**, an authenticated encryption mode that ensures:\n\n- **Confidentiality** – encrypted data cannot be read without the key\n- **Integrity** – tampering with ciphertext is detected\n- **Authenticity** – modified data cannot be successfully decrypted\n\n### v2 Encryption Model (Envelope Encryption)\n\nStarting from version 0.2.0, the vault uses an envelope encryption scheme:\n\n- A **Data Encryption Key (DEK)** is generated per file\n- A **master key** is derived from a user password using `scrypt`\n- The file is encrypted with the DEK (AES-256-GCM)\n- The DEK is encrypted with the master key\n\nThis provides key isolation per file and eliminates any persistent master key stored on disk.\n\n### Encryption Flow\n\n```\nFile is read locally\n        ↓\nRandom DEK generated per file\n        ↓\nAES-256-GCM encrypts the file using the DEK\n        ↓\nDEK is encrypted with the password-derived master key\n        ↓\nStructured binary blob (SCV2 format) uploaded to S3\n```\n\n### Tamper Detection\n\n*AES-GCM* provides **built-in authentication**. If even a single bit of ciphertext is modified, decryption will fail with an authentication error. This behavior is verified by the automated test `test_tamper_detection()`.\n\n\n## Testing\n\nRun the full test suite:\n```bash\npytest tests/\n```\n\nTests cover:\n\n- Encryption and decryption correctness\n- Tamper detection\n- S3 integration using LocalStack\n- End-to-end vault workflow\n- Full API integration tests (upload, download, listing, error cases)\n\n\n## Continuous Integration\n\n**GitHub Actions** automatically runs:\n\n- Lint checks (`flake8`)\n- Integration tests (`pytest`)\n- LocalStack environment\n\nOn every `push` and `pull` request.\n\n\n## Project Structure\n\n```\ns3-crypt-vault/\n├───.github/\n│   └───workflows/\n│       └───ci.yml\n├───app/\n│   ├───api/\n│   │   ├───__init__.py\n│   │   ├───dependencies.py\n│   │   ├───exceptions.py\n│   │   ├───main.py\n│   │   ├───routes.py\n│   │   └───schemas.py\n│   ├───crypto/\n│   │   ├───__init__.py\n│   │   ├───envelope.py\n│   │   └───kdf.py\n│   ├───__init__.py\n│   ├───cli.py\n│   ├───config.py\n│   ├───encryptor.py\n│   ├───exceptions.py\n│   ├───s3_client.py\n│   └───vault.py\n├───tests/\n│   ├───__init__.py\n│   ├───test_api.py\n│   ├───test_encryption.py\n│   ├───test_s3_client.py\n│   ├───test_s3_connection.py\n│   └───test_vault.py\n├───.env.example\n├───.gitignore\n├───CHANGELOG.md\n├───dev-requirements.in\n├───dev-requirements.txt\n├───docker-compose.yml\n├───LICENSE\n├───pyproject.toml\n├───README.md\n├───requirements.in\n└───requirements.txt\n```\n\n## License\n\nThis project is licensed under the MIT License.\n\nYou are free to use, modify, and distribute this software with proper attribution. See the LICENSE file for the full license text.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelgiuliano%2Fs3-crypt-vault","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichaelgiuliano%2Fs3-crypt-vault","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelgiuliano%2Fs3-crypt-vault/lists"}