{"id":33565819,"url":"https://github.com/bvdcode/octockup","last_synced_at":"2026-03-09T03:05:08.322Z","repository":{"id":264134940,"uuid":"892442759","full_name":"bvdcode/octockup","owner":"bvdcode","description":"All-in-one application for backup management with multiple sources","archived":false,"fork":false,"pushed_at":"2026-02-07T05:58:46.000Z","size":1499,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-07T15:54:05.014Z","etag":null,"topics":["autobackups","backup","dotnet","s3"],"latest_commit_sha":null,"homepage":"https://octockup.splidex.com","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bvdcode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":"COPYRIGHT","agents":null,"dco":null,"cla":null},"funding":{"ko_fi":"bvdcode"}},"created_at":"2024-11-22T05:49:29.000Z","updated_at":"2026-02-07T05:56:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"8eecd738-f4ca-4bcb-89b1-5dd9bdf45078","html_url":"https://github.com/bvdcode/octockup","commit_stats":null,"previous_names":["bvdcode/link2stream","bvdcode/octockup"],"tags_count":211,"template":false,"template_full_name":null,"purl":"pkg:github/bvdcode/octockup","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bvdcode%2Foctockup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bvdcode%2Foctockup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bvdcode%2Foctockup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bvdcode%2Foctockup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bvdcode","download_url":"https://codeload.github.com/bvdcode/octockup/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bvdcode%2Foctockup/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30281166,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T02:57:19.223Z","status":"ssl_error","status_checked_at":"2026-03-09T02:56:26.373Z","response_time":61,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autobackups","backup","dotnet","s3"],"created_at":"2025-11-28T03:07:04.105Z","updated_at":"2026-03-09T03:05:08.309Z","avatar_url":"https://github.com/bvdcode.png","language":"C#","funding_links":["https://ko-fi.com/bvdcode"],"categories":[],"sub_categories":[],"readme":"# Octockup\n\n[![License](https://badgen.net/github/license/bvdcode/octockup)](LICENSE)\n[![CI](https://github.com/bvdcode/octockup/actions/workflows/docker-image.yml/badge.svg)](https://github.com/bvdcode/octockup/actions)\n[![CodeFactor](https://www.codefactor.io/repository/github/bvdcode/octockup/badge)](https://www.codefactor.io/repository/github/bvdcode/octockup)\n[![Release](https://badgen.net/github/release/bvdcode/octockup?label=version)](https://github.com/bvdcode/octockup/releases)\n[![Docker Pulls](https://badgen.net/docker/pulls/bvdcode/octockup?icon=docker\u0026label=pulls)](https://hub.docker.com/r/bvdcode/octockup)\n[![Docker Image Size](https://badgen.net/docker/size/bvdcode/octockup?icon=docker\u0026label=size)](https://hub.docker.com/r/bvdcode/octockup)\n[![Github last-commit](https://img.shields.io/github/last-commit/bvdcode/octockup)](https://github.com/bvdcode/octockup/commits/main/)\n\n\u003e Live: [octockup.splidex.com](https://octockup.splidex.com)\n\nOctockup is an all-in-one client and server application for autobackup that includes both backend and frontend in a single Docker container. It allows you to gather and manage data from various sources, such as YouTube, SSH, FTP, Email, and more, directly through the browser.\n\n\u003cimg width=\"820\" height=\"319\" alt=\"image\" src=\"https://github.com/user-attachments/assets/9cdaf805-5b28-4a3c-8621-88b5dd847278\" /\u003e\n\n\n---\n\n## Key Features\n\n- **Containerization:** A single Docker container includes all necessary components.\n- **Backend and Frontend:** Full integration of backend and frontend for simplified deployment.\n- **Incremental Backups:** Save only the necessary changes with each backup.\n- **Connecting Various Sources:** You can connect YouTube, SSH, FTP, and many other sources to gather data.\n- **Web Interface:** User-friendly web interface for managing all application functions.\n- **Data Deduplication:** Efficient storage usage by avoiding duplicate data.\n- **Encryption:** All backup data is encrypted before leaving the server using AES-GCM.\n- **Dual Database Support:** Supports both SQLite (default) and PostgreSQL for flexible data management.\n\n---\n\n## Installation\n\nDockerhub: [Link](https://hub.docker.com/r/bvdcode/octockup)\n\n1. Make sure you have Docker and Docker Compose installed.\n2. Create `docker-compose.yml` file:\n\n```yaml\nservices:\n  octockup:\n    image: bvdcode/octockup:latest\n    ports:\n      - 8080:8080\n    environment:\n      # Required: 32 chars master key for encrypting everything\n      - OCTOCKUP_MASTER_KEY=${OCTOCKUP_MASTER_KEY}\n\n      # Optional: Allow multiple users (default: false)\n      - OCTOCKUP_ALLOW_MULTIUSER=false\n\n      # Optional: Use PostgreSQL instead of default SQLite\n      - OCTOCKUP_POSTGRES_HOST=postgres-server\n      - OCTOCKUP_POSTGRES_DB=octockup\n      - OCTOCKUP_POSTGRES_USER=octockup_client\n      - OCTOCKUP_POSTGRES_PASSWORD=${OCTOCKUP_DB_PASS}\n    volumes:\n      - /data/octockup:/app/data\n      # Mounts to backup if needed:\n      - /files:/app/data/mounts/files:ro\n      - /apps:/app/data/mounts/apps:ro\n```\n\n3. Start the application using Docker Compose:\n\n```bash\ndocker compose up -d\n```\n\n4. First login\n\n---\n\n## Usage\n\n1. Open your browser and navigate to the address where the application is running.\n2. Log in and set up connections to the necessary data sources (YouTube, SSH, FTP, etc.).\n3. Start gathering and managing data using the user-friendly web interface.\n\n---\n\n## Configuration\n\n### Configuration Files\n\n- **`docker-compose.yml`** - Docker Compose configuration for managing the container.\n\n### Environment variables\n\n```yaml\nMASTER_KEY: A 32-character master key for encrypting sensitive data in the database.\n```\n\n---\n\n## 🏗 Architecture Overview\n\n```mermaid\nflowchart LR\n    subgraph SRC[Backup Source]\n        S1[SFTP]\n        S2[IMAP]\n        S3[File System]\n        S4[Other]\n    end\n\n    subgraph CORE[Octockup Core]\n        B[Backup Job]\n        SNAP[Snapshot]\n        F[SnapshotFile]\n        CH[Chunker]\n        H[SHA-256 Hashing]\n        IDX[UploadedHash\u003cbr/\u003eGlobal Dedup Index]\n        ENC[AES-GCM Encryption]\n        CMP[Brotli Compression]\n    end\n\n    subgraph STOR[Backup Storage]\n        T1[S3]\n        T2[SFTP]\n        T3[Other Storage]\n    end\n\n    SRC --\u003e B\n    B --\u003e SNAP\n    SNAP --\u003e F\n    F --\u003e CH\n    CH --\u003e H\n    H --\u003e IDX\n    IDX --\u003e|exists| SNAP\n    H --\u003e|new| CMP --\u003e ENC --\u003e STOR\n\n```\n\n**High-level flow:**\n\n1. A **Source** provides a hierarchical list of files.\n2. A **Backup Job** scans files and creates a new **Snapshot**.\n3. Each file becomes a **SnapshotFile**.\n4. Files are split into fixed-size **chunks**.\n5. Each chunk is hashed (SHA-256).\n6. Hashes are checked against the global **UploadedHash** table.\n7. Only **missing chunks** are uploaded to the **Storage**.\n8. The Snapshot stores only **references to chunk hashes**, not raw data.\n\nThis allows:\n\n- true **block-level deduplication**\n- **incremental backups**\n- efficient storage usage across all snapshots\n\n---\n\n## 🧱 How Deduplication Works\n\nOctockup uses **content-addressed chunk-level deduplication**.\n\n1. Every file is split into fixed-size chunks.\n2. Each chunk is hashed using **SHA-256**.\n3. Before uploading a chunk, Octockup checks the `UploadedHash` table:\n\n   - `(ModuleId + Hash)` must be unique.\n\n4. If the hash already exists:\n\n   - the chunk is **NOT uploaded again**\n   - only a reference is stored in `SnapshotFile.ChunkHashes`.\n\n5. If the hash does not exist:\n\n   - the chunk is **compressed**\n   - **encrypted**\n   - uploaded to the storage\n   - recorded in `UploadedHash`.\n\nAs a result:\n\n- identical files across different backups are stored **only once**\n- even partial file changes reuse unchanged blocks\n- storage usage grows **only with truly new data**\n\nDeduplication works **globally per storage**, not only per snapshot.\n\n---\n\n## 🔐 How Encryption Works\n\nAll backup data is encrypted **before leaving the server**.\n\n1. A global **MASTER_KEY** (32 bytes) is provided via environment variable.\n2. Every chunk is encrypted using:\n\n   - **AES-GCM**\n   - streamed encryption (no full-buffer loading)\n\n3. The encryption pipeline is:\n\n```\nRaw Chunk → Brotli Compression → AES-GCM Encryption → Upload\n```\n\n4. Encryption is performed **on the fly during chunk streaming**.\n5. The storage backend **never sees plaintext data**.\n6. During download:\n\n   - encrypted chunks are fetched\n   - decrypted and decompressed in memory\n   - reassembled into the original file stream.\n\nSecurity guarantees:\n\n- All data at rest in storage is **encrypted**\n- Authentication is provided by **GCM tags**\n- Tampered chunks are **automatically rejected**\n- The storage provider **cannot read backups**\n\nThe master key is **never stored** in the database and must be backed up securely.\n\n---\n\n## Updating\n\nTo update to the latest version of the application, follow these steps:\n\n1. Update the image:\n   ```bash\n   docker compose pull\n   ```\n2. Restart the application:\n   ```bash\n   docker compose up -d\n   ```\n\n## Support\n\nIf you have any questions or issues, please create a new issue on GitHub or contact me via email:\n\noctockup-github-support@belov.us\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbvdcode%2Foctockup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbvdcode%2Foctockup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbvdcode%2Foctockup/lists"}