{"id":35066375,"url":"https://github.com/caasmo/restinpieces-sqlite-backup","last_synced_at":"2026-05-20T22:10:19.398Z","repository":{"id":301784034,"uuid":"1010297661","full_name":"caasmo/restinpieces-sqlite-backup","owner":"caasmo","description":"push pull sqlite backup for restinpieces","archived":false,"fork":false,"pushed_at":"2025-06-28T20:12:23.000Z","size":21,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-06-28T20:32:48.236Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/caasmo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-28T19:13:39.000Z","updated_at":"2025-06-28T20:12:26.000Z","dependencies_parsed_at":"2025-07-05T09:31:50.939Z","dependency_job_id":null,"html_url":"https://github.com/caasmo/restinpieces-sqlite-backup","commit_stats":null,"previous_names":["caasmo/restinpieces-sqlite-backup"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/caasmo/restinpieces-sqlite-backup","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caasmo%2Frestinpieces-sqlite-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caasmo%2Frestinpieces-sqlite-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caasmo%2Frestinpieces-sqlite-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caasmo%2Frestinpieces-sqlite-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/caasmo","download_url":"https://codeload.github.com/caasmo/restinpieces-sqlite-backup/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caasmo%2Frestinpieces-sqlite-backup/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28077914,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-27T02:00:05.897Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-27T11:06:08.864Z","updated_at":"2025-12-27T11:07:45.633Z","avatar_url":"https://github.com/caasmo.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# restinpieces-sqlite-backup\n\n`restinpieces-sqlite-backup` provides backup functionality for SQLite databases within the `restinpieces` framework. It uses a push-pull design, consisting of a server-side job to create local backups and a client-side binary to retrieve them.\n\nThis handler supports two distinct backup strategies (`vacuum` and `online`) to accommodate different production workloads. See the \"Backup Strategies\" section for a detailed comparison to help you choose the right one for your needs.\n\n## Architecture\n\nThe solution is composed of two main components:\n\n1.  **The Backup Job (Server-side)**: This component runs as a job within a `restinpieces` application. It periodically creates a compressed backup of a specified SQLite database and stores it locally on the server. An example of how to integrate the job can be found in **[cmd/example](https://github.com/caasmo/restinpieces-sqlite-backup/tree/master/cmd/example)**.\n\n2.  **The Pull Client (Client-side)**: This is a standalone command-line binary that can be run on any client machine. It connects to the server via SFTP, finds the latest backup by filename, downloads it, and verifies its integrity. A reference implementation is available in **[cmd/client](https://github.com/caasmo/restinpieces-sqlite-backup/tree/master/cmd/client)**.\n\nThis design decouples backup creation from retrieval, allowing backups to be pulled from a central server to any number of client machines.\n\n## Features\n\n-   **Flexible Backup Strategies**: Choose between a fast, locking `vacuum` strategy or a non-locking `online` strategy.\n-   **Push-Pull Design**: Decouples backup creation (server-side) from retrieval (client-side).\n-   **Gzip Compression**: Compresses backup files.\n-   **Descriptive Filenames**: Embeds the database name, timestamp, and strategy into filenames (e.g., `app-2025-07-01T10-30-00Z-vacuum.bck.gz`), which are used to determine the latest backup.\n-   **SFTP Client**: A client is provided to pull backups from a remote server.\n-   **Backup Verification**: The client verifies the integrity of downloaded backups using `PRAGMA integrity_check`.\n\n## Installation\n\nTo use this package, you need a working Go environment (Go 1.24.2 or later). You can add it to your project using `go get`:\n\n```bash\ngo get github.com/caasmo/restinpieces-sqlite-backup\n```\n\n## Deployment Workflow\n\nDeploying this add-on involves three main steps: configuring the job, inserting the recurrent job into the database, and running the main application.\n\n1.  **Configure the Backup Job**: First, you must add the backup configuration (in TOML format) to the `restinpieces` database using its secure `ConfigStore`. The configuration should be stored under the scope `sqlite_backup`. See the \"Backup Strategies\" section for details on which `strategy` to choose.\n    ```toml\n    # Example for the 'online' strategy\n    source_path = \"/path/to/your/database.db\"\n    backup_dir = \"/path/to/your/backups\"\n    strategy = \"online\"\n    pages_per_step = 200\n    sleep_interval = \"20ms\"\n    ```\n\n2.  **Insert the Recurrent Job**: Use the `insert-job` tool provided in this repository to create the recurrent job entry in the database. This only needs to be done once.\n    ```bash\ngo build ./cmd/insert-job\n./insert-job \\\n  -dbpath /path/to/restinpieces.db \\\n  -interval 24h \\\n  -scheduled 2025-07-01T10:00:00Z\n    ```\n\n3.  **Run the Application**: Start your main `restinpieces` application. It will load the configuration, register the backup handler, and automatically start executing the backup job at its scheduled time.\n\n## Backup Strategies\n\nChoosing the correct backup strategy is critical for ensuring your application remains performant.\n\n### `online` (Default Strategy)\n\nThis strategy uses SQLite's built-in Online Backup API.\n\n-   **Pros:**\n    -   **Non-Locking:** It copies the database page-by-page, yielding between steps. It does **not** block application writers for any significant length of time, making it safe for 24/7, high-traffic applications.\n-   **Cons:**\n    -   **Slower:** The total backup time is longer than `vacuum` due to the overhead of the incremental copy.\n    -   **Direct Copy:** The resulting file is a direct copy of the source, including any fragmentation.\n-   **When to use it:**\n    -   **Recommended for most production systems.**\n    -   Databases with high, unpredictable, or 24/7 write workloads.\n    -   When application availability is more important than raw backup speed.\n\n### `vacuum`\n\nThis strategy uses the `VACUUM INTO` command to create the backup.\n\n-   **Pros:**\n    -   **Fast:** It's the quickest way to get a backup.\n    -   **Defragmented:** The resulting backup file is clean and unfragmented, making it slightly smaller and faster to restore.\n-   **Cons:**\n    -   **Locks Writers:** This is the major drawback. It places a read lock on the source database, **blocking all write operations** for the entire duration of the backup.\n-   **When to use it:**\n    -   Databases with low write activity.\n    -   During scheduled maintenance or predictable off-peak hours where a brief write-pause is acceptable.\n    -   When you need a defragmented copy for analytical purposes.\n\n### Configuration Parameters\n\nThe `online` strategy can be tuned with the following parameters in your TOML config:\n\n-   `pages_per_step` (integer, default: `100`): How many pages to copy in a single step. A smaller value is \"politer\" to other connections but increases overhead.\n-   `sleep_interval` (duration, default: `\"10ms\"`): How long to pause between steps to yield system resources. A value of `\"0s\"` will run the backup as fast as possible, while a higher value will reduce its CPU/IO impact.\n\n## Tools and Examples\n\nThis repository contains several `cmd` utilities that serve as tools and examples.\n\n-   **[cmd/example](https://github.com/caasmo/restinpieces-sqlite-backup/tree/master/cmd/example)**: A fully working example of a `restinpieces` server that registers and runs the backup handler. This is the primary reference for integrating the handler into your own application.\n\n-   **[cmd/generate-blueprint-config](https://github.com/caasmo/restinpieces-sqlite-backup/tree/master/cmd/generate-blueprint-config)**: A simple tool that prints a template TOML configuration file to the console. This is useful for getting started with the configuration.\n    ```bash\ngo run ./cmd/generate-blueprint-config\n    ```\n\n-   **[cmd/insert-job](https://github.com/caasmo/restinpieces-sqlite-backup/tree/master/cmd/insert-job)**: The command-line tool used to insert the recurrent backup job into the database. See the \"Deployment Workflow\" section for usage details.\n\n-   **[cmd/client](https://github.com/caasmo/restinpieces-sqlite-backup/tree/master/cmd/client)**: An example of a client-side binary that connects to the server via SFTP to pull the latest backup. This can be adapted to your specific needs for retrieving backups.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaasmo%2Frestinpieces-sqlite-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaasmo%2Frestinpieces-sqlite-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaasmo%2Frestinpieces-sqlite-backup/lists"}