{"id":35066346,"url":"https://github.com/caasmo/restinpieces-litestream","last_synced_at":"2026-04-20T20:36:37.172Z","repository":{"id":288153645,"uuid":"967008030","full_name":"caasmo/restinpieces-litestream","owner":"caasmo","description":"litestream backup daemon for restinpieces framework","archived":false,"fork":false,"pushed_at":"2025-12-17T15:21:12.000Z","size":170,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-29T01:02:56.893Z","etag":null,"topics":["backup","golang","litestream","restinpieces","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/caasmo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-04-15T19:32:10.000Z","updated_at":"2025-12-17T15:26:57.000Z","dependencies_parsed_at":"2025-04-15T20:54:31.585Z","dependency_job_id":null,"html_url":"https://github.com/caasmo/restinpieces-litestream","commit_stats":null,"previous_names":["caasmo/restinpieces-litestream"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/caasmo/restinpieces-litestream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caasmo%2Frestinpieces-litestream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caasmo%2Frestinpieces-litestream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caasmo%2Frestinpieces-litestream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caasmo%2Frestinpieces-litestream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/caasmo","download_url":"https://codeload.github.com/caasmo/restinpieces-litestream/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caasmo%2Frestinpieces-litestream/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32065165,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T11:35:06.609Z","status":"ssl_error","status_checked_at":"2026-04-20T11:34:48.899Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","golang","litestream","restinpieces","sqlite"],"created_at":"2025-12-27T11:05:34.949Z","updated_at":"2026-04-20T20:36:37.038Z","avatar_url":"https://github.com/caasmo.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# restinpieces-litestream\n\nThis package allows you to use Litestream's continuous backup capabilities \"in-process\" with your [restinpieces](https://github.com/caasmo/restinpieces) application. It removes the need to manage a separate Litestream binary, systemd service, and standalone configuration files.\n\nInstead, it provides a `restinpieces` daemon that is compiled into your application binary and integrates directly with the framework's lifecycle and secure configuration store.\n\n## Quickstart Example\n\nThis section provides a step-by-step guide to get the example application running and see the backup/restore process in action. The example application itself can be found in [cmd/example/main.go](./cmd/example/main.go).\n\n### Step 0: Initialize Your Application Database\n\nThis package integrates with an existing `restinpieces` application. Before you begin, you must initialize the main application database (`app.db`) with the necessary tables.\n\nYou can do this using the `ripc` command-line tool from the [restinpieces framework](https://github.com/caasmo/restinpieces).\n```bash\n# This command creates a new app.db file with the required framework tables.\nripc -dbpath app.db app create\n```\n\n### Step 1: Configure and Store Litestream Settings\n\nFirst, you need to provide a Litestream configuration file. This package uses the standard `litestream.yml` format.\n\n1.  **Create `litestream.yml`:**\n    Create a `litestream.yml` file with your desired backup replica settings. For example, to back up to a local directory:\n    ```yaml\n    dbs:\n      - path: ./app.db\n        replicas:\n          - type: file\n            path: ./litestream-replicas\n    ```\n    For more details and other replica types like S3, see the [official Litestream configuration documentation](https://litestream.io/reference/config/).\n\n2.  **Store the Configuration:**\n    The `restinpieces` framework stores all configuration securely inside the application database. Use the `ripc` command-line tool (provided by the `restinpieces` project) to encrypt and save your `litestream.yml`.\n    ```bash\n    # This command assumes you have an age key and the ripc tool.\n    ripc -age-key age_key.txt -dbpath app.db config save -scope litestream litestream.yml\n    ```\n    This command saves the configuration under the `litestream` scope, which the daemon will use to load its settings on startup.\n\n### Step 2: Build the Example Application\n\nThis repository includes an example application that demonstrates how to integrate the Litestream daemon.\n\nBuild the application using the standard Go toolchain:\n```bash\ngo build -o myapp ./cmd/example\n```\nThis compiles the code in `cmd/example` and creates an executable file named `myapp`.\n\n### Step 3: Run the Application\n\nNow, run the compiled application, providing the same `age_key.txt` and `dbpath` you used to save the configuration:\n```bash\n./myapp -age-key age_key.txt -dbpath app.db\n```\nThis will start the main web server and, alongside it, the Litestream daemon will begin monitoring `app.db` and replicating any changes to your configured destination. You should see log output from both the application and Litestream indicating that they have started.\n\nWhile the app is running, you can make changes to the database (e.g., by using `sqlite3 app.db \"CREATE TABLE t(id INT);\"`) and you will see Litestream backing them up.\n\n**Note on Privileged Ports:** If your application is configured to listen on a privileged port (any port below 1024, like 80 or 443), you will need to grant the executable special capabilities to bind to it without running as root. You can do this with the `setcap` command:\n```bash\nsudo setcap cap_net_bind_service=+ep ./myapp\n```\n\n### Step 4: Perform a Local Restore\n\nTo simulate a disaster recovery scenario, you can use the official `litestream` binary to restore the database from your replica. This is useful for migrating a database, recovering from hardware failure, or testing your backup integrity.\n\n1.  **Install Litestream:**\n    If you don't have it, [install the Litestream binary](https://litestream.io/installation).\n\n2.  **Stop your application:**\n    Stop the running `./myapp` process (`Ctrl+C`). You can delete the local database files (e.g., `app.db`, `app.db-wal`) to simulate a complete data loss.\n\n3.  **Restore the database:**\n    The restore command depends on how your `litestream.yml` is configured.\n\n    **Scenario A: Simple Single Database**\n\n    If your `litestream.yml` specifies a single database with a `path` field, you can restore it by pointing to the config and specifying the database path.\n    ```bash\n    # Given a litestream.yml that contains:\n    # dbs:\n    #   - path: ./app.db\n    #     replicas: ...\n\n    # Restore command:\n    litestream restore -o app.db -config litestream.yml ./app.db\n    ```\n    This tells Litestream to find the configuration for `./app.db` in the `litestream.yml` file and restore it to a new file named `app.db`.\n\n    **Scenario B: Directory Monitoring**\n\n    If you are using Litestream v0.5.0's directory monitoring feature, your `litestream.yml` specifies a `dir` instead of a `path`.\n    ```yaml\n    # Given a litestream.yml that contains:\n    # dbs:\n    #   - dir: ./data/\n    #     replicas: ...\n    ```\n    In this case, Litestream is backing up every database inside the `./data/` directory. To restore, you must specify the **exact path** of the database you want to recover. For example, to restore a database named `customer1.db` that was in that directory:\n    ```bash\n    # Restore a specific DB from the monitored directory\n    litestream restore -o customer1.restored.db -config litestream.yml ./data/customer1.db\n    ```\n    This command looks at `litestream.yml`, finds the configuration for the `./data/` directory, and then uses that to find the correct replica for `./data/customer1.db` and restores it to `customer1.restored.db`.\n\nAfter the restore is complete, you can restart your application, and it will have all its data back.\n\n## Configuration\n\nThis package uses the standard `litestream.yml` configuration format. Litestream configuration is managed securely through the restinpieces `SecureConfigStore`, as shown in the Quickstart guide.\n\nFor more information on `ripc`, see the [`ripc` documentation](https://github.com/caasmo/restinpieces/blob/master/doc/ripc.md).\n\n**Note: No Environment Variables**\nA key principle of the `restinpieces` framework is that all configuration must be self-contained and securely stored in the database to create a single, auditable source of truth. Therefore, this package **does not support environment variable expansion** (e.g., `$HOME` or `${VAR}`) within the `litestream.yml` file.\n\nTo enforce this, a validation check runs on startup. The presence of environment variable syntax will cause the application to fail, **even if the variables are inside comments**. Please ensure your configuration contains only explicit paths and values.\n\n### Deactivated Upstream Features\n\nThe upstream Litestream project includes several features designed for when it is run as a standalone binary. When using `restinpieces-litestream` as an embedded Go library, some of these features are not suitable and have been deactivated. Specifically, configuration options related to the **`exec` subcommand**, the **MCP server**, and **Prometheus metrics** are not supported.\n\n## Logging\n\nThe upstream Litestream project is designed primarily as a standalone binary. This architecture makes it difficult to cleanly inject a custom `slog.Logger` when using Litestream as an embedded library, as its internal components fall back to a global default logger.\n\nTo solve this without requiring a heavily modified and hard-to-maintain fork, this package uses a \"split-logging\" model. Our compromise is to expose Litestream's own logging configuration, allowing us to control its output separately from the main framework logger.\n\nThis package follows a **\"split-logging\"** model:\n\n1.  **Framework Logs:** All logs generated by the `restinpieces-litestream` daemon wrapper itself are sent to the main framework's `slog.Logger` instance. If you are using the default `restinpieces` setup, these logs will be written to the SQLite database with full structured context.\n\n2.  **Internal Litestream Logs:** All internal logs from the core Litestream components (e.g., relating to replication, snapshots, compaction) are sent to either **`os.Stdout`** (the default) or **`os.Stderr`**. This behavior mimics running Litestream as a separate binary alongside the main application.\n\n### Configuring Internal Logs\n\nThe destination, format, and level of these internal logs can be controlled via the `litestream.yml` configuration file. This is handled by a function from our Litestream fork that configures the library's internal global logger.\n\nExample `litestream.yml` logging section:\n```yaml\nlogging:\n  # 'debug', 'info', 'warn', or 'error'\n  level: 'info' \n  # 'text' or 'json'\n  type: 'text'\n  # Direct logs to stderr instead of stdout. Defaults to false.\n  stderr: false\n```\nThis configuration would cause the internal Litestream logs to be written to `os.Stdout` as text at the `INFO` level, while the main framework logs continue to go to their own destination.\n\n## Driver Compatibility\n\nAs of v0.5.0, the underlying [Litestream library](https://github.com/benbjohnson/litestream) uses the excellent [modernc.org/sqlite](https://modernc.org/sqlite) driver, which is **pure-Go**.\n\nThis is a major advantage as it means this package has **no CGO dependency**.\n*   No C compiler or external dependencies are needed to build your application.\n*   Cross-compilation is simple.\n*   There are no conflicts with any other SQLite drivers (whether pure-Go or CGO-based) that your main application might use.\n\nThis aligns perfectly with the `restinpieces` framework, whose default database driver is [zombiezen.com/go/sqlite](https://zombiezen.com/go/sqlite), which is also a pure-Go driver. This ensures a CGO-free environment when using the default settings for both the framework and this package.\n\n## SQLite PRAGMAs for Litestream\n\nConsider setting the following PRAGMAs in your application when initializing the database connection for optimal performance and compatibility with Litestream:\n\nhttps://litestream.io/tips/\n\nDisable autocheckpoints for high write load servers.  Prevent aplication to do\ncheckpoints:\n\n    PRAGMA wal_autocheckpoint = 0;\n\nThis mode will ensure that the fsync() calls are only called when the WAL\nbecomes full and has to checkpoint to the main database file. This is safe as\nthe WAL file is append only. Should be probably default when using WAL:\n\n    PRAGMA synchronous = NORMAL;\n\nLitestream requires periodic but short write locks on the database when\ncheckpointing occurs. SQLite will return an error by default if your\napplication tries to obtain a write lock at the same time.\nThis pragma will wait up to a given number of milliseconds before failing a\nquery if it is blocked on a write.\n\n    PRAGMA busy_timeout = 5000;\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaasmo%2Frestinpieces-litestream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaasmo%2Frestinpieces-litestream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaasmo%2Frestinpieces-litestream/lists"}