{"id":15075232,"url":"https://github.com/techrah/sqlite3-compression-encryption-vfs","last_synced_at":"2026-01-18T06:13:28.449Z","repository":{"id":52955253,"uuid":"62763602","full_name":"techrah/sqlite3-compression-encryption-vfs","owner":"techrah","description":"Compression and Encryption Virtual File System for SQLite 3.","archived":false,"fork":false,"pushed_at":"2021-07-25T00:28:11.000Z","size":112,"stargazers_count":90,"open_issues_count":0,"forks_count":12,"subscribers_count":7,"default_branch":"master","last_synced_at":"2023-02-26T05:08:23.790Z","etag":null,"topics":["compression","encryption","sqlite3","vfs"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/techrah.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-07-07T01:03:23.000Z","updated_at":"2023-02-08T02:41:54.000Z","dependencies_parsed_at":"2022-08-26T14:34:17.001Z","dependency_job_id":null,"html_url":"https://github.com/techrah/sqlite3-compression-encryption-vfs","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techrah%2Fsqlite3-compression-encryption-vfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techrah%2Fsqlite3-compression-encryption-vfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techrah%2Fsqlite3-compression-encryption-vfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techrah%2Fsqlite3-compression-encryption-vfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/techrah","download_url":"https://codeload.github.com/techrah/sqlite3-compression-encryption-vfs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219871448,"owners_count":16554409,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","encryption","sqlite3","vfs"],"created_at":"2024-09-25T03:49:36.489Z","updated_at":"2025-09-25T22:31:27.084Z","avatar_url":"https://github.com/techrah.png","language":"C","readme":"# CEVFS: Compression \u0026 Encryption VFS for SQLite 3\nCEVFS is a SQLite 3 Virtual File System for compressing and encrypting data at the pager level. Once set up, you use SQLite as you normally would and the compression and encryption is transparently handled during database read/write operations via the SQLite pager.\n\n## Introduction\nCEVFS is an open source SQLite extension that combines some of the functionality of [CEROD](http://www.sqlite.org/cerod/doc/trunk/www/index.wiki) and [ZIPVFS](http://www.sqlite.org/zipvfs/doc/trunk/www/index.wiki) into one package. Even though no testing has been done yet beyond macOS and iOS, the goal is to make it as portable as SQLite itself -- with the help of the community.\n\nCEVFS gives you convenient hooks into SQLite that allow you to easily implement your own compression and encryption functions. As different operating systems come preloaded with different compression and encryption libraries, no default compression or encryption functions are supplied with CEVFS. The **cevfs_example** project uses Zlib and CCCryptor (3cc), both of which are included in macOS and iOS.\n\n## How It Works\nCEVFS is a [SQLite Virtual File System](http://www.sqlite.org/vfs.html) which uses its own pager and is inserted between the pager used by the b-tree (herein referred to as the upper pager) and the OS interface. This allows it to intercept the read/write operations to the database and seamlessly compress/decompress and encrypt/decrypt the data. See \"[How ZIPVFS Works](http://www.sqlite.org/zipvfs/doc/trunk/www/howitworks.wiki)\" for more details. Unlike ZIPVFS, the page size of the pager used by CEVFS (herein referred to as the lower pager) is determined when the CEVFS database is created and is a persistent property of the database. WAL mode is not yet supported.\n\n## How To Build\n\n#### Get the SQLite Source Code\n1. Go to the [Downloads page](http://www.sqlite.org/download.html).\n1. Select one of the geographically located sites ([Dallas TX](http://www.sqlite.org/cgi/src), [Newark NJ](http://www2.sqlite.org/cgi/src), [Fremont CA](http://www3.sqlite.org/cgi/src)) from the Source Code Repositories section at the bottom of the page.\n1. Select the **Tags** tab at the top of the page.\n1. Select the SQLite version you're interested in.\n1. Select the Fossil SHA1 (10-char hex string) commit link.\n1. Finally, download the tarball or ZIP archive.\n\n_For easy reference, it will be assumed that the SQLite source code is in the `sqlite/` directory._\n\n#### Create the Amalgamation file.\n1. Decompress the archive and `cd` to the root of the unarchived directory.\n1. `./configure`\n1. `make sqlite3.c`\n\n#### Build a Static Library\n1. Create a temporary `build` directory and `cd` to it.\n1. Copy `sqlite3.c`, `cevfs.c` and `cevfs.h` to the `build` directory.\n1. Combine the files: `cat sqlite3.c cevfs.c \u003e cevfs-all.c`\n1. Compile: `clang -c cevfs-all.c -o sqlite3.o -Os`\n1. Create static lib: `libtool -static sqlite3.o -o sqlite3.a`\n\n### Creating a Command-Line Build Tool\nIf you are using macOS, you can use the `cevfs_build` example which implements compression using Zlib and encryption using 3cc. Otherwise, modify the _xFunctions_ first to accommodate your operating system.\n\nCopy the following files to your temporary build directory:\n- sqlite/sqlite3.c\n- cevfs/cevfs.c\n- cevfs\\_build/cevfs\\_build.c\n- cevfs_build/xMethods.c\n\nBuild:\n```\nbuild\u003e $ cat sqlite3.c cevfs.c \u003e cevfs-all.c\nbuild\u003e $ clang cevfs-all.c cevfs_build.c -O2 -o cevfs_build -lz\n```\n\nThen to create a CEVFS database:\n```\n./cevfs_build UNCOMPRESSED COMPRESSED VFS_NAME KEY\n```\n\nparameters:\n- **UNCOMPRESSED**: path to uncompressed database\n- **COMPRESSED**: path to new compressed database\n- **VFS_NAME**: name to embed in header (10 chars. max.)\n- **KEY**: encryption key\n\nE.g.:\n\n```\n./cevfs_build myDatabase.db myNewDatabase.db default \"x'2F3A995FCE317EA22F3A995FCE317EA22F3A995FCE317EA22F3A995FCE317EA2'\"\n```\n\n(hex key is 32 pairs of 2-digit hex values)\n\nYou can also try different block sizes and compare the sizes of the new databases to see which one uses less space. To specify the block size, specify the destination path using a URI and append `?block_size=\u003cblock size\u003e`:\n\n```\n./cevfs_build myDatabase.db \"file:///absolute/path/to/myNewDatabase.db?block_size=4096\" default \"x'2F3A995FCE317EA2...'\"\n```\n\n### Creating a Custom Version of SQLite\nIt is helpful to have a custom command-line version of `sqlite3` on your development workstation for opening/testing your newly created databases.\n\nCopy the following files to your temporary `build` directory.\n- sqlite3.c (from SQLite source)\n- shell.c (from SQLite source)\n- cevfs/cevfs.c\n- cevfs_build/cevfs_mod.c\n- cevfs_build/xMethods.c\n\nAgain, modify your _xFunctions_ to accommodate your operating system. You may also need to install the Readline lib.\n\nBuild:\n```\nbuild\u003e $ cat sqlite3.c cevfs.c cevfs_mod.c \u003e cevfs-all.c\nbuild\u003e $ clang cevfs-all.c shell.c -DSQLITE_ENABLE_CEROD=1 -DHAVE_READLINE=1 -O2 -o sqlite3 -lz -lreadline\n```\n\n_If you get errors related to implicit declaration of functions under C99, you can add `-Wno-implicit-function-declaration` to disable them._\n\nThen, to open a CEVFS database:\n\n```\n$ ./sqlite3\nsqlite\u003e PRAGMA activate_extensions(\"cerod-x'\u003cyour hex key goes here\u003e'\");\nsqlite\u003e .open path/to/your/cevfs/db\n```\n\nBy specifying `SQLITE_ENABLE_CEROD` we can make use of an API hook that's built into SQLite for the CEROD extension that will allow you to conveniently activate CEVFS. It has the following signature:\n\n```\nSQLITE_API void SQLITE_STDCALL sqlite3_activate_cerod(const char *zPassPhrase);\n```\n\nIf you are using CEVFS, chances are that you are _not_ currently making use of this API hook. You can use the `const char *` param to pass something other than the intended activation key, such as the encryption key. This `sqlite3_activate_cerod` function has been implemented in `cevfs_build/cevfs_mod.c` as an example. Alternatively, you can roll out your own [Run-Time Loadable Extension](http://www.sqlite.org/loadext.html) for use with a standard SQLite 3 build.\n\n## Limitations\n\n- WAL mode is not (yet) supported.\n- Free nodes are not managed which could result in wasted space. Not an issue if the database you create with `cevfs_build()` is intended to be used as a read-only database.\n- VACUUM not yet implemented to recover lost space.\n\n## SQLite3 Compatible Versions\n\n|Version|Combatibility|\n|-|-|\n|3.10.2|Most development was originally with this version.|\n|3.11.x|Testing OK. No changes were required.|\n|3.12.0 - 3.15.2|Default pager page size changed. CEVFS updated for these versions.|\n|3.16.0 - 3.34.0|`sqlite3PagerClose` API signature changed: CEVFS updated for these versions.|\n\n## Contributing to the project\nIf you would like to contribute back to the project, please fork the repo and submit pull requests. For more information, please read this [wiki page](https://github.com/ryanhomer/sqlite3-compression-encryption-vfs/wiki/Developing-in-Xcode)\n\nHere are some things that need to be implemented:\n- Proper mutex \u0026 multi-threading support\n- TCL unit tests\n- WAL support\n- Full text indexing support\n- Keep track of free space lost when data is moved to a new page and reuse free space during subsequent write operations\n- Implement database compacting (using 'vacuum' keyword if possible)\n","funding_links":[],"categories":["extentions"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechrah%2Fsqlite3-compression-encryption-vfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftechrah%2Fsqlite3-compression-encryption-vfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechrah%2Fsqlite3-compression-encryption-vfs/lists"}