{"id":20299356,"url":"https://github.com/kd2org/apache-sqliteblob","last_synced_at":"2026-05-12T19:08:32.266Z","repository":{"id":146808023,"uuid":"210433423","full_name":"kd2org/apache-sqliteblob","owner":"kd2org","description":"Apache2 SQLite blob module","archived":false,"fork":false,"pushed_at":"2019-09-23T19:16:35.000Z","size":8,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-14T10:29:58.185Z","etag":null,"topics":["apache","file-storage","haystack","sqlite"],"latest_commit_sha":null,"homepage":"https://fossil.kd2.org/apache-sqliteblob/","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kd2org.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-23T19:15:22.000Z","updated_at":"2024-10-16T09:58:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"5724c713-7358-4f55-8bda-b18985d46ffd","html_url":"https://github.com/kd2org/apache-sqliteblob","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kd2org%2Fapache-sqliteblob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kd2org%2Fapache-sqliteblob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kd2org%2Fapache-sqliteblob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kd2org%2Fapache-sqliteblob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kd2org","download_url":"https://codeload.github.com/kd2org/apache-sqliteblob/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241794136,"owners_count":20021193,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache","file-storage","haystack","sqlite"],"created_at":"2024-11-14T16:14:25.524Z","updated_at":"2026-05-12T19:08:27.217Z","avatar_url":"https://github.com/kd2org.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache SQLite Blob module\n\nThis module provides a handler to send file contents from a SQLite file instead\nof the filesystem.\n\nThe idea is to use a SQLite database file as a file storage. This is specially\nuseful if you have a lot of small files, eg. images.\n\nThe idea is similar to what Facebook does with Haystack :\n\n* [Paper (PDF)](https://www.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf)\n* [Needle in a haystack: efficient storage of billions of photos](https://code.fb.com/core-data/needle-in-a-haystack-efficient-storage-of-billions-of-photos/)\n\nIt's just adapted to a smaller scale, without the need of running specialized\ndaemons.\n\nInstead of having millions of files scattered in the filesystem, you would\nstore your files directly in a SQLite file.\n\n## Pros and cons\n\nCompared to an object storage, this approach requires much less overhead. There\nis no need to run new daemons or complex setups, just insert files to the\nSQLite database.\n\nSQLite also provides handy built-ins, for example you can just delete files\nfrom the database and then just use `VACUUM` to have SQLite clean up the\ndatabase from old file fragments.\n\nIt's also easy to store extra data, just add columns to the table!\n\nFinally, it's very hard to actually corrupt a SQLite database.\n\nThe major con will be that it's probably not as efficient as a dedicated\nservice, and you might reach limits faster if your data set grows to be\nvery large.\n\n## Performance\n\nAccording to my own benchmarks, HTTP performance is about 2 to 5% below serving\nregular files from the filesystem.\n\nI experimented successfully with databases up to 10 GB containing about a dozen\nmillion files. It worked OK, but I recommend to not exceed 100,000 files or\n1 GB per SQLite database to keep things simple and fast (just use\npartitioning, or sharding, to keep separate database files).\n\nNote that it's not a hard limit, and SQLite can manage up to 8 TB of data in a\nsingle database using its default settings, or up to 140 TB if you change its\npage size setting. This is just a suggestion to keep things simple for you, as\nit may take a while to do a VACUUM or other heavy operation on a database once\nit gets very large.\n\n## Installation\n\nOn Debian/Ubuntu just start by installing the `apache2-dev` package:\n\n```\n\tapt install apache2-dev\n```\n\nUsually, just running `make` from the src directory will be enough:\n\n```\n\tcd src/\n\tmake\n```\n\nIf you want to install the module in the Apache directory you can also run\n`sudo make install`. This will also enable the module in Apache2 config.\n\nLast thing to do is to reload the Apache config using `sudo apache2ctl graceful`.\n\n## Configuration\n\nJust enable the correct handler in your configuration for the files you want to\nhandle. Here is an example:\n\n```\n\t\u003cDirectory /var/www/images\u003e\n\t\t# Treat filenames ending with \".img\" as SQLite blob database\n\t\tAddHandler sqliteblob .img\n\t\tRewriteEngine On\n\t\tRewriteRule /img/((\\d)\\d+)\\.jpg /store_$2.img?id=$1\n\t\u003c/Directory\u003e\n```\n\n\n\n## Creation of the database\n\nTo be accepted by this module, the SQLite database file must have an\n`application_id` set to `0x01021234`. To do that just run:\n\n```\nPRAGMA application_id = 0x01021234;\n```\n\nThis is just so it's not possible to use this module to query any random\nSQLite database.\n\nThe database must have a table named `blob` containing at least those columns:\n\n* `hash` (TEXT), containg the hash identifier of the file\n* `mimetype` (TEXT), a string containing the file mimetype (eg. `image/jpeg`)\n* `updated` (INTEGER), a UNIX timestamp of the last change to the file (this is\n  used for the `Last-Modified` HTTP header)\n* `content` (BLOB), containing the file itself\n\nYou can add more columns if you wish, but if any of these columns is missing,\na 500 error will be returned.\n\nHere is a simple example of a basic blob database file:\n\n```sql\nPRAGMA application_id = 0x01021234;\nCREATE TABLE IF NOT EXISTS blobs (\n\thash TEXT NOT NULL PRIMARY KEY,\n\tmimetype TEXT,\n\tupdated INT,\n\tcontent BLOB\n);\n```\n\nThen you can just insert files to the database. It is recommanded that you use\n`sqlite3_blob_open` to write the blob to the database, as it is faster than\nusing binded parameters.\n\nAn example PHP script lies in `src/make-blob-archive.php` to create blob\narchives.\n\n`php make-blob-archive.php test.images images/` will append all files from the\n`images` directory to the `test.images` blob archive.\n\n## Thanks\n\nThis module was written thanks to the help of docs from [Apache](https://httpd.apache.org/docs/2.4/developer/modguide.html)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkd2org%2Fapache-sqliteblob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkd2org%2Fapache-sqliteblob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkd2org%2Fapache-sqliteblob/lists"}