{"id":16689982,"url":"https://github.com/eddieantonio/blobfs-poc","last_synced_at":"2025-05-15T11:31:49.857Z","repository":{"id":39450614,"uuid":"95252882","full_name":"eddieantonio/blobfs-poc","owner":"eddieantonio","description":"BlobFS—access your database from the comfort of your file manager!","archived":false,"fork":false,"pushed_at":"2017-06-24T20:25:30.000Z","size":18,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-03T08:23:52.499Z","etag":null,"topics":["blob","filesystem","fuse","fuse-filesystem","sql","sqlite3"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eddieantonio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-06-23T20:09:48.000Z","updated_at":"2023-09-08T17:26:43.000Z","dependencies_parsed_at":"2022-09-15T10:50:21.549Z","dependency_job_id":null,"html_url":"https://github.com/eddieantonio/blobfs-poc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddieantonio%2Fblobfs-poc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddieantonio%2Fblobfs-poc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddieantonio%2Fblobfs-poc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddieantonio%2Fblobfs-poc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eddieantonio","download_url":"https://codeload.github.com/eddieantonio/blobfs-poc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254330720,"owners_count":22053034,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blob","filesystem","fuse","fuse-filesystem","sql","sqlite3"],"created_at":"2024-10-12T15:49:56.082Z","updated_at":"2025-05-15T11:31:49.570Z","avatar_url":"https://github.com/eddieantonio.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"BlobFS proof of concept\n=======================\n\nBlobFS is a [FUSE] filesystem that lets you access fields in an SQLite3\ndatabase with the convenience of a regular file system.\n\nInspired by the news that accessing blobs from SQLite3 can be [around\n35% faster than from the filesystem][fasterthanfs], BlobFS is meant to\nbe a way of accessing that same data—locked away as a `BLOB` in the\ndatabase—with your favourite software that reads conventional files.\n\nBlobFS is proof of concept software. As such, it is lacking in security,\nreliability, speed, and testing. For more information, see\n[Limitations](#limitations).\n\n[fasterthanfs]: https://www.sqlite.org/fasterthanfs.html\n[FUSE]: https://github.com/libfuse/libfuse\n\n\nInstall\n-------\n\nYou need Python 3.6 and [fusepy]. You can install the latter using pip:\n\n    python3.6 -m pip install fusepy\n\n[fusepy]: https://github.com/terencehonles/fusepy\n\n\nUsage\n-----\n\n    blobfs \u003cdatabase\u003e \u003cmountpoint\u003e\n\nMount an SQLite3 `\u003cdatabase\u003e` on the provided empty directory\n`\u003cmountpoint\u003e`.\n\n### Example\n\nSay you want to mount the database called `sources.sqlite3`. This\ndatabase contains Python source code downloaded from GitHub and has the\nfollowing schema:\n\n```sql\nCREATE TABLE repository (\n    owner, name,\n    PRIMARY KEY (owner, name)\n);\n\nCREATE TABLE source_file (\n    hash PRIMARY KEY,\n    source BLOB NOT NULL\n);\n\nCREATE TABLE repository_source (\n    owner, name, hash, path,\n    PRIMARY KEY (owner, name, hash, path)\n);\n```\n\nCreate a new directory for the mount called `mnt/`.\n\n    $ mkdir mnt\n    $ ls -F\n    blobfs.py\n    mnt/\n    sources.sqlite3\n\nRun the FUSE filesystem in the foreground.\n\n    $ ./blobfs.py sources.sqlite3 mnt/\n    \u003cA whole wacktonne of log messages from FUSE\u003e\n\nNow switch to a different terminal (or use your file manager!) and\nnavigate into the newly mounted filesystem.\n\nThe root directory contains subdirectories for each table:\n\n    $ ls -F mnt/\n    repository/\n    source_file/\n    repository_source/\n\nWithin a table directory are subdirectories for every row in that table.\n\n    $ cd mnt/source_file\n    $ ls -F\n    98c2d41c472c435aa3d06180a626f1690b681fb07499e3f633a85007f25bed18/\n\nWithin the directory for a row in table is a regular file for all\nfields. You may then access any field as a regular file. Blobs are read\nverbatim, and while other data types are converted to strings, and then\nencoded in UTF-8. Here, the field `source` is a blob containing Python\nsource code.\n\n    $ cd 98c2d41c472c435aa3d06180a626f1690b681fb07499e3f633a85007f25bed18/\n    $ ls\n    hash\n    source\n    $ wc source\n         331    1066    9567 source\n    $ file source\n    source: a python3 script text executable\n    $ python3 source --help\n    usage: source [-h] database mountpoint\n\n    positional arguments:\n      database\n      mountpoint\n\n    optional arguments:\n      -h, --help  show this help message and exit\n\nIn summary, mounting this database has created the following file\nstructure:\n\n```\n.\n├── repository\n│   └── 1\n│       ├── name\n│       └── owner\n├── repository_source\n│   └── 1\n│       ├── hash\n│       ├── name\n│       ├── owner\n│       └── path\n└── source_file\n    └── 98c2d41c472c435aa3d06180a626f1690b681fb07499e3f633a85007f25bed18\n        ├── hash\n        └── source\n```\n\n\nLimitations\n-----------\n\nThis proof of concept is vulnerable to SQL injection, as it does not\nvalidate that the table names and primary keys have sanitary names.\nAdditionally, it does not validate that names stored in the database\nmake for reasonable filenames—filenames beginning with `-`, `.`, or\ncontaining `/`, or `\\0` anywhere are examples of unreasonable filenames.\n\nAdditionally, every system call often implies several database queries.\nNo database queries are ever cached, and related queries are never run\nwithin a transaction.  Additionally, several high-level operations (such\nas `ls -l`) require numerous system calls, degrading performance\nsignificantly.\n\nSince `readdir(3)` is implemented by copying the name of each primary\nkey into memory and returning then all as one big list, it is a quite\ninefficient operation, and is incapable of being interrupted. As such,\nrunning `ls` in large table (over 500 thousand rows) will usually fail,\ndue to timeouts.\n\nThis script runs the FUSE filesystem in foreground mode and in one\nthread. A practical implementation would be capable of running in the\nbackground, and would be thread-safe.\n\n\nCopying\n-------\n\nLicensed under the terms of the GPLv3 license. See [LICENSE].\n\n[LICENSE]: ./LICENSE\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feddieantonio%2Fblobfs-poc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feddieantonio%2Fblobfs-poc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feddieantonio%2Fblobfs-poc/lists"}