{"id":15373044,"url":"https://github.com/vi/forsnapshotfs","last_synced_at":"2025-06-27T11:33:11.704Z","repository":{"id":9144648,"uuid":"10937075","full_name":"vi/forsnapshotfs","owner":"vi","description":"Storage for incremental backup/snapshots (as single big files)","archived":false,"fork":false,"pushed_at":"2014-04-08T11:10:33.000Z","size":524,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-28T08:11:58.598Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"laizhouxia/fyp-chrome-extension","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-06-25T12:01:56.000Z","updated_at":"2024-03-05T07:45:30.000Z","dependencies_parsed_at":"2022-07-12T15:03:22.690Z","dependency_job_id":null,"html_url":"https://github.com/vi/forsnapshotfs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vi%2Fforsnapshotfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vi%2Fforsnapshotfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vi%2Fforsnapshotfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vi%2Fforsnapshotfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vi","download_url":"https://codeload.github.com/vi/forsnapshotfs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248951897,"owners_count":21188419,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T13:54:09.430Z","updated_at":"2025-04-14T19:52:56.384Z","avatar_url":"https://github.com/vi.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"forsnapshotfs allows storing multiple big files which are expect to have many common blocks (like LVM snapshots or virtual machine images) efficiently.\n\nUsage\n---\n\nStoring the first file:\n\n    fsfs-write /path/to/storage/directory name1 \u003c /dev/mapper/vg-vn1\n    \nStoring the modified version of the file (based on \"name1\"):\n\n    fsfs-write /path/to/storage/directory name2 name1 \u003c /dev/mapper/vg-vn1\n\nExtracting:\n\n    fsfs-read /path/to/storage/directory name2 \u003e output\n    \nAccessing the stored file without extrantion:\n\n    fsfs-mount path/to/storage/directory /path/to/mountpoint\n    mount -o ro,loop /path/to/mountpoint/name1 /mnt/name1\n    \n* There is built-in simple in-memory copy-on-write for replaying \njournaled filesystems. Changes are preserved until the file is closed.\n\nStorage format\n---\n\nStorage directory have files with extensions \"\\*.dat\", \"\\*.dsc\", \"\\*.idx\" and \"\\*.hsh\".\n\ndsc file have two decimal numbers (block size and number of blocks in a block group), followed by newline and maybe by dependency name.\n\n    4096 1020\n    root_20130429\n\nidx file contains 2048-byte entries, \neach descibing 1020 blocks. It is 8 bytes of big endian base \noffset in the dat file, for this block group, followed by 1020 2-byte \nsigned big endian numbers (0 - unallocated block (near the end of file), \npositive - length of this block compressed, negative - this block is \nnot stored here and should be retrieved from a dependency, 0x8000 - \nzero block, 0x8001 - uncompressed block). Blocks are usually 4096 bytes each.\n\ndat file contains lzo1x-compressed blocks contatenated, referenced by idx file.\n\nhsh file contains 8-bit pearsong-style hashes of blocks. i'th byte of \nthis file correspond to i'th block.\n\nExample:\n\n    test.dsc:\n        4096 1020\n\n    test.idx:\n\n        0x0000: 0000000000000000 0030 0090 0100 1034 06F3\n             (then 1015 times of 0000, for example)\n        0x0800: 00000000000018E7 0055 0000 0000 0000 .....\n        0x1000: end of file\n        \n    test.dat:\n    \n        0x0000: (0x30 bytes of compressed data for block 0)\n        0x0030: (0x90 bytes of compressed data for block 1)\n        0x00C0: (0x100 bytes of compressed data for block 2)\n        0x01C0: (0x1034 bytes of compressed data for block 3)\n        0x11F4: (0x6F3 bytes of compressed dta for block 4)\n        0x18E7: (0x55 bytes of compressed data for block 1020)\n        0x193C: end of file\n        \n    test.hsh:\n    \n        0x0000: 53 99 32 11 67 00 00 00 00 00 00 .. \n        0x03FC: 34 (end of file)\n        \n    \n        \n    \"qqq\" data corresponds to the \"test\", but after \n        pwrite(fd, \"qqq\", 3, 0x302C)\n        \n        \n    qqq.dsc:\n        4096 1020\n        test\n        \n    qqq.idx:\n    \n        0x0000: 0000000000000000 FFFF FFFF FFFF 1036 FFFF 8000 8000 ...\n        0x0800: 0000000000001036 FFFF 0000 0000 ...\n        0x1000: end of file\n        \n    qqq.dat:\n    \n        0x0000: (0x1036 bytes of compressed data for block 3)\n        0x1036: end of file\n       \n    qqq.hsh:\n        0x0000: 53 99 32 9C 67 00 00 00 00 00 00 .. \n        0x03FC: 34 (end of file)\n\n    \n        \n    \"qqq2\" is based on \"qqq\" and have the same data as \"qqq\":\n    \n    qqq2.dsc:\n        4096 1020\n        qqq\n        \n    qqq2.idx:\n    \n        0x0000: 0000000000000000 FFFE FFFE FFFE FFFF FFFE 8000 8000 ...\n        0x0800: 0000000000000000 FFFE 0000 0000 ...\n        0x1000: end of file\n        \n        FFFE means \"refer to the second dependency\"\n        FFFF means \"refer to the first(directly specified in dsc) dependency\n        \n    qqq2.dat:\n    \n        0x0000: end of file\n       \n    qqq2.hsh:\n        0x0000: 53 99 32 9C 67 00 00 00 00 00 00 .. \n        0x03FC: 34 (end of file)\n    \nCurrently maximum the number of dependencies is limited to 64.\n\nComparison with [bup](https://github.com/bup/bup)\n---\nIf you are interested in forsnapshotfs, you may also like [bup](https://github.com/bup/bup).\n\nDifferences:\n\n* fsfs is simpler. it is implemented in C. The algorithm is simpler than bup's one;\n* `bup` compress better;\n* fsfs require you to specify previous backup's name when creating new one;\n* `bup` also can manage many little files, file modes, etc;\n* `bup` have remote access features;\n* `fsfs-mount` have little additional cosy feature: simplistic in-memory copy-on-write to allow journal replay when mounting a filesystem. *Update*: Implemented this feature as separate filesystem: https://github.com/vi/simplecowfs. You can use it as overlay for mounting filesystem images from `bup fuse`.\n\nSimilaritues:\n\n* Compression support;\n* Efficient storage of incremental snapshots (fsfs's deduplication is simpler and won't work when offsets do change);\n* Random access to stored files, FUSE interface;\n* Both can accept snapshot input from stdin;\n* Newer backups depend on earlier ones.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvi%2Fforsnapshotfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvi%2Fforsnapshotfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvi%2Fforsnapshotfs/lists"}