{"id":34187558,"url":"https://github.com/pkramme/btsoot","last_synced_at":"2026-03-10T08:04:09.024Z","repository":{"id":57589019,"uuid":"69793097","full_name":"pkramme/btsoot","owner":"pkramme","description":"Incremental offsite backups","archived":false,"fork":false,"pushed_at":"2017-08-27T12:26:17.000Z","size":5405,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-18T14:42:44.649Z","etag":null,"topics":["backup","backup-solution","nas","offsite","offsite-backup","scans","server","sysadmin"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pkramme.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-02T11:13:48.000Z","updated_at":"2017-07-24T20:08:13.000Z","dependencies_parsed_at":"2022-09-26T19:33:39.425Z","dependency_job_id":null,"html_url":"https://github.com/pkramme/btsoot","commit_stats":null,"previous_names":["paulkramme/btsoot"],"tags_count":24,"template":false,"template_full_name":null,"purl":"pkg:github/pkramme/btsoot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkramme%2Fbtsoot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkramme%2Fbtsoot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkramme%2Fbtsoot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkramme%2Fbtsoot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pkramme","download_url":"https://codeload.github.com/pkramme/btsoot/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkramme%2Fbtsoot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30327574,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T05:25:20.737Z","status":"ssl_error","status_checked_at":"2026-03-10T05:25:17.430Z","response_time":106,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","backup-solution","nas","offsite","offsite-backup","scans","server","sysadmin"],"created_at":"2025-12-15T15:16:02.863Z","updated_at":"2026-03-10T08:04:08.997Z","avatar_url":"https://github.com/pkramme.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BTSOOT\n**contains old information, wip**\n## What is BTSOOT\n`tl;dr: A data selective backup/cloning tool that only manipulates changed data`  \nFirst of: BTSOOT should only be used under very special circumstaces. If these requirements are\nmet, BTSOOT could be **your** backup/cloning solution. A list of this circumstaces:\n- You want to create offsite backups\n- You have much faster internal drive speed (in my case 400 - 600 MB/s) than your external connection (LAN/WAN/remote HDD etc...) and/or\n- You have much unchanged data\n\n### Practical Example\nLets say you have\n- 2 TBs Movies\n- 200 GBs Music\n- 100 GBs Files, like PDFs, DOCXs, Mails, whatever...\n- 50 GB Photos\n\nI bet you do not change 90% of these files. You put movies one time on your server, \nand then don't touch them anymore. Same goes for Music and Photos. Your files however,\nare daily used. You move a few gigabyte here, add a few megabytes there, delete something.\nHowever, if you want an offsite backup, most backup software doesn't care. It will still copy any file,\nevery Movie, every single MP3, everything, even if only your \"Files\" folder changed, aside from \nmaybe 3 added Music files.  \nThis software fixes that, as it will only copy the real changed data.\nBTSOOT now scans all files in that directory and gives them a checksum. I now knows, when it scans them again,\nif they have been changes, or if files have been added or removed. After that it copies the changed and new files to\nthe remote directory, and deletes the non existing ones. This happens every scan + backup phase. If you want to do\non a regular basis, and don't want to do this manualy, it might be a good idea to add the commands \n`btsoot scan \u003cblockname\u003e` and `btsoot backup \u003cblockname\u003e` to your cron list.\nIf, for some reason, no file is found at a new scan, the backup gets aborted to prevent corrupting your functioning\nbackup. Same goes for Specific deletion levels, which can be configured.\n\n---\n\n## Current project status\nBTSOOT is still under heavy development, thus it is only in 0.x release. The project has left its prototyping implementation\nin Python, and will be rewritten in C with the currently anticipated features, that are (partly) available in 0.6-stable.\nThe 0.6-stable release will be maintained with bugfixes, but no new features will be added. 0.7-stable release will be the\nfirst C implementation of BTSOOT and will hopefully be faster and better executed than the prototype.  \nAltough it's a prototype, 0.6-stable is totally usable for the public, and I am encouraging you to use it, as it is a good\nproduct.\n\n---\n\n## Dependencies\n- 64bit Linux OS (no specific distribution)  \n\nIf you want to build from source:  \n- clang\n- make\n\n---\n\n## Usage\n### Installation\n1. Clone the repository to the folder where you want it\n2. Check out your desired release branch e.g. `0.4-stable`, so `git checkout 0.4-stable`\n2. Execute `make` and `sudo make install`\n4. Done\n\nBTSOOT is now in `/usr/local/bin/` and can be used. It needs root permissions to run. The config or all scans are in `/etc/btsoot/`. If you want to \nuninstall it, run `sudo make uninstall`. This will delete all scans and your config, too.\n\nNOTE:  \nI am currently engaged in a project in which a [cURL package manager](https://github.com/eddyx9/install.paukra.com/) is developed. I'm hopeing it can be used to \ndistribute this project in the future. There are `.deb` packages available, but it is recommended to build the project yourself, as it will give you the best \npossible performance.\n\n### Create a block\n`btsoot add \u003cblock-name\u003e \u003csource-path\u003e \u003cpath-to-remote-dir\u003e`  \nThis is written to a file named `btsoot.conf` which is created inside the directory where BTSOOT lies.\n\n### Backup a block\n`btsoot backup \u003cblock-name\u003e`  \nThe program will search for the latest changes, and compare them with the previous, which it then copyies to their \npaths on the remote location.  \nPrior to 0.5 there were two commands `scan` and `backup`. This wasn't practical, \nas both were used together while one on its own was unecessary, so they got removed.\n**This also means that you MUST NOT change the remote files per hand. BTSOOT will not know about any changed file that \nit didn't changed itself.** BTSOOT does not check the remote files for integrity. It is your responsibility that they \nare not broken\n\n### Restore a block\n`btsoot restore \u003cblock-name\u003e`  \nThe program will delete the entire source folder, and then reload the files from the latest scan. If no files are\nfound, everything is lost. Fortunately there is no scanning involved, so its just copying, and this is fast.  \n`--override` will override the safety time, so the restoring will begin immediately. Use with caution.\n\n---\n\n## Backstory\n### What i have\nI do not have\n- Much time\n- Much money\n\nWhat i do have\n- nearly 3 TB's of data\n- 2 Routers with NAS features\n- 2 Raspberry Pi's\n- Fast RAIDz1 on my primary NAS (450 - 600 MB/s)\n- Gigabits LAN\n- Very important files, like backups and other mission critical files, that are irreplaceable\n- Sysadmin and programming skills\n\nNow i have a problem. I don't have much money, so buying new hardware for an offsite \n(which is defined here as not in the same room or the same server) NAS is not an option.\nSo i have to work with what i have. Let's see:\n#### Raspberry Pi\nRaspberry Pi's are not suited for NAS's. You can read this anywhere on the internet.\nThey are slow, because they lack of a dedicated Ethernet Interface, everything is runned over\none single USB 2.0 Host. This is slow. On the plus side, they are small, silent, have low energy\nusage, and are cheap - 35 Euros, here in Germany.\n#### FritzBox Router\nI have two of them, one is my primary gateway, one my Wifi AP, both have rudimentary NAS features like SMB and FTP.\nAlso, one of them has USB3.0 which theoreticly should be fast, when it is combined with my 3TB HDD from WD, which also has \na USB 3.0 interface. It even has dedicated Gigabits LAN (not so surprising, as it is a router) but for a NAS, it's great.  \n### Testing\nWell. The Raspberry Pi was slow. The very definition of slow. 7 MB/s at best with ext4 for the HDD, NFSv4 and NFSv3\nand all speed fixes i know. My other Raspberry Pi, a second generation device, brought it up to 15MB/s max. So i turned to the FritzBox.\nAnd was disappointed again: 10MB/s without even the possibility of speed improvements, because hacking a Router OS is beyond what one should do\nwith absolute mission critical hardware. If the FritzBox get broken, i have to spend money to fix it, which i do not have, and my family members and me\nare out of internet until a replacement device arrives. So, no. I have to work with this. \n### Planning and Development\nAt this point i decided that there was not much i could do about the performance.\nI decided to use the FritzBox, because i only could use the slower of them, the other one were redisigned to a PiHole DNS server, and the speed \nwhere more unstable that the router's. The initial datatransfer wouldn't be a problem, as i could do it on every device and then just \nplug it into the FritzBox after that. So i had to find a way to speed up the transfer, without increasing connection speed, or tuning protocol.\nI tought about the thing every sysadmin would have though about: rsync. I would just mount the SMB on my NAS, use rsync to transfer any changed parts of the file\nand would therefore only have to deal with a minimum filesize, as most of the files are media files, which doesn't change that often. There were a problem though:\nWhat happens when a file gets renamed, or a folder moved, or deleted? Rsync doesn't cover that. I a file is deleted, rsync doesn't remove it, it will just remain.\nSame with renamed directories.  \nIn this moment i changed from sysadmin to developer:  \n\n**I had to create a program which identifies a file, and mirrors a tree over the network, without having to send already existing files.**  \n\nI thought about a project which i already planned before: A program where you can create certain blocks, and monitor them.\nEarly drafts of this are found in [this](https://git.paukra.com/open-source/backup) repository, where i was eager to write it in system dependent languages, like C\nfor Linux and C# for Windows. I wasn't experienced enough to write it, and altough i could have learned it, it would need time, which i don't had.\nSo i created a new project, this project, called BTSOOT, and began to write it in Python.\nI wasn't good in Python, but it is way faster to learn than C with all Linux system calls. And here i am. The program as it is runs on a Linux host under Python3.6 (formatted srings\nwere to nice to ignore them) and copies changed files to a mounted network folder on the host.\n\n\n---\n\n## Project Information\nI am currently enforcing GitLab Flow, with [semantic versioning](http://semver.com) in release channels.\n\n## Dependencies\nFor the sake of portability, i try to statically link anything possible. [A list of additional dependencies and their licenses](https://git.paukra.com/open-source/btsoot/wikis/additional-dependencies).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpkramme%2Fbtsoot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpkramme%2Fbtsoot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpkramme%2Fbtsoot/lists"}