{"id":23586550,"url":"https://github.com/doctorwkt/search_backups","last_synced_at":"2025-10-06T02:33:48.548Z","repository":{"id":96057410,"uuid":"91871909","full_name":"DoctorWkt/search_backups","owner":"DoctorWkt","description":"Small scripts to build a catalog on backups and search the catalog","archived":false,"fork":false,"pushed_at":"2017-05-20T08:24:09.000Z","size":25,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-17T04:07:01.098Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DoctorWkt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-20T07:10:02.000Z","updated_at":"2018-07-31T19:55:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"a1f91115-3814-4b60-94cd-ac3836932ed3","html_url":"https://github.com/DoctorWkt/search_backups","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DoctorWkt/search_backups","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DoctorWkt%2Fsearch_backups","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DoctorWkt%2Fsearch_backups/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DoctorWkt%2Fsearch_backups/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DoctorWkt%2Fsearch_backups/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DoctorWkt","download_url":"https://codeload.github.com/DoctorWkt/search_backups/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DoctorWkt%2Fsearch_backups/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278550159,"owners_count":26005227,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-27T04:11:34.080Z","updated_at":"2025-10-06T02:33:48.519Z","avatar_url":"https://github.com/DoctorWkt.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"# search_backups\nThis a set of small Perl scripts to build a catalog on a set of mountable\nbackups, and to search the catalog.\n\nI have a set of five or six USB hard disks, which I've used over the years\nto backup my home and work systems. I've used various tools such as _rsync_,\n_rsnapshot_ and a Btrfs tool to copy files over to the hard disks. Now I\nhave 100 million or so files out on several disks, and trying to find a file\nis becoming hard. Hence these scripts.\n\n# Creating the Initial Database\n\nThe backup catalog is stored in an Sqlite3 file called `files.db`. To create\nan initial database file:\n\n```\n$ cat files.sql | sqlite3 files.db\n```\n\n# Adding a Volume's Details to the Database\n\nBefore you add new backup entries to the database, you first must add\ndetails of the \"volume\". This helps you remember the details of each\nbackup device and where it is. To add the details of a volume, do:\n\n```\n$ ./add_volume [-db dbfile] volume_name description location\n```\n\n_volume_name_ should be a short word or phrase that describes the volume.\n_description_ can be a sentence or two that describes the volume. I use\nthe text that I write on the sticky label that I put on each USB drive.\n_location_ can be a sentence or two that describes where you can find the\nvolume.\n\nAs an example, here is one volume that I added to the database:\n\n```\n$ ./add_volume apr2014 'Warrens Back Drive Data from June 2011 to April 2014' 'Study cupboard at home'\n```\n\nThe _-db_ option allows you to choose a different Sqlite3 database file\ninstead of the default `files.db`.\n\nThere is a short script, `list_volumes`, to list what volumes are available.\nIt simply sends an SQL command to the database to list the _volume_ table:\n\n```\n$ ./list_volumes\n1|fred|fred|fred\n2|offsite|Off-site Backup|Study cupboard at home\n3|may2011|Warrens Backup Drive, Data up to May 2011|Study cupboard at home\n4|iso|ISO Images past, 2012, onwards|Study cupboard at home\n5|apr2014|Warrens Back Drive Data from June 2011 to April 2014|Study cupboard at home\n6|bob|Backup of Backups|Study cupboard at home\n```\n\nThe first entry was a test entry that I didn't use.\n\n# Building a Database Catalog\n\nAssume that you have a backup mount at `/mountpoint` and you have given this\nthe volume name _fred_. To add all the files to the catalog, you would do:\n\n```\n$ ./add_files fred /mountpoint\n```\n\nThis will print out a decimal point `.` as the script enters a new directory.\nThe script will also print out an asterisk `*` every 30 seconds, so that you\nget some other indication of progress.\n\nThe insert speed seems to be reasonably constant regardless of the size\nof the database.\n\nThere are some command-line options to the script:\n\n```\nUsage: ./add_files [-v] [-s] [-db dbfile] volume_name mountpoint [startdir]\n```\n\n* _-v_ set a verbose flag, which I used when debugging\n* _-s_ tells the script to look out for and skip directories already processed\n* _-db_ chooses a different a different Sqlite3 database file instead of the default `files.db`\n\nYou should use the _-s_ flag when you are rescanning a backup volume for new\nentries, otherwise it will add everything back into the database and you will\nget duplicate entries. Note, however, that the _-s_ flag does slow things\ndown considerably.\n\nIf you know specifically what has been added, it's easier to use the\n_startdir_ option at the end of the command-line. For example, assume\nthat your volume is mounted on `/mountpoint` and that latest backup was\nplaced at `/mountpoint/2017-April`. You would run the command:\n\n```\n$ ./add_files fred /mountpoint /mountpoint/2017-April\n```\n\nThis will automatically set the _-s_ flag, and only scan from\n`/mountpoint/2017-April` downwards.\n\nYou should expect a decent-sized USB drive to take several hours for the\nscript to build the catalog. I have a 3T USB drive which is about 60% full\nand this took 5 or 6 hours to scan.\n\n# Size and Contents of the Catalog\n\nThe catalog contains:\n * the full pathname of each file and directory\n * the size of each file and directory in bytes\n * the last modification timestamp for each file and directory\n\nFilenames are stored only once, with a numeric id assigned to each name.\nFull pathnames are stored as a set of pointers from one row in the database\nto another row.\n\nAs I am storing snapshots of my systems on the same USB drive, a lot of\nfiles and directories have the same name. My database is using just under\n30 bytes per file entry in the database, on average. My current database\nhas 103,062,302 file entries for a size of 2,919,738,368 bytes (2.9 Gibytes).\n\n# Searching the Database\n\nYou can search for a filename or directory in the catalog in one of three\nways:\n* an exact name match using the SQL _=_ operation\n* a 'like' match using the SQL _like_ operation; this is the default\n* a regexp pattern using the Sqlite _regexp_ operation\n\nThe command-line usage is:\n\n```\nUsage: ./find_files [-e] [-r] [-db dbfile] pattern\n```\n* _-e_ turns on exact matching\n* _-r_ turns on regular expression matching\n\nIf you want to use regular expressions, you may need to install a version of\nSqlite3 with a regular expression library. On Ubuntu:\n\n```\n$ sudo apt-get install sqlite3-pcre\n```\n\nand then add this line to your `$HOME/.sqliterc`:\n\n```\n.load /usr/lib/sqlite3/pcre.so\n```\n\n## Examples of Catalog Searches\n\nExact searches, obviously, will only match filenames exactly. Like searches\nuse the SQL `like` syntax, so you should use the percent sign `%` to\nmatch on any number of any characters. A regexp search uses the\nPerl-compatible regular expressions.\n\nHere are some example searches on my 2.9 Gibyte catalog. Note that the\nfirst part of the retrieved pathname is actually the volume name.\n\n```\n$ ./find_files -e pyr.txt\n      2245  Tue Mar 21 14:37:06 1989  /offsite/Neddie/2015-10-01-10:21:23/home/wkt/Misc/pyr.txt\n      2245  Tue Mar 21 14:37:06 1989  /offsite/Neddie/2016-02-05-15:58:13/home/wkt/Misc/pyr.txt\n      2245  Tue Mar 21 14:37:06 1989  /offsite/Neddie/2016-02-07-12:58:42/home/wkt/Misc/pyr.txt\n      2245  Tue Mar 21 14:37:06 1989  /offsite/Neddie/2016-08-09-21:09:29/home/wkt/Misc/pyr.txt\n...\n      2245  Sun Apr 26 09:12:55 1998  /may2011/Archives/Misc/WBAOT2_May1998/MS-DOG/30M-Disk/pyr.txt\n      2245  Tue Mar 21 14:37:06 1989  /may2011/Neddie/home/wkt/Misc/pyr.txt\n      2245  Tue Mar 21 14:37:06 1989  /apr2014/Neddie/2012-11-21/home/wkt/Misc/pyr.txt\n...\n      2245  Tue Mar 21 14:37:06 1989  /bob/2014_April/Neddie/2011-12-19/home/wkt/Misc/pyr.txt\n      2245  Tue Mar 21 14:37:06 1989  /bob/2014_April/Neddie/2012-04-14/home/wkt/Misc/pyr.txt\n      2245  Tue Mar 21 14:37:06 1989  /bob/2014_April/Neddie/2012-07-31/home/wkt/Misc/pyr.txt\n```\nThe first time I did the search it took about 50 seconds. The second time\nit took 8 seconds as the disk blocks were cached in memory.\n\n```\n$ ./find_files '%clex.%'\n...\n     10380  Sun Aug  7 14:28:30 2016  /bob/Offsite/Neddie/2017-04-09-17:36:26/usr/local/src/Github/xv6-minix2/cmd/wish/clex.c\n     17461  Wed Nov  3 06:45:18 2004  /bob/Offsite/Neddie/2017-04-09-17:36:26/usr/local/unixtree/OpenBSD-4.6/gnu/usr.bin/binutils/binutils/rclex.c.gz\n      3915  Wed Nov  3 06:22:04 2004  /bob/Offsite/Neddie/2017-04-09-17:36:26/usr/local/unixtree/OpenBSD-4.6/gnu/usr.bin/binutils/binutils/rclex.l.gz\n       537  Tue Jan 24 10:39:13 2017  /bob/Offsite/Neddie/2017-04-09-17:36:26/usr/local/10audit/V10/usr/src/cmd/odist/pax/src/lib/libx/port/fclex.c.html\n      1202  Sat Mar  6 05:14:11 2010  /bob/Offsite/Neddie/2017-04-09-17:36:26/usr/local/v10tree/OpenSolaris_b135/cmd/fm/eversholt/common/esclex.h.gz\n       268  Wed Dec 13 08:03:44 1989  /bob/Offsite/Minnie/2017-05-14-11:37:45/usr/500/Backup/Minnie/daily.0/usr/local/v10tree/V10/usr/src/cmd/odist/pax/src/lib/libx/port/fclex.c.gz\n...\n       537  Tue Jan 24 10:39:13 2017  /bob/Offsite/Minnie/2017-05-14-11:37:45/usr/500/Backup/Minnie/daily.0/var/www/v10lobby/V10/usr/src/cmd/odist/pax/src/lib/libx/port/fclex.c.html\n```\n\nBoth the first and second _like_ searches took about 45 seconds.\n\n```\n$ ./find_files -r '[Cc]lex\\.[ch]'\n     19897  Mon Nov 20 12:28:48 1989  /bob/2014_April/Neddie/2012-11-21/home/wkt/Old/OldCDs/WarrensBigArchiveOfThings/Archive/Source/Local/Clam/1.3c/clex.c\n     21067  Tue Nov 23 10:51:11 1993  /bob/2014_April/Neddie/2012-11-21/home/wkt/Old/OldCDs/WarrensBigArchiveOfThings/Archive/Source/Local/Clam/1.4/clex.c\n     67208  Wed Nov  3 06:45:18 2004  /bob/2014_April/Neddie/2012-11-21/usr/local/src/Src/OpenBSD-4.6/gnu/usr.bin/binutils/binutils/rclex.c\n     21987  Sat Mar  6 05:14:11 2010  /bob/2014_April/Neddie/2012-11-21/usr/local/src/Src/OpenSolaris_b135/cmd/fm/eversholt/common/esclex.c\n      2422  Sat Mar  6 05:14:11 2010  /bob/2014_April/Neddie/2012-11-21/usr/local/src/Src/OpenSolaris_b135/cmd/fm/eversholt/common/esclex.h\n...\n       544  Mon Jan 23 15:34:49 1989  /bob/Offsite/Neddie/cur/usr/local/Unix/UnixArchive/Applications/News/C-News/Feb_1993_Release/libcnews/fopenclex.c\n     11264  Sun Aug  7 14:28:23 2016  /bob/Offsite/Neddie/cur/usr/local/src/Github/Wish/clex.c\n     10380  Mon Aug 15 16:44:35 2016  /bob/Offsite/Neddie/cur/usr/local/src/Github/xv6-freebsd/cmd/wish/clex.c\n...\n     21987  Sat Mar  6 05:14:11 2010  /apr2014/Henry/2011-06-29/usr/local/unixtree/OpenSolaris_b135/cmd/fm/eversholt/common/esclex.c\n      2422  Sat Mar  6 05:14:11 2010  /apr2014/Henry/2011-06-29/usr/local/unixtree/OpenSolaris_b135/cmd/fm/eversholt/common/esclex.h\n     67208  Wed Nov  3 06:45:18 2004  /apr2014/Henry/2011-12-19/usr/local/unixtree/OpenBSD-4.6/gnu/usr.bin/binutils/binutils/rclex.c\n```\n\nBoth searches took 1 minute 47 seconds.\n\nOne important thing to note about searches is that the search pattern only applies to each component of the pathname, not the full pathname. So if you searched\nfor 'a%b', it _won't_ find a pathname with _...a/b..._.\n\nSimilarly, a match on a directory name _won't_ list the contents below the\ndirectory, only the directory itself. If you search for the exact pattern\n_.git_, then you will get results like this:\n\n```\n$ ./find_files -e .git\n       166  Mon Apr 11 21:26:57 2016  /bob/Offsite/Minnie/2017-05-14-11:37:45/usr/500/Backup/Minnie/daily.0/usr/local/src/sccpdp7/.git\n       138  Sat Mar 11 10:05:13 2017  /bob/Offsite/Minnie/2017-05-14-11:37:45/usr/500/Backup/Minnie/daily.0/usr/local/src/simh/.git\n       138  Wed Apr 27 09:58:40 2016  /bob/Offsite/Minnie/2017-05-14-11:37:45/usr/500/Backup/Minnie/daily.0/usr/local/src/simple-rcs2git/.git\n       138  Tue Apr 12 07:37:11 2016  /bob/Offsite/Minnie/2017-05-14-11:37:45/usr/500/Backup/Minnie/daily.0/usr/local/src/swieros/.git\n       138  Mon Feb 22 17:44:52 2016  /bob/Offsite/Minnie/2017-05-14-11:37:45/usr/500/Backup/Minnie/daily.0/usr/local/src/unix-jun72/.git\n```\n\nbut not anything in the `.git` directories.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoctorwkt%2Fsearch_backups","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoctorwkt%2Fsearch_backups","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoctorwkt%2Fsearch_backups/lists"}