{"id":20334845,"url":"https://github.com/integrii/homebackup","last_synced_at":"2025-04-11T21:51:32.812Z","repository":{"id":6621466,"uuid":"7865043","full_name":"integrii/homebackup","owner":"integrii","description":"Family picture management and backup.  De-duplicates, sorts and backs up your home picture and video share.","archived":false,"fork":false,"pushed_at":"2013-02-21T07:39:49.000Z","size":148,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-25T09:51:09.445Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/integrii.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-01-28T07:21:47.000Z","updated_at":"2025-02-26T23:44:55.000Z","dependencies_parsed_at":"2022-09-16T18:11:24.032Z","dependency_job_id":null,"html_url":"https://github.com/integrii/homebackup","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/integrii%2Fhomebackup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/integrii%2Fhomebackup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/integrii%2Fhomebackup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/integrii%2Fhomebackup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/integrii","download_url":"https://codeload.github.com/integrii/homebackup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248487733,"owners_count":21112188,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T20:38:22.696Z","updated_at":"2025-04-11T21:51:32.784Z","avatar_url":"https://github.com/integrii.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"homebackup.sh\n=============\n\nTurns the chaos of your family picture storage into de-duplicated and sorted niceness; then backs it up.\n\n\nOver the many years of taking digital photography and video, my home server was just piling up with media.  Because I knew that I had a backup script running each week for that file share, I would get lazy and drag everything I found on my PC onto the samba server and consider it safe. To make matters even more complicated, my wife would come along with Picasa and export multiple copies of things into the share in various folder names.  Eventually this turned into complete file chaos and I was rather sure that I had tens or hundreds of gigs of storage wasted.  \n\nEven worse still, I had no idea where anything was.  Folders were all named whatever we felt like naming them at the time of import.  While it made sense at the time - in the long run it ended up being a cluttered mess.\n\nTo fix the problem, I needed to do three things at a regular interval:\n* Remove duplicate files by MD5\n* Sort the files by EXIF data or creation date\n* Back the leftovers files up to a remote FTP\n\nThus homebackup.sh was born.  The script does exactly those things using the exiftags command and lftp.\n\nThe script is bash so it runs nearly everywhere.  This also means it is easy to modify and add your own code.  If you want to do a network rsync instead of lftp, you can just replace the lftp command with an rsync and consider it done.\n\nThe overall idea here is that you can setup this script to run as a cron job every few days and then mindlessly dump any photos you care about into your server.  The script will make sense of it all and delete any duplicates you don't need along the way.  Everything organized, safe and happy.\n\n\n\nsetup\n=====\n\nSetup is almost too simple. All you need to do is make a config with your FTP server credentials in it like below:\n\n~/.homebackup.cfg\n```\nUser yourftpuser\nPassword yourftppass\nServer ftp.mybackupserver.com\n```\n\nThen set the variables at the top of homebackup.sh:\n```\nshare=\"/data/Pictures\" # the target share to operate against\nfiledb=\"/tmp/filelist.txt\" # location for temp list of files\nmd5db=\"/tmp/md5list.txt\" # location for temp md5 list\nlockfile=\"/tmp/backup.lck\" # lockfile\ncheckdays=0 # number of days back to check for dupes - 0 means all\nbackuptries=10 # number of tries to back up your data with a clean return code\nownuser=\"nobody\" # the user that should own all the files in the share\nowngroup=\"samba\" # the group that should own all the files in the share\nnoexif=\"Unsorted\" # name of folder in share to put files that dont have exif data. Relative to share target\nremotedir=\"backup\" # remote FTP target directory for backups\nfiledepth=\"8\" # maximum levels deep to rename files and folders with spaces\nbwlimit=\"262144\" # bandwidth limit in bytes per second\nfileperms=\"775\" # chmod files to this permissions set\ndirperms=\"775\" # chmod diretories to this permissions set\n\n```\n\nYou also may want to schedule this with a cron job.  Run crontab -e as root and enter the following.  The MAILTO variable is optional but will make your server send you email reports to you instead of root (provided your email server works at home).  This cron job is for every three days.  Change the 3 to however many days you prefer.  The 0 0 means on the first minute of the day (midnight).\n\n```\nMAILTO=myemail@mydomain.com\n0 0 */3 * * /path/to/homebackup.cfg\n```\n\ndisclaimer\n==========\nI am not responsible if you run this script and it burns down your house or deletes your life long anime collection.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintegrii%2Fhomebackup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintegrii%2Fhomebackup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintegrii%2Fhomebackup/lists"}