Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ku1ik/bitpocket

"DIY Dropbox" or "2-way directory (r)sync with proper deletion"
https://github.com/ku1ik/bitpocket

Last synced: 4 days ago
JSON representation

"DIY Dropbox" or "2-way directory (r)sync with proper deletion"

Awesome Lists containing this project

README 

        
# bitpocket



## About



**bitpocket** is a small but smart script that does 2-way directory

synchronization. It uses _rsync_ to do efficient data transfer and tracks local

file creation/removal to avoid known rsync problem when doing 2-way syncing

with deletion.



bitpocket can use any server which you have ssh access to for its central

storage. If you have gigabytes of free disk space on your hosting server you

can finally make use of it.



## Installation



Clone repository and symlink `bitpocket` bin to sth in your `$PATH`:



    $ git clone git://github.com/ku1ik/bitpocket.git

    $ ln -s `pwd`/bitpocket/bin/bitpocket ~/bin/bitpocket



Or download script and place it in a directory in your `$PATH`:



    $ curl -sL https://raw.github.com/ku1ik/bitpocket/master/bin/bitpocket > ~/bin/bitpocket

    $ chmod +x ~/bin/bitpocket



### Setting up master



Create an empty directory on some host that will be the master copy of your files:



    $ ssh [email protected]

    $ mkdir ~/BitPocketMaster



### Setting up slaves



On each machine you want to synchronize initialize an empty directory as your bitpocket:



    $ mkdir ~/BitPocket

    $ cd ~/BitPocket

    $ bitpocket init [email protected] ~/BitPocketMaster



## Usage



After installation, you use the `bitpocket` command for synchronization and other tasks.

Running `bitpocket help` will display the following message.



    usage:  bitpocket { init [] 

                      | sync | help | pack | log | cron | list }



    Available commands:

       sync    Run the sync process. If no command is specified, sync is run by

               default.

       init    Initialize a new bitpocket folder. Requires path and optional

               remote host params. Remote path must already exist.

       pack    Pack any existing (automatic) backups into a git repository.

       cron    Run sync optimized for cron, logging output to file instead of

               stdout.

       log     Display the log generated by the cron command

       list    List all files in the sync set (honoring include/exclude/filter

               config).

       help    Show this message.



    Options:

       -f, --force      Clean up stale lock files automatically

       -p, --pretend    Don't really perform the sync or update the current

                        state. Instead, show what would be synchronized.



    Note: All commands (apart from help), must be run in the root of a

          new or existing bitpocket directory structure.



### Manual sync (bitpocket sync)



To synchronize your local slave with master just run _bitpocket sync_ inside your

bitpocket directory:



    $ cd ~/BitPocket

    $ bitpocket sync



Ensure that you run bitpocket at least once immediately after creating a new slave and

before adding new files to the slave directory. If there are files in the master they

will be pulled into the slave.  You may then move files into your slave directory and 

they will be detected as added.



### Maintaining backups (bitpocket pack)



bitpocket does not include a full-fledged versioning system at the moment, but

it does automatically create a local backup of any files before overwriting or 

deleting with changes from the BitPocketMaster. These backups are placed into a 

timestamped directory (one directory per sync). The path to this directory is:



    .bitpocket/backups/YYYY-MM-DD.hhmmss



As these files accumulate, you may want to remove them, but you can also run

`bitpocket pack` to combine all the backup files into a git repository contained

within the _.bitpocket_ directory:



    $ cd ~/BitPocket

    $ bitpocket pack



This requires an installation of _git_, but

allows all the space saving advantages of _git_ when making repeated changes

to the same files.



There is a discussion about potential directions for versioning direction here:

[github.com/ku1ik/bitpocket/issues/15](https://github.com/ku1ik/bitpocket/issues/15)



### Redirecting output to log file (bitpocket cron)



If bitpocket is run with the cron parameter ( _bitpocket cron_ ), it will perform

a sync, but instead of showing the progress on stdout, it will redirect all 

output to a log file: 



    $ cd ~/BitPocket

    $ bitpocket cron



As the name of this parameter implies, this is mainly useful when running bitpocket

through the _cron_ command. (See "Automatic sync with cron" for more information 

about how to configure this).



### Displaying logs (bitpocket log)



When running bitpocket in cron with `bitpocket cron` it will append its output

to _.bitpocket/log_ file. You can review the tail end of an existing log file,

or watch live log as it is generated, with following command:



    $ cd ~/BitPocket

    $ bitpocket log



### Displaying list of files to be synchronized (bitpocket list)



You may want to know which files will be synchronized before actually performing

the syncronization. You can verify which files are in the synchronization set

by running `bitpocket list`:



    $ cd ~/BitPocket

    $ bitpocket list



Note that this does _not_ list changed files, it only lists all the local files 

that bitpocket will look at in determining which files to sync. Also, note that

if there are new files in the master that will be added on a sync, they will 

not be included here. This command is only intended to verify which files are

in the synchronization set. (See "Configuring file exclusion and inclusion" for

information about how to control which files are in the synchronization set).



## Configuration



### Automatic sync with cron



Add following line to your crontab to synchronize every 5 minutes:



    */5 * * * * cd ~/BitPocket && nice ~/bin/bitpocket cron



Note that cron usually has very limited environment and your ssh keys with

passphrases won't work in cron jobs as ssh-agents/keyrings don't work there.

Thus it's preferable to generate passphrase-less ssh key for bitpocket

authentication:



    $ cd ~/BitPocket

    $ ssh-keygen -t rsa -C bitpocket-`hostname` -N '' -f .bitpocket/id_rsa

    $ ssh-copy-id -i .bitpocket/id_rsa [email protected]



and uncomment line with `RSYNC_SSH` in _.bitpocket/config_ file.



### Configuring file exclusion and inclusion



If you want some files to be ignored by bitpocket you can create

_.bitpocket/exclude_ file and list the paths there:



    *.avi

    jola

    /misio.txt



_*.avi_ and _jola_ will be matched anywhere in path, _misio.txt_ will be

matched at bitpocket root dir ( _~/BitPocket/misio.txt_ ).



This exclude file is passed to `rsync` as `--exclude-from` argument, check `man

rsync` for _INCLUDE/EXCLUDE PATTERN RULES_.



You can set up even more advanced exclusion/inclusion rules. In all, there

there are three files that you can create to change this configuration:



    .bitpocket/exclude

    .bitpocket/include

    .bitpocket/filter



Be aware that all the quirks from rsync exclusion/inclusion rules carry over

into bitpocket. If you decide that you need such advanced configuration, make

sure that you understand those rules very well, and consider double checking

them before syncing by running `bitpocket list`.



### Backup options



Both local and remote backups can be enabled or disabled in the configuration.

By default, local backups are enabled and remote backups are not. Based on how

intend to use bitpocket, you may wish to change this behavior. Add or change

these lines in your _.bitpocket/config_ file:



    # BACKUPS=true

    # REMOTE_BACKUPS=false



### Custom rsync options



You can pass additional switches to `rsync` by setting `RSYNC_OPTS` in

_.bitpocket/config_ file. Generated config file includes (commented out)

example setting for dereferencing symlinks:



    # RSYNC_OPTS="-L"



Just uncomment it and change at will.



### Slow sync callbacks



When syncing takes more than 10 seconds (SLOW\_SYNC\_TIME setting) bitpocket

can fire off user provided command in background. This can be usefull to notify

user about long sync happening, preventing him from turning off the machine

during sync etc.



There are 3 settings that can be enabled in _.bitpocket/config_ file:



    # SLOW_SYNC_TIME=10

    # SLOW_SYNC_START_CMD="notify-send 'BitPocket sync in progress...'"

    # SLOW_SYNC_STOP_CMD="notify-send 'BitPocket sync finished'"



Just uncomment them and change at will.



You can show tray icon during long sync with

[traytor](https://github.com/ku1ik/traytor) and following settings:



    SLOW_SYNC_START_CMD='~/bin/traytor -t "BitPocket syncing..." -c "xdg-open ." .bitpocket/icons & echo $! >.bitpocket/traytor.pid'

    SLOW_SYNC_STOP_CMD='kill `cat .bitpocket/traytor.pid`'



### Failsafes



You can add a remote mount point check to ensure a remote path is available

prior to the sync running. If the folder is not mounted on the remote server,

then the sync will be aborted. The path must be an absolute path which is

expected to be a mountpoint on the remote server. Add or change this lines in

your _.bitpocket/config_ file:



    # REMOTE_MOUNTPOINT=/



## Author



* Marcin Kulik | @ku1ik | https://github.com/ku1ik | http://ku1ik.com/

* torfason | https://github.com/torfason