https://github.com/telent/muddle

MuDDLe / MailDir Down Load. : Faster than rsync! Simpler than an IMAP server!
https://github.com/telent/muddle

Last synced: 13 days ago
JSON representation

MuDDLe / MailDir Down Load. : Faster than rsync! Simpler than an IMAP server!

• Host: GitHub
• URL: https://github.com/telent/muddle
• Owner: telent
• Created: 2014-01-29T23:41:42.000Z (almost 11 years ago)
• Default Branch: master
• Last Pushed: 2014-02-02T23:30:25.000Z (almost 11 years ago)
• Last Synced: 2024-10-12T00:29:55.598Z (about 1 month ago)
• Language: Ruby
• Homepage:
• Size: 129 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# MuDDLe / Maildir Download

Faster than rsync!  Simpler than an IMAP server!

As Dr. Andrew Tridgell himself said:

> There are always more efficient algorithms than rsync. If you have

> structured data, and you know precisely the sorts of updates, the

> constraints on the types of updates that can happen to the data,

> then you can always craft a better algorithm than rsync.

  -- speaking on the Rsync Algorithm at OLS, 21 July, 2000

     http://olstrans.sourceforge.net/release/OLS2000-rsync/OLS2000-rsync.html

In the case of maildir, the above applies in spades:

- files once created are never changed

- most of the time when you've got something that looks like a file

  change, it's actually a cross-directory rename

- chances are that you can identify most of the changes on most

  occasions just by looking at the files with ctime newer than the

  last time you did a sync

## Requirements

On the client, it needs Ruby of some kind (possibly 1.9 or later, I've

forgotten how to use earlier versions, but this may be fixed to

accommodate 1.8.7 later)

On the server it needs a find(1) command which supports the `print0`

option, and cpio.  Possibly GNU cpio at that, haven't tested anything

else

## Invoking it

Do something like

```

$ ruby ./muddle.rb --verbose [email protected]:/home/remotename/Maildir ~/mail-from-server/

```

varying the paths and directories as appropriate.  You may first wish

to edit your `.ssh/config` file to insert a stanza for your mail host

that turns on compression:

```

Host mail.example.com

        Compression yes

```

(A future version will install as a Gem, and probably will also add a

-C option so that you can turn compression on only for maildir syncing

and not for everything)

## Basic algorithm

* given the server cwd is a maildir

* and the client cwd is a maildir

* then for each message_name on server matching new/name or cur/name:*

    *  unless there is already new/name or cur/name:* on client

        * copy message to client:tmp/$(basename message)

        * then rename it to client:cur/name:{existing suffix or "2,"}

## Making it faster

Instead of copying the files one at a time, we tar them up and

transfer in bulk, meaning first that the file is big enough for TCP's

window size and congestion control stuff to get its teeth into, and

second that if you enable ssh compression then the later messages will

be compressed using the dictionary created by the earlier ones.