Wuala, SpiderOak, and Dropbox: Feature Summary and a Little Testing

Dropbox, SpiderOak, and Wuala seem to be the current contenders for someone who wants synchronized cloud storage together with Linux/Mac support. I've tried all three and come up with a few observations that may be of use.

First I did a little test of Dropbox, SpiderOak, and Wuala. I wrote to a text file on one machine inside my LAN and timed how long it took for the file to appear on the other machine. It took 5, 24, and 68 seconds respectively.

These times are not surprising given that Dropbox implements a "LAN sync" capability, whereas SpiderOak goes through the central server, and Wuala not only goes through the server (in Europe; I'm in Indiana) it further does not use inotify, i.e. it has to periodically scan the file system for changes.

But there are some major reasons to chose SpiderOak or Wuala over Dropbox. One of them is that both SpiderOak and Wuala have client-side encryption such that their employees shouldn't be able to access your files.

Further, there's the handling of symbolic links, which I have complained about before ***. Wuala syncs them. SpiderOak ignores them (which is still better than Dropbox, which follows them resulting in weird behavior).

Wuala also has some other unique desirable features as well as some major technical drawbacks.

1. Wuala used to allow a P2P mode in which users traded space on their own machines to become part of the storage network. A very neat concept and a good way to get a lot of cloud storage at a reasonable price. (By contrast, 3TB of storage on Amazon, to match one external HD, is $5000 a year).
2. Wuala has a time travel view that lets you see the previous state of an entire folder. Dropbox and SpiderOak have a single file fixation. You can view previous versions of an individual file using their respective UIs. This is great for, say, a word document, but very poor for a directory containing code or multiple latex files.
3. FUSE support. Wuala allows the remote drive to be directly mounted via FUSE without requiring everything to be sync'd locally. In theory this would seem to combine the benefits of something like Dropbox with traditional network file systems like NFS and AFS.

And then the drawbacks. Unfortunately these are drawbacks that strike at the core -- the bread and butter of syncing. First, as mentioned above, Wuala doesn't use inotify for to allow the OS to alert it of changed files. Second, Wuala doesn't allow exclusion of files based on name or extension -- a major drawback it shares with Dropbox which makes it inefficient to rebuild compiled projects inside a synced folder. (Note: Wuala also used to not support incremental/differential uploads. That has been implemented.)

In summary:

Wuala

• + Syncs symbolic links
• +++ Time travel view
• + FUSE-mount remote drive without syncing
• - No inotify support yet
• - No exclusion of files from sync yet
• - No systray icon / background mode under mac os yet
• - [US] Data centers in Europe

SpiderOak

• + Ignores symbolic links (rather than doing something terrible)
• + Exclude files from sync via extension or regex

Dropbox

• + LAN sync
• ? More geographic coverage (amazon)

*** P.S. In an ironic act of data non-preservation it looks like the work-arounds I'd posted to the dropbox wiki (http://wiki.dropbox.com/Symlink%20Semantics%20and%20Workarounds) were lost because they took down the entire wiki.

Dropbox - another way to shoot your foot

Whew, the data loss in the last post actually wasn't that bad because the folder was version controlled anyway (and stored on another server). I thought only using dropbox for working copies would keep me safe.

So much for that idea! In a sleepy state just now I managed to get Dropbox to delete scattered files outside of my Dropbox folder!

This comes back to the symlink semantics, but it can bite you in strange ways. Here's how you can really screw yourself. First, let's say you've got a working copy that stores all your notes and other reference text files that you search often:

   mv foobar ~/dropbox/foobar

Let's further suppose that you are on a Mac or Linux client and foobar contains symbolic links that point to other scattered text files elsewhere in your personal archives that you may want to have at hand (for grepping).

Next, you want to make a major change to foobar, so you back it up, just to be safe:

   cp -a ~/dropbox/foobar ~/dropbox/foobar_bak

A while later you come across this backup (on a different dropbox-linked machine) and delete it:

rm -rf foobar_bak

BAM! That's it, you just screwed yourself.

How? When Dropbox went back to your original Mac/Linux machine to delete foobar_bak, it decided to FOLLOW THE SYMLINKS. That's right, it deleted the original files. Even though the original foobar directory is still there, its links are now broken of course.

The whole point of these links was organizational. They pointed all over the place. Even if you have backups you now have to track them down and restore those scattered files. (Which is what I just spent my time doing.) I guess I'm a synchronization masochist because I seem to ask for this kind of punishment.

Bottom line. If you're a power user, avoid Dropbox or keep your usage very light, e.g. stick an office document in there every once in a while. Or be ready to suffer.

A New Dropbox Nightmare

I was cautiously optimistic and hopeful -- "maybe Dropbox is the synchronization solution I've been waiting for" -- but that sentiment is quickly being replaced with jaded.

First, it is good to recognize that backup and synchronization (mirroring) are orthogonal. Mirroring only insures that mistakes get propagated. Well, I just experienced my second Dropbox spontaneous deletion event, which, due to the miracle of synchronization, was applied to all of my computers and to the cloud.

What can be worse than a silent deletion event for a service like Dropbox? Nothing that I can think of. I can't babysit the 96,000 files in my Dropbox to catch something like this.

Now the details. I came across a directory that had 95% of the files deleted (leaving a random few remaining). Scrolling back through the event history I found that this happened after linking a new client, which happened to be a Windows virtual machine. Below is a snippet of the event audit trail. The entire series of events happening at 10:09 was spurious (including deleting 411 files and 206 folders).

You edited the file .saves-25811-rnmacbook13~.      11:27 AM
You renamed the file intel_aes.c to test038.h.ERR. 10:09 AM
You renamed the file setup to test047.h.ERR.        10:09 AM
You added 686c9fad58174636354 and 78 more files.    10:09 AM
You edited simple_cycle.out and 49 more files.      10:09 AM
You deleted 6c88b2f3aaaa585db12 and 411 more files. 10:09 AM
You added eb and 6 more folders.                    10:09 AM
You deleted intel_aes_lib and 206 more folders.     10:09 AM
You moved 04 and 24 more folders.                   10:09 AM
You added the folder fabl.                          12:42 AM
The computer rn-waspvm was linked to your account   12:24 AM
You edited the file ghc_setup.txt.                  12:24 AM

A couple additional oddities show up in the above:

• it waited over 9 hours after the new client was linked to go crazy
• Files were "'renamed", intel_aes.c to test038.h.ERR Huh? Is that a remove being aliased with an add to create a fake rename? Possibly a secondary bug.

I went back and checked out the state of that Windows client just now. A clue! The only files it successfully downloaded were the ones not deleted in the above nonsense. So Dropbox confused a partially downloaded directory with a new version of the directory (deleting most of the contents). Unison has never done anything like that to me in ten years of heavy use!

It may be corroboration that the Windows client currently thinks it "hasn't connected yet" [Screenshot below]. Oh really?

Well, perhaps this is "fool me twice, shame on me". I should have known better -- my wife tried Dropbox about a year ago, dumped her ~50 gigabyte personal archive in all at once, then moved a folder, and ended up with her data in a never-fully-sychronized, confused state. It still hasn't been completely fixed; there's a backup, but also new stuff mixed into the messed up Dropbox version. I chalked that one up to bad behavior while under load and before the first sync is successful (plus the dangers of syncing being too implicit and automatic). Still, I wanted to give Dropbox another chance; trying it myself this time, thinking that:

• It's surely improving and ...
• I would be more cautious, spoon-feed it, and pay more attention to its syncing state.

But clearly that didn't save me.

Dropbox semantics - oh that there were such a thing

After some recent Dropbox problems I've been having (not the first unfortunately) I added an entry on the Dropbox wiki explaining what happens if you are so impertinent as to add directories containing symbolic links to your Dropbox folder:

http://wiki.dropbox.com/Symlink%20Semantics%20and%20Workarounds

[UPDATE - dropbox took down their wiki, but I reposted the page here.]

One day maybe we'll have something with the correctness and robustness of Unison and the convenience of Dropbox. People will dance in the streets. I think Dropbox can be that solution if they work at it.

UPDATE - these issues are explained at greater length in this post:

http://aurelio.net/articles/dropbox-symlinks.html