It amazes me that Syncthing handles rapidly-changing files better for me than Dropbox. Where Dropbox has left different versions of files on different computers even after they have changed multiple times, Syncthing happily synchronizes them without issues. And the fact that Syncthing is open source and allows me to keep files on my own machines is a big plus. However, I would love to see a paid option for easy backup of selected files to a machine in the cloud. Some people want their files backed up and accessible through a web service and I wish I could recommend Syncthing to them as well.
Just a heads up: that almost certainly means SyncThing is failing to recognize some classes of conflicts and concurrency races, and is destroying some changes. Doing this right is a hard problem. I know they can be a pain to deal with, but we don't spin out conflicts for no reason!
Do you have any specific bug or design reference to the Syncthing codebase that illustrates this, or do you simply believe that it's impossible to do it faster than Dropbox does it?
Fwiw, i frequently get notifications from Syncthing that it's found a conflict it can't resolve and asks me to resolve manually.
Sorry, not super familiar with Syncthing. It's definitely not impossible to do better than Dropbox, but to some degree, as in all things, it's a function of engineering resources, telemetry, and usage.
We've likely put 25-100x more resources into solving this problem over the last 10 years, and we just have a lot more data b/c we have 100s of millions of users on lots of platforms using Dropbox with every application you can imagine. So we're able to tease out the "long tail" of weird file system and application behavior in a way that's very difficult for smaller projects. Truly durable conflict management in the face of arbitrary mutations by user applications on the filesystem ends up being a really, really hard problem to cover exhaustively. The Dropbox client handles literally hundreds of special cases.
So, yeah, I believe it is (in general) safe to assume that Dropbox is probably doing A-more-correct-thing for a complicated (and admittedly confusing) reason when it comes to sync behavior. But we're not perfect--we do still find surprises from time to time, so feel free to contact support if you see something that looks wrong!
That was my initial thought - however after rereading his post he seems to refer to changing files quickly on a single system and seeing the changes replicated to multiple additional systems. This reading suggests Dropbox fails to keep up and eventually decides some of its own yet-unfinished syncs in additional systems are conflicts.
Are you saying that you solve conflicts specifically for different types of applications (e.g., Word, Photoshop, Excel)? That's impressive. However, now I'm wondering how I get my own apps supported.
I think that Dropbox typically handles conflicts quite well, and the issues I had are more likely bugs outside the conflict resolution implementation. I was a bit brief in my comment above, so let me elaborate in case you or someone else is interested:
The issues I had didn't result in conflicted files. Rather, after making a big change (i.e. switching git branches) some files were never updated or synced. Dropbox stopped picking up changes in the folder and eventually removed new changes once restarted.
The order of events were something along the lines of:
1) Did work on computer A that caused massive file changes (i.e. moving between git branches).
2) Moved to computer B to continue work.
3) Noticed files were old or missing on B.
4) Syncing files in some other folders worked, but nothing happened in the folder with missing files.
5) Restarted Dropbox on both machines in hope that this would trigger a fresh sync.
6) Observed files being reverted to old versions or deleted on machine A.
The end result was that Dropbox threw away the changes I had made on A and left me with the original state of B. I was able to recover the changes from a backup, so it was no big deal in the end (although it left me a bit scared I could have lost those files without noticing).
I was in contact with Dropbox support about the issue and explained in detail what I had done and what happened. I was offered help to recover the files, but since I had already done so, I just told them I didn't need any more support on the issue. I thought it might be because /proc/sys/fs/inotify/max_user_watches had a low value on one machine, so I wrote back that they might want to add back the old warning about this. However, the same problem with deleted files happened again after I had verified that this value was high enough on all machines.
I have also seen how a script run by a colleague managed to confuse Dropbox. The script was running a test which repeatedly created and deleted the same file before checking its correctness. Running the script in the Dropbox folder left him with some old version of this file and a failed test. Running the scirpt in a folder outside Dropbox left him with the correct final version of the file. He was only working on one machine.
And yes, I know it's "bad" to run scripts like this or switch git branches on top of sync software, but it happens, and it is interesting to see how different software handles these cases.
It should be noted that Dropbox usually handles these massive file changes well, so moving to Syncthing has for me been more about it being open source and the possibility to keep files on my own machines. I was just glad to see that Syncthing also handles heavy use cases gracefully.
Last I recall it didn't, which kind of soured me on it - in an ideal world every sync operation would be prefaced with "send a backup to one of my backup endpoints first".
Backing up your filesystem can and really should be part of an entirely different process. Let syncthing focus on syncing. Get CrashPlan, rsync, or some other process going on your filesystem for backups. Unless you are hoping for backing up every single version of your documents.
