> How often the translog is fsynced to disk. Defaults to 5s. [...] In this test we kill random nodes and restart them. [...] In Elasticsearch, write acknowledgement takes place before the transaction is flushed to disk, which means you can lose up to five seconds of writes by default. In this particular run, ES lost about 10% of acknowledged writes.
Something bothers me about this: if the bug was merely a failure to call fsync() before acknowledging an operation, then killing processes shouldn't be enough to cause data loss. Once you write to a file and the syscall returns, the written data goes into the OS's buffers, and even if the process is killed it won't be lost. The only time fsync matters is if the entire machine dies (because of power loss or a kernel panic, for instance) before those buffers can be flushed.
So is the data actually not even making it to the OS before being acked to the client? Or is Jepsen doing something more sophisticated, like running each node in a VM with its own block device instead of sharing the host's filesystem?
It's probably the flush interval that also defaults to 5 seconds that is responsible for losing data from simple a kill -9 to the process. Before that the data is in the process memory but not the OS buffer.
I believe Jepsen is killing (VM) systems, not just processes. It is meant to model the failure modes in a data center, which would include machines bursting into flame, etc.
Hmm. Upon looking at the documentation [1], it looks like the recommended configuration is to run nodes in LXC containers. Maybe I'm mistaken, but I thought "LXC" basically just meant some automation around kernel features like chroots, user namespaces, etc. In which case, shouldn't the containers share the host's VFS and pagecache?
Write acknowledgement in this case means committed to memory, i.e. the transaction was recorded to a translog buffer (which is fsync'ed to disk every 5 seconds), and changes were applied to the database in memory. The translog (aka write ahead log) has a separate thread which will do fsync every 5 seconds.
Unless ES blocks on the translog fsync, you can have lost transactions in the 5 second window.
The only thing that fsync() does is take data which is already in the operating system buffers associated with a file descriptor, and write it physically to the disk. If the thread that runs every 5 seconds is really just fsync'ing, then when the process gets killed, any un-synced data is still in those buffers and will be available to read when it restarts, exactly the same as if fsync() had been called. The OS is responsible for maintaining a consistent view of the filesystem such that, unless the machine totally fails (or you circumvent the FS and actually inspect the live block device), fsync() appears to be a no-op.
It sounds like what you're saying is that the documentation is wrong, and that the translog thread is actually pulling data from Elasticsearch's internal buffers and writing it to files. In that case, the documentation which refers to that operation as "fsync" is very badly misleading because it disguises what failure modes it's actually protecting against.
The point of a return from fsync is that you are guaranteed the file has been written to disk[1]. If you don't block on fsync, you can't guarantee the file was written to disk, because the server may have died in any number of ways.
[1] This guarantee occasionally fails too; If you have a battery-backed NVRAM RAID controller, the guarantee is that the write has hit the NVRAM controller with the expectation that it will hit a disk before the battery dies. Throw in a 72 hour power outage, a controller failure, or a massive disk failure, and you can't even guarantee that.
No, I understand that. Maybe I'm not explaining my point properly, so I'll try again:
If you issue a write() syscall from a process, and the syscall succeeds, then the data that was written is present in the OS's cached view of the filesystem, even if the process dies a nanosecond later. That view is shared consistently by all processes on the system. It's true that the changes may not actually be stored persistently on disk, but that difference is unobservable unless something happens to make the kernel lose its cached data.
So from the test suite's point of view, unless part of the test involves actually killing VMs and not processes, it should not be possible for the results to depend on whether or not fsync() was called.
Something bothers me about this: if the bug was merely a failure to call fsync() before acknowledging an operation, then killing processes shouldn't be enough to cause data loss. Once you write to a file and the syscall returns, the written data goes into the OS's buffers, and even if the process is killed it won't be lost. The only time fsync matters is if the entire machine dies (because of power loss or a kernel panic, for instance) before those buffers can be flushed.
So is the data actually not even making it to the OS before being acked to the client? Or is Jepsen doing something more sophisticated, like running each node in a VM with its own block device instead of sharing the host's filesystem?