Hi, here to answer question number 4.

Short answer: Our configuration data is most similar to git. Any machine can be used as an administrative node via the WebUI or the CLI. It will make changes to the metadata which then get pushed to the other nodes. If 2 nodes make conflicting changes you get a conflict which the system will help you to merge.

Long Answer Cluster configuration is stored in semilattices which are a neat mathematical structure with a few very desirable properties. Semilattices support have a join operator. For our cluster metadata joining is the means by which metadata is updated. When one server connects to another the two swap metadata and each joins the other's metadata into his own. In essence learning what the other knows.

There are two properties in particular of the joining that are nice. First off joining is commutative. This means machines can exchange data in whatever order they want and get the same result at the end. Secondly they're indempotent. That means machines can resend their data without fear. The value doesn't change if the same value is joined in twice. These help us with a lot of the worries of distributed systems.

Interesting. Do you have any way of checking that the change has actually propagated through the system before starting to act on it? Is the system consistent at all times?

If I understand correctly, the client can connect to any instance and its request will get routed appropriately. Let's assume that you take a master offline and promote one of the replicas to be a new master. Won't that lead to a window in which (from the point of view of different instances) there are two masters at the same time and some writes are sent to the wrong instance?

EDIT:

One solution for such things is to use something like Zookeeper (or some other system whose documentation mentions "Paxos" ;)). Have you considered that? How does what you are doing compare with that?

Joe may be responding to this soon, but in the meantime I'll chime in. There is no way to verify the propagation reliably without either introducing strong performance inefficiencies (e.g. two phase commit protocol), or divergence (paxos, semi lattices, etc.) In our implementation we're using immediately consistent algorithms for data, but eventually consistent algorithms for cluster metadata. This means that if there is a metadata conflict, the user is presented with an issue (via the web ui or CLI) that they have to resolve. We'll also be adding automated resolution soon.

We basically have something very similar to zookeper baked into rethinkdb. We wrote it internally from scratch to better suit the needs of our architecture.

Now that you mention it, it would be very nice to have a database suited for configuration that behaved like git in that branching, restoring old states, reverting selected commits was built in, while at the same time supporting ACID features and replication?

Does anyone know if something like that exist?

1. As long as active dataset everything fits in RAM, performance will be great. E.g. you can have terabytes of data but as long as the actively accessed dataset is < ~80GB the system will perform well. Once things get out of RAM, everything will work well if you have an SSD. On a rotational disk, performance will degrade very very quickly. This is a bottleneck on all modern databases, but clustering makes this problem go away because you can effectively increase the amount of RAM at linear cost by just adding more nodes (e.g. two nodes at 100GB of RAM each cost about four times less than one node with 200GB of RAM).

2. We do do block-level replication. On each node of the btree we store replication timestamps. When a node asks for new data, we can cull away parts of the tree the node has almost instantly. So replication is very very efficient for most OLTP workloads. We don't have statement-level replication yet, so if you do a range update on a large table, we'll have to replicate data block by block. It'll take a while to add statement-based replication - we'd have to do a pretty significant refactoring to make it happen.

3. Either. Replicas are great for failover -- if the master dies, you just failover and a replica picks up where the master left off. If you're ok with out-of-date reads, you can also hit replicas directly (e.g. for reports, etc.) and spread out the read load across the cluster.

4. This is a really complex question - we didn't document this because doing it properly would take a lot of time. I'll ask jdoliner to chime in -- he designed the architecture and wrote most of the code, perhaps he can describe it succinctly while we write deeper docs on this :)

5. We use protocol buffers between the client drivers and the server. We picked that because there were libraries for the initial three languages we picked (Ruby, JS, Python), they were really easy to use, and very efficient. We could also have a single spec for the client/server API. Internally we use our own serialization scheme which allows us to dump arbitrary C++ objects on the network. It doesn't support other languages (which we didn't need), but is much more versatile for writing complex cross-machine code.

Small correction to this. There actually wasn't a library for JS we had to write that for ourselves.