Consul has some nice features around change notification that make features like pushing a change in conf without needing a code deploy trivial to implement. Also consul servers are strongly consistent but agents are eventually consistent, fwiw, so that statement will mean different things depending on how you've set up your infrastructure.

There's also nothing to stop you from pushing config into git and then having a build that just syncs files to consul kv.

That feature is hacked around http, where the client is constantly connected over http (through long standing connections) and reconnects every 5 minutes (by default) if there was a change. You would be much better to use Zookeeper for this (through watches) even though Zookeeper is also not the best tool for this for similar reasons.

Using a distributed locking system for notification is really missing why it is for, why not use Postgres' notifications, AMQP is it because it's easier to tell it is not the right tool for the job?

I currently have colleagues implementing an event bus (although the use cases they mentioned is that they will most of the time send messages to individual machines or small group of them) for over 10k machines. Their first choice was to use Consul's KV, but after persuading them that we potentially could have scaling issues, they decided to drop it, unfortunately they are still misguided and plan to use Serf which will have its own issues as well, this time it might be network capacity.

The issue though is much simpler. We are all communicating through IP protocol, which is inherently point to point, why building abstractions to the problem, when network already matches perfectly the requirements. I think we are so used to using a tool for everything that it feels weird to not have to use one. I was able to have a working POC in 10 lines of Python code (of course a final solution would be more). Argument against using what I proposed was to not implement custom solutions because of maintenance issues (what will happen if I decide to quit), the thing is that one will write far more code to implement the solution with Serf or Consul.

> Also consul servers are strongly consistent but agents are eventually consistent, fwiw, so that statement will mean different things depending on how you've set up your infrastructure.

This is not true, KV is strongly consistent including agents (otherwise there's no point in being strongly consistent since all communication supposed to happen through agents), you could relax requirements on individual requests by adding ?consistency=stale, but if your application makes all requests this way, then it is a sure sign that you chose a wrong tool for the job.

> There's also nothing to stop you from pushing config into git and then having a build that just syncs files to consul kv.

Similarly there's nothing stopping you from using Solr as NoSQL storage, encode data using thrift and then store it in 2 column table (id and blob) in Postgres (then have a tool to edit the data directly which encodes/decodes the thrift data), using Coherence (Oracle's P2P caching solution) as primary source of data that is ETLed from Postgres data[1] etc. All those things I have seen done and it is really painful to watch.

[1] Perhaps this doesn't look as bad, but ETL to coherence took 5 hours, and Coherence since it was a caching solution if it detected a network partition just to be safe it decided to invalidate all data it had. The solution was to throw money at a problem, so why not have 2 or 3 environments with initial setup so while something happens to one you can switch to another while reloading the data. This kind of worked until a contractor installing a security system in a data center decided to plug in a dumb switch to two main distribution switches configured to work in HA, creating a loop and shutting entire network down. The site was down for hours until all of the data was loaded back.