If that needs to be scaled out to many services, then you also want to scale out the data model. Meaning, events in a commit log, and services that subscribe to that commit log to create their own schemas that are tuned for their own needs. The more I think about commit logs, the more I think giant monolithic database schemas are just kind of weird - a strange compromise between history and state.
this sounds like an absolutely enormous pain in the arse (cognitive overhead if you prefer.) what, specifically, is the bigger pain in the arse out there that one needs all this in order to avoid? it must be pretty awful.
also, what happens down the line when you suddenly need a service which is contingent on the data stored in multiple other services? do you add another synchronisation layer over the top?
That's an interesting idea. If I understand correctly, you're suggesting that services simply broadcast the data mutations they are going to perform. Then, any other services that wish to use that data can filter those broadcasted mutations down to what is relevant to the other services, and store the data in a way that is efficient for what the other services want to do.
Are there any examples of things architected in that way?
Doesn't it also run into the problem of needing to version the events that you broadcast?
It's the Kafka cool-aid being regurgitated.
Kafka is the pipe that you need. Micro-services provide APIs to state that they manage that they get from Kafka. That all scales out and services have nice independent failure modes (as long as Kafka doesn't fail).
What it doesn't consider, however, is if you have to perform atomic operations across micro-services (you can't easily do this with existing technology).
And, yes, you would like exactly-once processing of events, which is non-trivial - even with the new exactly-once message reception feature in Kafka.
If your services are completely separable, this approach is great. Otherwise, monolith it.
Event Sourcing [0] & CQRS [1] come to mind. I'm also a fan of Bottled Water [2] that turns Postgres ops into a stream of events that you can then push to other services.
