> On the other hand, if you have the stream, do you need Kafka? Can't you just push it directly to ElasticSearch?
I think the separation is something very nice here. We have something like an Apache Storm topology (though we use a own Mesos based framework here) for every datastore we want to populate. If we want to add a new datastore we just have to find a library for it and can whip up a new topology. That is much more convenient than having to build support for each datastore into something central like Botteled Water and can be tweaked nicely to the specialities of the datastore.
> since you don't want to keep the entire change log around forever (and can't, since the log only starts at the point when you started running this system).
If we initialize a new Kafka topic, we push the relevant data into it once from the production database, and after that Kafka dedupe will keep it from growing too large.
The separation is nice, although I would counter that if your only primary data store is Postgres, and you want to go the logical decoding route, Postgres already has the queue: The decoded transaction log. There's no need for a queue on top of a queue. All you need now is a client that can process the log sequentially and emit each change to the appropriate data store.
Things like Kafka would be more appropriate if you have multiple producers that aren't all Postgres.
I think the separation is something very nice here. We have something like an Apache Storm topology (though we use a own Mesos based framework here) for every datastore we want to populate. If we want to add a new datastore we just have to find a library for it and can whip up a new topology. That is much more convenient than having to build support for each datastore into something central like Botteled Water and can be tweaked nicely to the specialities of the datastore.
> since you don't want to keep the entire change log around forever (and can't, since the log only starts at the point when you started running this system).
If we initialize a new Kafka topic, we push the relevant data into it once from the production database, and after that Kafka dedupe will keep it from growing too large.