Out of interest, is there no way to rearchitect the whole thing to be event-based, ie more like a producer-consumer situation? Or do you have to loop back through all the source data and poll every hotel to fetch its current prices?
Sure, it is producer-consumer - we use Postgres' LISTEN/NOTIFY mechanism (mostly because we have no other use cases for queueing, so "exploiting" an already existing feature in Pg was easier).
The example in article shows all hotels getting refreshed, but that's just because that's the quickest way to reproduce the problem locally. In reality we refresh/reindex only those hotels which have changed since the last refresh - over day(s) this accumulates and the OOM was actually happening not immediately on the first refresh, but after a couple of days (which is part of what made it difficult to catch).
