Since one person ends up paying the whole thing, it's only fair if the same group of people goes out together frequently, such that there is a reasonable expectation that everyone will end up paying their fair share eventually.
Another downside is the liquidity requirement. To adequately participate, you need to be able to pay for a fairly large number of meals.
"A common complaint about stochastic schemes is that they’re “only fair if you do it repeatedly with the same group of people”. That’s true if you insist on ex post fairness. We’re usually happy with ex ante fairness. Consider selling me a (perfectly fairly priced) lottery ticket for a dollar. That’s guaranteed to be unfair, ex post. Either you sold me a worthless piece of paper for a dollar, or I got a million dollars and only paid a dollar for it. But the fact that none of us knew which would happen made the one dollar price fair. Same story with venture capital investment, for example. You may need a gambling mentality to be down with it, but it’s quite fair even if only done once. The fact that it averages out in the long term to be perfectly fair ex post is icing on the cake."
Another downside is the liquidity requirement. To adequately participate, you need to be able to pay for a fairly large number of meals.