But that misses their own threshold. Rate 4 and you're not asked a thing (if I understand you correctly), but the driver might be kicked out if enough people do the same, right?

If there 4.5 is the minimum average rating, everyone not rating 5 should be asked what was wrong. Or .. fix the rating so that not everything needs 5 out of 5.