The single-level approach is what Recut does too, and it tries to take a guess at a threshold with clustering but it's not always perfect. Maybe a better way to go would be a dynamic noise gate or kalman filtering or something.

Vidbase is looking awesome btw! I bet it's going to be huge. It looks like you've paid an insane amount of attention to the details.