I think your fundamental assumption is based on "possible" performance. I would argue that required performance has flattened over the last few years. A nexus 4 is a perfectly serviceable phone still.
I think as technology improves, the advantages of modularity with trump any loss of performance.
More battery than generic performance. Performance may have improved but components haven't shrunk much and batteries definitely haven't shrunk.
You can likely reclaim some space by making a really tiny chipset, but batteries and screens combined are over 70% of your phone's total size. Project Ara makes components larger (due to connectors and housing), so you need to find space from somewhere, and there isn't a lot to be taken from most components (they aren't very large already).
I think as technology improves, the advantages of modularity with trump any loss of performance.