That's an interesting question that I don't think anyone would be able to quantify, just because performance is so subjective to the applications that are run. In an interview, Alan Kay remarked that bad processor architectures degrade performance by three orders of magnitude [1]. If he's right, that's a pretty hefty tax. But I don't see anyone being able to prove or disprove his hypothesis. Maybe most of the tax is an inevitable byproduct of increasingly complex chipsets.
I seem to remember from one article that is was the same amount of transistors as the older ARM cores (not the A8 or A9). This is a pretty big anchor to overcome.
