I ran a little comparison with both Oh-my-zsh and Prezto with roughly the same plugins: http://i.imgur.com/cDSRZuO.png. There's almost a 0.4s difference. In practice it makes for a noticeable 0.5-1s difference (with all the other stuff I have in my .zshrc).