Could just be your hosting. For example, with Media Temple (years ago) I found it often took 7+ seconds just to redirect www to non-www, up to 12 seconds. Pingdom is a good tool to see where you're getting stuck http://tools.pingdom.com/fpt/ Another common problem, your MySQL database could be overloaded (especially with shared hosting) and more caching won't necessarily solve the problem.
If it's my hosting (which was an eee pc btw) then why does my self-written blog load in 15ms? But yeah on shared hosting with more cpu power, wordpress performed much better.