I usually approach JavaScript frameworks the same way I approach high-level languages: write the app in whatever way is most convenient, then profile and rewrite the performance-critical sections without the framework. I've never had performance problems with JQuery, because I usually just write my mouseMove handlers so that they operate on the raw DOM without it.
(I wrote something to this effect on Proggit once, and John Resig reply was something like, "Of course. JQuery's not intended for performance-critical code - usually you don't want to be using any framework for that. JQuery's meant to make the other 90% of JavaScript development go faster.)
BTW, I won't go near Prototype (after having used it pretty extensively at a past employer) because it messes with Object.prototype. When libraries do that, you have no way of knowing which libraries are compatible, which means that you'll suddenly get bitten by very odd, hard-to-track-down bugs.
I should say that /any/ framework is not intended for performance-critical code. A framework is designed to be an abstraction - abstractions implicitly remove you from what you're trying to do, causing a performance hit of some sort. If performance is your #1 priority then you should be writing everything from scratch.
Now if development time is a greater priority over performance then I think the choice is pretty obvious.
(I wrote something to this effect on Proggit once, and John Resig reply was something like, "Of course. JQuery's not intended for performance-critical code - usually you don't want to be using any framework for that. JQuery's meant to make the other 90% of JavaScript development go faster.)
BTW, I won't go near Prototype (after having used it pretty extensively at a past employer) because it messes with Object.prototype. When libraries do that, you have no way of knowing which libraries are compatible, which means that you'll suddenly get bitten by very odd, hard-to-track-down bugs.