You also have less stuff for the browser to consider when parsing the HTML. Depending on the amount of CSS, I'd rather have that and less memory footprint per tab. And its only as long as the browser doesn't have that permutation of CSS "includes" cached already, basically, once per each "page type" if you will, and only when you change from your current page type to a new one.

However, I don't say this from a usability perspective, but strictly from a "this is neat to code" standpoint. If you do it right, you should always be able to fallback to "include all CSS always" at the flick of a switch; so you have to code this first before you can even test if it helps ;)