Yeah, this is exactly right. To a decent approximation, I spent college and grad school just basically plowing through math and physics texts. In eight or so years, with extensive help from the faculty of two universities, I grew to understand the contents of several dozen thick hard science and math books, and for my efforts they gave me two degrees.

Since quitting school I've dragged myself through a small pile of classic CS texts, and have learned a ton (and, as a side effect, become nicely employable as a programmer). It is insanely hard (and rewarding) work. If it's hard, then you're learning something; keep it up!