Level 9 is a war of attrition between the player and the dungeon. The dungeon is intentionally designed with a lot of non-critical areas in order to get the player lost, so that he'll run across more sets of enemies, and eventually be ground down to a point of insufficient resources (health, potions, ammo, etc.) to complete the level. In this sense, beating Level 9 is really an exercise in efficiency, rather than the free-roaming exercise of exploration that carried the player through previous dungeons.

There's also a very subtle psychological dimension at play in Level 9. By the time the player gets to Level 9, he is so used to random exploration, bombing of random walls, discovery of secret areas, and so forth, that his natural instincts are to bomb every wall and explore every room. It's precisely these inclinations that will land him in trouble in Level 9. Almost every wall is bombable; almost every room seems interesting; almost every path is interconnected. Accordingly, in order to beat Level 9, the player must "unlearn" a lot of the behaviors that made him successful, throughout the game, up to that point.

Arguably more dead-space later in the game makes sense. You want to make it really hard to get lost early on, but start to make it harder later on.

Also you want a feeling of greater size in the later dungeons, for aesthetic reasons. They're supposed to be more important and powerful as you go on.