User space drivers are possible, the problem is efficiency nosedives. The premise of microkernels[1] is that the kernel itself is "trivially" small and everything else runs as a userspace task.

[1] http://en.wikipedia.org/wiki/Microkernel