FWIW, I learned Vim as a QWERTY touch-typist, then learned Dvorak, and it works just as well with Vim. I never changed any of the keybindings (and hjkl end up in reasonable places anyhow).

Couldn’t say if Colemak would work similarly well, but if your mental mapping for Vim commands is character-based (as opposed to position-on-keyboard-based) I imagine it would.

Worst case I suppose you could always revert to QWERTY.