I used VNC a few times to connect to a Mac build box ( I only used Linux professionally ) and ran into a lot of problems with artifacting ( even on a dedicated Gigabit LAN ) or more annoying the remote service would crash and the only way back in was to SSH into the Mac box, kill the remote service, kill all user sessions, and restart.