The difficulty can only adjust by a factor of 4 which also limits the incentive change. You'd need more than 90% of miners to disappear to start seeing actual problems.