I can imagine this being a real issue if it happens at the toolkit level; for example, GTK dropping X11 support. But until Wayland proves itself in the field as a real and superior competitor to X, it's absurd to suggest that even a significant proportion of distros would abandon X altogether. And as long as any distro of note (Linux or otherwise) still ships with X, it seems really unlikely that the major toolkits would indulge in such madness.

I wouldn't worry about it for another decade or so.