There "the defining property" if you want to mod out isometries. :) They're nice, because they encode the geometry of the data.
You could very reasonably try things like cosine distance. And I did some experiments, to good results, with sqrt(d(x,y)), to emphasize really close together data points as special. But these don't feel as motivated.
Hm. It might also be interesting to try with the p_ij values from t-SNE, which model the topology of the data. Then you'd really be getting meta. :)
Interesting. IIUC, what you're implying is that defining a metric defines the topology and they're equivalent.
Isn't p_ij in t-SNE also derived from the distances themselves, where p_ij ~ student_t(d_ij, degrees_of_freedom) (I forget how the d.o.f. is actually computed in t-SNE.)
Which leads me to one way this distance based approach might be limited: It models similarities using distances, which are symmetric. If similarities aren't symmetric, then this visualisation could hide some information. For example: The specific entity "BMW car" is more similar to the more general entity "car" than the entity "car" is to "BMW car." It seems this asymmetry could capture things (such as the generality of concepts), not reflected in metric spaces (on first thought).
You could very reasonably try things like cosine distance. And I did some experiments, to good results, with sqrt(d(x,y)), to emphasize really close together data points as special. But these don't feel as motivated.
Hm. It might also be interesting to try with the p_ij values from t-SNE, which model the topology of the data. Then you'd really be getting meta. :)