Ensemble (and also boosted) models: Very nice idea.
I like the takeaway that meta-SNE idea is powerful to compare the space of models by through the lens of pairwise distances as a proxy for the distance metric. Are distances the defining property for a vector space R^d? Could you have used some other quantity instead of pairwise distances?
There "the defining property" if you want to mod out isometries. :) They're nice, because they encode the geometry of the data.
You could very reasonably try things like cosine distance. And I did some experiments, to good results, with sqrt(d(x,y)), to emphasize really close together data points as special. But these don't feel as motivated.
Hm. It might also be interesting to try with the p_ij values from t-SNE, which model the topology of the data. Then you'd really be getting meta. :)
Interesting. IIUC, what you're implying is that defining a metric defines the topology and they're equivalent.
Isn't p_ij in t-SNE also derived from the distances themselves, where p_ij ~ student_t(d_ij, degrees_of_freedom) (I forget how the d.o.f. is actually computed in t-SNE.)
Which leads me to one way this distance based approach might be limited: It models similarities using distances, which are symmetric. If similarities aren't symmetric, then this visualisation could hide some information. For example: The specific entity "BMW car" is more similar to the more general entity "car" than the entity "car" is to "BMW car." It seems this asymmetry could capture things (such as the generality of concepts), not reflected in metric spaces (on first thought).
I like the takeaway that meta-SNE idea is powerful to compare the space of models by through the lens of pairwise distances as a proxy for the distance metric. Are distances the defining property for a vector space R^d? Could you have used some other quantity instead of pairwise distances?