It'd be unfair to great startups in the experiment batch -- not that it's pg's job to make life fair for people, but it'd put a big dent in YC's reputation. Why spend time flying out there if there's a chance your interview doesn't even matter?
You can still do a pretty similar experiment. For instance, startups 1-50 get into yc like normal. Startups 51-100, together with 50 randomly selected startups get grouped together and tagged as "the bubble group". Meaning that they were on the bubble of getting into yc. Pretty high honors. However, they don't get any access, consultation, etc. to yc itself. Then after a few months you unblind which were highly selected and which were random, to see who does better.
Can't you pick the best interviews, then hold your experiment with random groups as long as you mix in a the data from the interview groups in such a way to build a representative sample?
We would only be able to conclude that it's unfair for "great start ups" if the experiment showed that random acceptances performed worse than the interview process.
