The way he presents the priming results makes me really suspicious that they made statistical errors.
First, the numbers are all given as percentages relative to a control. This makes the measured changes appear larger. If the control percentage was 5%, then a relative increase of 20% is just an absolute increase from 5% to 6%. (Also, the relative increase inherits variation from both the control and the target bin.)
Second, the data is presented as if it were cherry picked. They give the most interesting bins or transitions between similar bins, instead of discussing metrics over all of the bins.
I did a very rough estimate of the variation you'd expect to see, given 217 bins of binomial distributions with n=10^5 and p=0.05 where you report percentages relative to one of the bins. It's on the order of +-5%, despite the large sample sizes. Minor changes in protocol could increase this to +-30% and explain all of their results.
The wild swings when the color or font is changed increase my suspicion that they're seeing noise (because I really don't expect priming to be that strong).
Note that this is based entirely on the very superficial amount of data the presenter included in his slides. It may also be the case that they just omitted statistical details because they expected it to be boring to the audience.
I kept watching for the Standard Error, Im very surprised they didnt show it as that is very important to know how accurate, he said they had millions of data points but considering all the stats Im really surprised they didnt mention it at all.
> because I really don't expect priming to be that strong
Have you looked into it? I never heard of it but I read a lot about suggestion, conversational hypnosis... I'm going to spend my night studying this priming thing but I'm not skeptical about it.
First, the numbers are all given as percentages relative to a control. This makes the measured changes appear larger. If the control percentage was 5%, then a relative increase of 20% is just an absolute increase from 5% to 6%. (Also, the relative increase inherits variation from both the control and the target bin.)
Second, the data is presented as if it were cherry picked. They give the most interesting bins or transitions between similar bins, instead of discussing metrics over all of the bins.
I did a very rough estimate of the variation you'd expect to see, given 217 bins of binomial distributions with n=10^5 and p=0.05 where you report percentages relative to one of the bins. It's on the order of +-5%, despite the large sample sizes. Minor changes in protocol could increase this to +-30% and explain all of their results.
The wild swings when the color or font is changed increase my suspicion that they're seeing noise (because I really don't expect priming to be that strong).
Note that this is based entirely on the very superficial amount of data the presenter included in his slides. It may also be the case that they just omitted statistical details because they expected it to be boring to the audience.