The main point as I understood it was to illustrate, more or less, why a 100% increase in conversions to the purchasing page does not equal a 100% increase in conversions from the purchasing page to actual purchase.
Yes, it offered a theory about why this was so. Yet my experience is that a 100% increase in initial conversions typically results in approximately a 100% increase in sales.
My further experience from doing A/B testing for many years is that lots of people are eager to grab any numbers you give them, then run with them and form grand theories that aren't backed up by the actual statistics. Those theories have a remarkably low success rate in explaining the results of the next A/B test you run. (Or, frequently, the current test once we let it run longer.)
They are saying that once you start attracting traffic beyond the early adopters, your additional traffic is now comprised of a different group of people who exhibit fundamentally different behavior in how likely they are to make a purchase even once they've hit the purchasing page.
It was an attractively presented theory. True, I've learned to be cautious of attractively presented theories which aren't actually backed by data. But it was definitely attractively presented.
However no evidence was offered that the people progressing in group A were actually significantly different than the people progressing in group B. And if the test truly was letting crappier traffic through, and the crappiness of that traffic was the primary cause of trends in subsequent behavior, then that traffic has to be REALLY crappy to explain the difference. Occam's razor says that random chance was the cause of the data, or at least a large enough contributor that there is no immediate need to think too hard about other possibilities. And so until better data becomes available, I'm going to suggest that "chance fluctuation" deserves a hearing.
Medicine sometimes calls these zebras. Why? Because it is like someone hearing something with 4 hooves run past an open window and immediately figuring out that it is a zebra. Sure, it _COULD_ be a zebra. _SOMETIMES_ it proves to be a zebra. But the odds are much better that it was a horse.
Guess the garden variety answer before guessing the exotic one.
So, regarding this being a fluke one-off or the assertion that there isn't data to back this up, I have reams of tests that display similar behavior, many with tens of thousands of unique participants. As the other commenter said, the example in the post was icing, not the cake, but I take your point and it's a good message that people are reading my posts who know something about this and maybe I can get a little deeper into advanced topics and results and stuff, so that's good to hear.
If your early-funnel optimizations directly predict the end-funnel conversion, I think that 1) that's awesome 2) the products you are testing have a huge amount of room to grow (also awesome, hope the hiring goes well it's always tricky finding great people) and 3) the phenomenon i was describing doesn't really apply to you and those products yet.
One thing that may not have been crystal clear is that it's one group of people that we are talking about, and examining the segments of that group. Let's put it another way. There are 1,000 people who see a certain offer page on a given day. 1 of those people is a fellow who has read 3 reviews, used competing products, and decided 100% that he is going to buy the product. The page could have a tiny little 10px link to add the product to cart, it could be in the footer and #EEE and this dude would still find it and complete the purchase.
Now, as you make it clearer to people why and how they should buy the product, you are not generating more interest like the first fellow's. You are simply inviting a broader selection of the audience to consider the purchase a bit more. That's really the principle that is at play here.
Yes, it offered a theory about why this was so. Yet my experience is that a 100% increase in initial conversions typically results in approximately a 100% increase in sales.
My further experience from doing A/B testing for many years is that lots of people are eager to grab any numbers you give them, then run with them and form grand theories that aren't backed up by the actual statistics. Those theories have a remarkably low success rate in explaining the results of the next A/B test you run. (Or, frequently, the current test once we let it run longer.)
They are saying that once you start attracting traffic beyond the early adopters, your additional traffic is now comprised of a different group of people who exhibit fundamentally different behavior in how likely they are to make a purchase even once they've hit the purchasing page.
It was an attractively presented theory. True, I've learned to be cautious of attractively presented theories which aren't actually backed by data. But it was definitely attractively presented.
However no evidence was offered that the people progressing in group A were actually significantly different than the people progressing in group B. And if the test truly was letting crappier traffic through, and the crappiness of that traffic was the primary cause of trends in subsequent behavior, then that traffic has to be REALLY crappy to explain the difference. Occam's razor says that random chance was the cause of the data, or at least a large enough contributor that there is no immediate need to think too hard about other possibilities. And so until better data becomes available, I'm going to suggest that "chance fluctuation" deserves a hearing.
Medicine sometimes calls these zebras. Why? Because it is like someone hearing something with 4 hooves run past an open window and immediately figuring out that it is a zebra. Sure, it _COULD_ be a zebra. _SOMETIMES_ it proves to be a zebra. But the odds are much better that it was a horse.
Guess the garden variety answer before guessing the exotic one.