> If every single range it’s at at or below the lower bound for one candidate and at or just past the upper bound for the other, systematically, that’s the definition of a flawed model.
Yes, I agree that individual low-level polls for 2 elections in a row have underrepresented Trump's actual support. But, 538's model addresses exactly your point: low-level indicators like state polling can potentially be systematically flawed and biased. By simulating those unreliable low-level indicators through 40k scenarios, the scenario where Biden systematically underperforms biased pre-election polls in key swing states but Biden still wins (as we're seeing) ends up being a very reasonable outcome and the large reason why Biden was favored overall. Trump needed to outperform the polls in 7 states to win, and that just didn't happen.
And the polls weren't off dramatically in every single race. Here are some of 538's last predictions in key states:
AZ: 50.7% Biden / 48.1% Trump
NE-2: 51% Biden / 47.8% Trump
GA: 50.1% Biden / 49.2% Trump
PA: 52% Biden / 47.3% Trump
NC: 50.5% Biden / 48.8% Trump
Each poll has a margin of error, usually +/-2.5% or more. So, 2-3% swings on Election day are completely normal. NC was off by a tiny margin and enough to flip the state to Trump, while the others were also off by tiny margins, but not enough to flip the state.
Sizable polling swings are not uncommon and historically have gone in both directions, but the model was more concerned about just how narrow Trump's path to 270 was, more so than a routine or even historical polling error.
> The betting markets were much more accurate.
Everyone who bet on Trump at a 30/40% chance to win is about to lose their bet. I definitely wouldn't have taken those odds for Trump to run the table in 7 straight must-win swing states all polling as a coin-flip and within the margin of error. In a completely random scenario Trump would have only had a 14% chance of pulling that off.
“The polls are not supposed to be off by that much, which is why we said that the polls messed up, and our forecast failed (in some part) by not correcting for these errors”
You have a series of predictions that happens incredibly rarely (4 year interval). They are within a close range of accuracy. Your model is working.
Then, one year, the model is off by a full std dev or two.
How will it perform 4 years later? How do you correct for it if you don’t even have a high prior on why it was off last time? How many effects do you just guess and try and model for/against and can you find any data to use for it that’s actually reliable.
How many unknown number of forces is your team arbitrarily deciding were acting on it last time vs now based entirely on their personal histories, combined.
Models can be useful no doubt, whether they work 6/10 times or 9/10 - it entirely depends on the use. But if your model is right 8/10 times and the misses were the last 2? It’s time to rattle it and see what’s broken.
The takeaway: until someone makes a model that proves it can do a few elections in a row (not retroactively) who will really care about the model?
I don't think Taleb or other's have put predictions on paper for nearly as long.
For this year's model Nate Silver has been doing a post-mortem on complaints similar to yours on Twitter:
> We have years and years of well-calibrated forecasts. (If anything, underdogs have won a bit less than they're supposed to, although not to a degree that's statistically significant.) We know what we're doing with these probabilities.
> So I'm admittedly kind of prickly about criticism after elections that both our model and our broader reporting handled quite robustly, which only happened because we put so much work into them. Especially given that the topline outcome + 48/50 states were "called" correctly.
> The reason Biden's win probability was ~90% is precisely because he could withstand a fairly large polling error when Clinton couldn't, which is exactly what happened. Indeed, our model assumes polls are fairly error-prone.
> One slightly unrelated point, but the fact that Biden could lose any 2 states from among the GA/WI/AZ/PA group and still win the election makes his win a bit more robust (and a bit less "close") than if he had to sweep those exact states.
> Especially given that those states are in different parts of the country (South/Midwest/West/Northeast). We've seen in the past few elections largely regionally-driven polling errors. Polls badly underestimated Trump in WI, but only modestly in AZ and PA and not at all in GA.
> This [polling error] averages out to Trump +3.3. Probably pretty similar in national polls, where we'll most likely wind up with an error in the 3.5- or 4-point range I think.
> One of the questions I'm more intrigued by is whether polls missed Trump support among lower-propensity Hispanic (and perhaps also Black) voters. These voters can be hard to get on the phone and/or may be screened out by likely voter models.
It seems like less a case of "shy" voters (which you would expect to see in all states - or at least all blue-leaning states) and perhaps more an instance of pollsters missing sizeable minority voting blocks in certain states, which would influence some state polling more than others.
Of course Nate Silver thinks Nate Silver is right... can you find another prominent statician not employed by him that’s defending his model? I linked to Gelman, a legend in the field and he’s critical of himself and trusting polls, he admits he was wrong for this election.
And here’s Nate admitting he’s wrong the day of (if your model moves from 90/10 to 50/50 with one state change, maybe you shouldn’t be predicting 90/10):
> Look up "unbiased estimator" in your statistics textbook. Now relate that concept to the fact that the vast majority of outcomes were on one side of the sample mean.
Yes, I agree that individual low-level polls for 2 elections in a row have underrepresented Trump's actual support. But, 538's model addresses exactly your point: low-level indicators like state polling can potentially be systematically flawed and biased. By simulating those unreliable low-level indicators through 40k scenarios, the scenario where Biden systematically underperforms biased pre-election polls in key swing states but Biden still wins (as we're seeing) ends up being a very reasonable outcome and the large reason why Biden was favored overall. Trump needed to outperform the polls in 7 states to win, and that just didn't happen.
And the polls weren't off dramatically in every single race. Here are some of 538's last predictions in key states: AZ: 50.7% Biden / 48.1% Trump NE-2: 51% Biden / 47.8% Trump GA: 50.1% Biden / 49.2% Trump PA: 52% Biden / 47.3% Trump NC: 50.5% Biden / 48.8% Trump
Each poll has a margin of error, usually +/-2.5% or more. So, 2-3% swings on Election day are completely normal. NC was off by a tiny margin and enough to flip the state to Trump, while the others were also off by tiny margins, but not enough to flip the state.
And here's how much polling has traditionally been off: https://en.wikipedia.org/wiki/Historical_polling_for_United_...
Sizable polling swings are not uncommon and historically have gone in both directions, but the model was more concerned about just how narrow Trump's path to 270 was, more so than a routine or even historical polling error.
> The betting markets were much more accurate.
Everyone who bet on Trump at a 30/40% chance to win is about to lose their bet. I definitely wouldn't have taken those odds for Trump to run the table in 7 straight must-win swing states all polling as a coin-flip and within the margin of error. In a completely random scenario Trump would have only had a 14% chance of pulling that off.