Ten Thoughts On FiveThirtyEight
Lamenting the loss of an innovator and hoping for something better
On Wednesday, ABC announced it was shutting down FiveThirtyEight after a seventeen year run, and laying off the entire FiveThirtyEight team.
FiveThirtyEight made the coverage of polling — and politics in general — more quantitative and (mostly) more rigorous, while also having some less positive effects.
I happen to have a lot of overlap with the domains of FiveThirtyEight, as a data scientist, the founder of a sports analytics company and a public opinion polling company, and someone with a longtime involvement with journalism. So I have a lot of thoughts. Here goes:
FiveThirtyEight’s poll aggregation made us significantly more informed. Nate Silver’s early system aggregated polls in relatively simple but clever ways. Averaging polls and then making adjustments — e.g., in seeing movement in one state in the primaries and incorporating cross-state correlations in projecting to other states — was a big step forward over the narrative-driven, data-free coverage that has long dominated political campaigns.
FiveThirtyEight helped to make journalism a little more data-oriented. While journalism still has a long ways to go in being as rigorous and data-driven as it could be, there is a lot more creative, rigorous use of data than there was seventeen years ago. FiveThirtyEight helped to chart a path forward.
FiveThirtyEight got too much credit for being “right” and too much blame for being “wrong.” In 2008, the site was widely credited as being correct in forecasting the winner in 49 of 50 states; in 2012, it received even more plaudits for “predicting” the winner in all fifty states. In reality, that praise underscored a misunderstanding of probabilistic forecasting, and too much praise for being relatively fortunate. Leading up to the 2016 election, FiveThirtyEight showed Hillary Clinton with a 71% chance to win, leading many Americans to assume a victory for her was a sure thing. However, 29% events happen fairly regularly, even if they didn’t really happen in the 2008 and 2012 elections, and FiveThirtyEight received far too much blame for being “wrong” about the 2016 election.
FiveThirtyEight deserves substantial kudos for making past polls part of a public database. This was a real public service.
FiveThirtyEight was less innovative in sports. While the site’s analysis and features were solid and frequently provided compelling data visualization, FiveThirtyEight was not transformative in sports analytics.
FiveThirtyEight was smart to differentiate pollsters based on quality, but sometimes got out over their skis. The site rightly treated different sorts of polls in different ways. However, some of their methodological and product choices — assigning highly precise-looking ratings based on a handful of polls, not properly accounting for the huge shifts which can occur in primary elections, and placing too much weight on decades-old data when polling methodologies were quite different — were questionable.
FiveThirtyEight’s business model didn’t seem well thought through. I believe that, through several incarnations (as an independent site, as part of the NY Times, as part of ABC/ESPN/Disney) it was always free to use. This seems like an iffy decision for a product that has a moderate number of very heavy users, who likely have a high willingness to pay. The rise of Substack — where Nate Silver has a large number of followers — shows one path forward. In the domain of sports, Team Rankings shows another path forward. Both models make some writing and content free, while other content (and in the case of Team Rankings, tools) sit behind a paywall. It’s surprising that Disney never tried this approach.
A slightly more hesitant assertion: FiveThirtyEight was really always about Nate Silver. I’m hesitant on this because I don’t have data on the site’s following and usage after Silver left. I assert that with no judgment in either direction. Clearly there are some media sites that are really about one person, and others that are not; there are pros and cons in either direction.
I’ll conclude with two thoughts of where FiveThirtyEight guided the world, and where I hope it will go and grow in the years to come.
There is a path for a wide range of meaningful reporting to have the same quantitative rigor as FiveThirtyEight. It’s nice to have good data and statistical models of who will become president; it would be even nicer to have good data and statistical models on the day to day things — healthcare and government effectiveness and educational attainment — that have an even bigger effect on most of our lives. The level of data-centered coverage of these topics has increased since FiveThirtyEight launched in 2008, and there is still a long way to go.
FiveThirtyEight made political and campaign reporting more quantitative and rigorous, but it didn’t placate Americans’ unhealthy obsession with the day to day of political campaigns. In 2004, we scoured the news to read about the latest gaffes by Kerry and Bush or the latest campaign ads; in 2024, we impatiently refreshed 538 to see if Kamala Harris’ odds had gone up or down by a percent. Neither of these behaviors are healthy or productive. I don’t blame FiveThirtyEight for this, but I hope to see it improve.
538 will be remembered for bringing a false precision to the imprecise science of polling. They went to great lengths to make their aggregations seem like a scoreboard, even though that wasn't the case.
Treating politics like a sport has made it significantly worse, and that's a lot of their legacy IMO.