Lessons from Nate Silver

“If you’re betting on a correlation and don’t know the causality, I’ll bet the other side.”

That was just one of the gems – part one-liner, part deep insight  – delivered by data journalist Nate Silver in a talk at Cal Performances in Berkeley, California on Sunday afternoon, May 4, 2014.

At first glance, it’s odd to think that a data geek would appear – with PowerPoint slides! – on a stage that typically hosts musicians and dancers. But the fact that Silver can fill a 2,500-seat theater on a sunny weekend afternoon demonstrates the booming currency of data in modern life – in business and science, of course, but also in journalism and popular culture.

Silver first demonstrated his analytic chops more than a decade ago when he developed the Player Empirical Comparison and Optimization Test Algorithm (PECOTA) to predict baseball player performance. He then became widely famous in 2008 by applying probabilistic analytical methods to predict the U.S. presidential election. He’s worked for ESPN and the New York Times, and in 2014 started his own data-driven web venture, fivethirtyeight.com.

Most of Silver’s talk was dedicated to stories and examples from his latest book, The Signal and the Noise: Why So Many Predictions Fail – But Some Don’t. He also said he’s shifting his future efforts away from politics – in part because his methods have been reverse-engineered by competitors – and focusing more on popular culture. One big upcoming project: To find the best burritos in the United States, based in part on analyzing data from Yelp reviews. I eagerly await the results.

In a more serious vein, I drew several lessons from Silver’s talk that apply to businesspeople and developers who work with personalized analytics.

  • Analytics tools must be flexible. Data sources will evolve, new data will become available, and the analysis you perform will be refined over time (or be replaced altogether). Your tools need to be able to respond to these changing needs.
  • Multiple data sources are the norm. Silver has consistently beaten established pundits and pollsters by combining multiple, often competing, datasets. Analytic tools have to work well with a variety of data sources, types, and velocities.
  • Scalability is key. The volume of data we need to analyze, and the number of people performing analysis, will only grow over time. So it’s critical to choose a tool that handles growth well.
  • Data and tools belong in users’ hands. Silver was an anonymous blogger when he developed PECOTA; now he’s one of the biggest names in data. His prominence grew out of his unique application of analytic tools on already existing data.

I’m admittedly biased (and Silver constantly warns against bias), but I think BIRT meets those four needs very well.

Have you heard Nate Silver speak, or read his books? What do you think of his ideas and methods? Share your thoughts in the comments.



OpenText is the leader in Enterprise Information Management (EIM). Our EIM products enable businesses to grow faster, lower operational costs, and reduce information governance and security risks by improving business insight, impact and process speed.

Related Articles