SXSW Interactive: Nate Silver

With a few friends and basic blog tools, Nate Silver built a website (FiveThirtyEight) that wielded tremendous political power in the 2008 election. By figuring out how to be smart about data, Silver trumped pollsters and accurately predicted the margins of Obama's win and of Congressional races across the country.

In Sunday's keynote at South by Southwest Interactive, Silver talked about his work in analyzing statistical data in politics and for Baseball Prospectus. But interviewer Stephen Baker of Business Week never quite got Silver to talk about how and why his approach has had such great success.

Silver admitted that he had always been a politics fan "in the background", and became frustrated watching the cable networks report poll results from the initial primaries that were not really empirically valid.

"Polls were too much a part of the narrative," in those early days, Silver said. "I thought there was room for some commonsensical expertise. The elections were one of the best experiments possible from a data geeks' perspective. There was a lot of data happening in slow motion."

Prior to his work during the election cycle, Silver gained fame by inventing PECOTA, a system for forecasting the performance of pro baseball players. And while the prediction algorithms for baseball and politics are quite different, there are some similarities that led to Silver's success in both.

"One commonality is having good habits. I believe in being very meticulous about things," Silver said. "A lot of times decisions are made on the margins. Being able to forecast 2 percent better when you're spending $150 million on a baseball team's payroll saves you $3 million."

Silver said he is working on a book about the value and limitations of predictions, looking at such diverse industries as fashion design, hurricane forecasting, and the search for extraterrestrial life. He had a good piece of advice for anyone looking to create a successful prediction.

"Never come out with a forecast that you know is going to be wrong," Silver said. "If it's going to be wrong, keep working on your model."