Lies, Damned Lies, and Statistics...

[uh_no]

The fundamental flaw in 538 and kenpom is that they equally weight all possessions throughout the season.

As others have pointed out, this is simply false...at least for kenpom. It's quite well known that recent games are given significantly more weight than far past games, and games in which you blow out an overmatched opponent are given less weight.

That doesn't counter the fact that all posessions within a game are treated the same. Kenpom has in his blog (if i recall) tried to make adjustments for end games, but generally when he doesn't adjust for something like that it's for one of 3 reasons

1) it doesn't significantly affect the ratings
2) he can't find a good way to determine what is or isn't a "garbage" posessions
3) adding in the factor decreased the overall predictive power of the model

[OZ]

I am grateful to the sports gods, that with all the "insiders," analyses, prognosticators, stats, computers, percentages and or "premium" sights, no one can guarantee a winner. Last year Connecticut, this year Duke. Each time, I drive the 200 miles round trip to Cameron or follow them elsewhere, there is always -NO MATTER whom we play-that bit of fear (I was actually at the Wagner game) accompanied by a greater since of hope.
Before we played the first game of the tournament, there were those who filled the boards with their dire warnings of North Florida... I got that!!! Though, we were expected to win easily against Robert Morris - I was still nervous. When we played Wisconsin, most of the basketball "experts" that I heard didn't give Duke much of a chance. In fact, there were quit a few, who didn't think it would be a close game. Yes, perhaps, "statistically, Duke's championship was very unlikely;" but thankfully, stats can't predict a player's heart... or a coach's decision in critical times...nor can they foretell a lesser known having his "shinning moment" at just the right time.
And that is why I honestly try to avoid any and all analyses and predictions of games. They would not affect me any way; because I always enter the stadium the same way - a bundle of nerves with a fearful respect of the other team and a healthy dose of optimism for Duke. With all due respect, while they might be fun, amusing, interesting or informative, the moment the likes of "538 blog" is able predict Duke's basketball games is the moment I turn in my tickets and stay home.

[Kedsy]

Of course, Kedsy, to compare teams across conferences the ONLY data are games from Nov. and Dec. -- therefore, they HAVE to have a high weight.

You can view CONFERENCE data as tightly bounds balls of string with 150-200 data points (individual games) that provides gobs of data for the analyst to evaluate teams WITHIN the same conference. But these balls of data are all floating in the ether, and the only things that put them in a relative framework are the non-conference games -- there weren't vary many significant games and they were early in the season.

Yes, of course I know your viewpoint on this. And there's a lot of validity to it. I still believe, however, that it's a lot more complicated than you make it seem.

We have no way of knowing for sure that Kentucky was a better team than Wisconsin, except when they played one another in the Final Four.

Well, first of all, that's not true. We have ways of knowing who the best team is. You just don't believe those ways. Tell me point blank, do you think Miami was a better team than Duke this year? Head-to-head is a good data point, but it doesn't tell us who the better team is. Ever.

Second, I was at the Wisconsin/Kentucky game. I was rooting for Wisconsin, I thought they were a great team. But I thought Kentucky was the better team. I thought if they played an extended series that Kentucky would win more than half the games. So at least in my case, the "eye test" coincided with the computer ratings.

Strength of schedule statistics failed to consider the weakness of the SEC teams.

Actually, I'm pretty sure the strength of schedule statistics totally considered the weakness of the SEC teams. Do you think the SOS numbers in the various rating systems were only non-conference?

The bookies predicted Duke's margin of victory at 2.5. As it happened, Duke won by 5, and a good reason for that is that two of the smallest guys on the team got an emotional boost and saw to it that our team won. Why didn't the statisticians include that possibility in their calculations? They didn't have enough to go on since it was a single elimination tourney, perhaps. In the end emotions trump statistics, and predictions are a crap shoot.

My understanding was Wisconsin was favored by 1. Duke ended up winning what was basically a toss-up game, but I don't understand your point. Are you saying that because an underdog can win a game then there's no point to figuring out who's favored? Because that doesn't make much sense to me. Are you saying statistics don't tell us anything? Are you suggesting that every team has a 50/50 chance against every other team? Because that's clearly false. I honestly have no idea what you're trying to say.

[darthur]

What makes a good predictive model? Is it acceptable to include the winner in the top 50% most likely half the time? Does it need to have the winner in the top two most of the time...There's an awful lot of room in between to gauge success.

Has anyone compared which of Nate Silver's and Pomeroy's model was "better"? Each one assigns a probability of the tournament going the way it did. Which one assigned it the higher probability? Or even which one favored the winners most often?

It always makes me sad that we have so many ranking systems but nobody publishes data on how good they actually are. Seems like it would be straightforward to measure kenpom vs sagarin vs seeds vs AP rankings vs RPI vs BPI vs whatever year in and year out, but nobody does it. And sadly, historical records of these rankings are often not available.

[Kedsy]

Has anyone compared which of Nate Silver's and Pomeroy's model was "better"? Each one assigns a probability of the tournament going the way it did. Which one assigned it the higher probability? Or even which one favored the winners most often?

It always makes me sad that we have so many ranking systems but nobody publishes data on how good they actually are. Seems like it would be straightforward to measure kenpom vs sagarin vs seeds vs AP rankings vs RPI vs BPI vs whatever year in and year out, but nobody does it. And sadly, historical records of these rankings are often not available.

Don't know if this gives you what you want, but here's a composite of a lot of rating systems dating back to 2002. Anyway, if you want to compare to actual results, there's a fair number of data points here.

[YmoBeThere]

...is the moment I turn in my tickets and stay home.

I'll take them.

[bjornolf]

If we had such an easy road to the FF, then how come everybody and their brother picked us not to survive our region, saying Utah would beat us, and if not, Gonzaga would?

[Mtn.Devil.91.92.01.10.15]

It is amazing to me how many people misunderstand statistics. Silver saying Duke had a six percent chance of winning before the tournament is not necessarily wrong. It works much like predicting the weather - the only way predictive statistics can "be wrong" is if you use absolutes.

If Silver said Duke had a zero percent chance of winning, we could say without question he was incorrect. If he said Duke had a six percent chance and we won, he is not necessarily wrong. If he said we had a six percent chance and we won ten times in a row, either he is mistaken in his calculations or we are wildly fortunate.

I still don't understand why weather forecasters ever say there is either a zero percent or one hundred percent chance of rain. It is the only way to be proven wrong.

[CDu]

It is amazing to me how many people misunderstand statistics. Silver saying Duke had a six percent chance of winning before the tournament is not necessarily wrong. It works much like predicting the weather - the only way predictive statistics can "be wrong" is if you use absolutes.

If Silver said Duke had a zero percent chance of winning, we could say without question he was incorrect. If he said Duke had a six percent chance and we won, he is not necessarily wrong. If he said we had a six percent chance and we won ten times in a row, either he is mistaken in his calculations or we are wildly fortunate.

I still don't understand why weather forecasters ever say there is either a zero percent or one hundred percent chance of rain. It is the only way to be proven wrong.

Exactly. There is just no way to know if the model was right in its estimated probabilities, because we only see one outcome and not the 1,000 (or 10,000, or whatever number of simulations were run) that the models are based on. Even if the model said Duke had a 0.000000001% chance of winning, it's possible (though roughly 0.000000001% likely, ignoring any of the other games of course) that the model could be correctly and Duke could win. It's very possible that the true probability of Duke winning this year was around 6%, and that we beat the odds. No way to know, simply because we only get to observe the one observation.

[wilson]

Lots of nice statistical analysis and explanation here from people who know a lot more about the topic than I do...thanks to all for your insights.
Could it also be that statistical predictions, though they are often highly astute and have some strong predictive abilities, are losing their ability to confidently predict college basketball results? The rise of statistical sports analysis has been a pretty big story in the last decade or so, but so too has been the supposed decay of college hoops, owing to the high rate of turnover among top-level talent and the lack of long-term chemistry and development. As that trend continues, could the constant flux be introducing a higher level of variance into the various statistical models? Any given college basketball roster among serious contenders is almost always going to have a low number of data points, because it's so difficult to keep an entire team intact for any length of time. I think perhaps that is making it more difficult to predict the game, especially in a volatile, traditionally unpredictable environment like the NCAA Tournament.

[Bostondevil]

"The one in a million occurrence happens with no more and no less than the expected frequency, no matter how surprised we might be when it happens to us." R.A. Fisher (the inventor of regression analysis)

538 gave Duke a 6% chance of winning at the BEGINNING of the tournament. I'd say that was about right. Granted before the tournment, Kentucky had the best odds at the beginning, but so what? The odds would change after every round.

[tbyers11]

Has anyone compared which of Nate Silver's and Pomeroy's model was "better"? Each one assigns a probability of the tournament going the way it did. Which one assigned it the higher probability? Or even which one favored the winners most often?

It always makes me sad that we have so many ranking systems but nobody publishes data on how good they actually are. Seems like it would be straightforward to measure kenpom vs sagarin vs seeds vs AP rankings vs RPI vs BPI vs whatever year in and year out, but nobody does it. And sadly, historical records of these rankings are often not available.

Ask an ye shall receive. Sort of. 538 looks at how their forecasts did.

1) Comparisons relative to Vegas in the first rounds and later rounds and 2) against other models throughout the course of the tourney.
Interesting data but not any real clear cut answers. 538 did quite well in the first round but not as well later on.

[roywhite]

"The one in a million occurrence happens with no more and no less than the expected frequency, no matter how surprised we might be when it happens to us." R.A. Fisher (the inventor of regression analysis)

538 gave Duke a 6% chance of winning at the BEGINNING of the tournament. I'd say that was about right. Granted before the tournment, Kentucky had the best odds at the beginning, but so what? The odds would change after every round.

At the start of the tournament, Duke came in:
ranked #4 in the country
1 loss since January
road wins @Wisconsin, Louisville (full roster for the Cards), Virginia (full roster), Syracuse, and UNC.
with no major injuries

So, a 6% chance of winning? Not in my opinion, given those facts

To each his own, but I don't see much predictive value from 538 here, and even kenpom has it's limitations.

[alteran]

Nate Silver's predictions don't factor in Duke's decided officiating advantage...

Excellent point. You'd think a genius statistician like Nate would have caught that based on numbers analysis of previous games. I wonder how it missed him.

[ns7]

While 1999 Duke was -- I believe -- the best team that didn't win the tournament in the last quarter century (better than 1991 UNLV and 2015 Kentucky), I think you are incorrect to describe them as the "overwhelming favorite."

Yes, Duke finish the 1999 season No. 1 after a dominant run, but we forget that they were No. 1 for just eight weeks during the season, while UConn was No. 1 for 10 weeks.

Duke was 36-1 going into the title game, but UConn was 33-2 -- and both losses had come when center Jake Voskuhl was out with a sprained ankle.

Just trying to explain that Duke was favored that night in St. Pete, but the Devils weren't an overwhelming favorite -- not nearly as big a favorite as UNLV in '91 or Kentucky vs. Wisconsin in '15.

PS And I would argue that Duke's dominance of the ACC that season was a little like Kentucky's dominance of the SEC this season. The ACC in 1999 was unusually down for the ACC. Just three teams made the NCAA Tournament. Maryland was very good (better than anyone in the SEC other than UK this year) -- 28-6 and a Sweet 16 team. UNC was better than any non-UK SEC team this year -- 24-10 and No. 13 in the nation, but that was the team that lost to Weber State in the first round. The only other ACC team to win 20 was Clemson and they were 5-11 in the ACC. The fourth-place team in the league standings was Wake at 7-9 -- and they had a losing record overall.

Edit: looks like this has been answered already...

We were a 9.5 favorite over UConn in that title game. I can't remember the exact pre-tournament odds, but remember it being better than even money.

I believe UK was a ~5-6 point favorite over Wisconsin. Not sure there was a line for the UNLV game because of the special circumstances.

I agree that the ACC was down, but that 1999 team absolutely destroyed everyone. Compare that to UK this year and their multiple close wins.

[ns7]

At the start of the tournament, Duke came in:
ranked #4 in the country
1 loss since January
road wins @Wisconsin, Louisville (full roster for the Cards), Virginia (full roster), Syracuse, and UNC.
with no major injuries

So, a 6% chance of winning? Not in my opinion, given those facts

To each his own, but I don't see much predictive value from 538 here, and even kenpom has it's limitations.

That prediction was based on Duke playing mediocre defense and excellent offense. Most people didn't think that Duke's defense would suddenly become better than Kentucky's defense. But it did and I am still super thrilled that it happened.

But I don't believe you can fault any person or model who did not predict that.

[Richard Berg]

Statistically, Duke's championship was very unlikely - 538 blog

So?

Statistically, the chance the tourney would play out the way it did was unfathomably small. Millions of people filled out brackets, yet not a single one predicted the outcome exactly.

And yet, it happened. In fact, the chance of all of the above statements being true was extremely high. Doesn't make them meaningful.

[Lar77]

We were a 9.5 favorite over UConn in that title game. I can't remember the exact pre-tournament odds, but remember it being better than even money.

I believe UK was a ~5-6 point favorite over Wisconsin. Not sure there was a line for the UNLV game because of the special circumstances.

I agree that the ACC was down, but that 1999 team absolutely destroyed everyone. Compare that to UK this year and their multiple close wins.

I have to agree with OF. We were favored but my recollection was that it was not a shock that they beat us.

I've enjoyed the back and forth on the "validity" of statistical analyses. Kedsy has done a good job defending it and admits that we don't have all the answers because they are proprietary models. And I am sure that Pomeroy et al will continue to refine their methodology in the pursuit of perfection. The problem is see is how ESPN, CBS, and other media seem to assume it is already perfection. And that's what gets us all wound up: "You said Duke only had a 6% chance of winning. Can't you see we're better than that!!!!" I enjoy Men's CBB because of the variables and the unpredictability that no model can pick up. As in the ACC ND game, we were not the team we had been or would be during the first 24 minutes. Did Wisconsin really expect Grayson Allen to enter the pantheon of Duke legends when he didn't play against them at the Kohl Center and was the 8th man on an 8 man rotation? But he did.

Without hijacking the thread, what is the probability of UNC sanctions coming before next season?

[Kedsy]

Could it also be that statistical predictions, though they are often highly astute and have some strong predictive abilities, are losing their ability to confidently predict college basketball results?

The answer is either no or they never had such ability. Upsets happen in every sport and college basketball is certainly not immune. The NCAA tournament, with its one-and-done format and intertwined bracket where one upset can affect so many other matchups, is by design especially prone to unpredictable results.

In any event, this is not a new phenomenon. The very first tournament with real seeding (1979) featured a 9-seed (Penn) in the Final Four, and the next year featured a 5-seed (Iowa), a 6-seed (Purdue), and an 8-seed (UCLA) in the Final Four (8-seed UCLA made the title game). In fact, the first ten seasons of seeded NCAA tournaments contained nine years with 6-seeds or worse in the Final Four (including an 11-seed (LSU) in the Final Four in 1986), and four national champions seeded #3 or worse: Indiana (3-seed in 1981), NC State (6-seed in 1983), Villanova (8-seed in 1985), and Kansas (6-seed in 1988).

The first ten year detail:

1979: 9-seed Penn in Final Four;
1980: 8-seed UCLA in title game (as well as 5-seed Iowa and 6-seed Purdue in Final Four);
1981: 3-seed Indiana winning championship;
1982: 6-seed Houston in Final Four;
1983: 6-seed NCSU winning championship (and 4-seed Georgia in Final Four);
1984: 7-seed Virginia in Final Four;
1985: 8-seed Villanova winning championship;
1986: 11-seed LSU in Final Four;
1987: 6-seed Providence in Final Four;
1988: 6-seed Kansas winning championship.

The next ten years weren't as crazy, but still had three very unlikely champions (Michigan in 1989, Duke in 1991, and Arizona in 1997), as well as:

1989: Title game between two 3-seeds (Michigan and Seton Hall);
1990: Three teams seeded #3 or worse in the Final Four (#3 Duke, #3 Arkansas, and #4 Georgia Tech);
1991: Duke over UNLV
1992: 6-seed Michigan in title game (and 4-seed Cincinnati in Final Four);
1996: 4-seed Syracuse in title game (and 5-seed Mississippi State in Final Four);
1997: 4-seed Arizona winning championship;
1998: Two 3-seeds (Utah and Stanford) in Final Four.

In the 16 years of the new century, we've seen four champions seeded #3 or worse and 19 Final Four teams seeded #4 or worse, including:

2000: 5-seed Florida in the title game and 8-seed UNC and 8-seed Wisconsin in Final Four;
2002: 5-seed Indiana in the title game;
2003: 3-seed Syracuse winning championship;
2005: 5-seed Michigan State in Final Four;
2006: 3-seed Florida winning championship and 11-seed George Mason in Final Four (as well as 4-seed LSU);
2010: 5-seed Butler in title game (and 5-seed Michigan State in Final Four);
2011: 3-seed UConn winning a Final Four that included 4-seed UK, 8-seed Butler, and 11-seed VCU;
2012: 4-seed Louisville in Final Four;
2013: 9-seed Wichita plus two 4-seeds (Syracuse and Michigan) in Final Four;
2014: 7-seed UConn beating 8-seed Kentucky in title game;
2015: 7-seed Michigan State and "unlikely" champion Duke.

So basically the NCAA tournament has been a crapshoot at least since they started seeding and probably a lot longer than that. Anyone who expects a rating system to accurately predict the results of the tournament is generally going to be disappointed.

[Mtn.Devil.91.92.01.10.15]

So?

Statistically, the chance the tourney would play out the way it did was unfathomably small. Millions of people filled out brackets, yet not a single one predicted the outcome exactly.

And yet, it happened. In fact, the chance of all of the above statements being true was extremely high. Doesn't make them meaningful.

I suppose at the end of the day, I agree with you here.