PDA

View Full Version : Lies, Damned Lies, and Statistics...



Pages : [1] 2

Duke95
04-16-2015, 12:41 PM
So, I revisited the 538 blog to see how the predictions worked out.

http://fivethirtyeight.com/interactives/march-madness-predictions-2015/#mens

These are their estimated probabilities of winning it all.

Kentucky - 41%
Villanova - 11%
Wisconsin - 10%
Arizona - 9%
Virginia - 8%
Duke 6%

Duke had the 6th highest probability of winning, lowest among 1 seeds and lower than two 2 seeds. Nobody else had higher than 3%.
So, basically, UK had almost 7 times the probability of winning it all than Duke. Duke was only given a 32% chance of getting out of their bracket, again, lowest among the 1 seeds. (In fact, Arizona was favored to get to the FF over Wisconsin.)

I guess there are two points:
1. Some opposing fans claim we had an easy bracket. However, we weren't given as high a chance to make it out as the other 1 seeds.
2. Duke came together at the right time, and some of the adjustments at the NCAAs should be credited to the coaching staff and the players, who seemed to step it up even another notch compared to previous games.

Pretty amazing year.

FerryFor50
04-16-2015, 12:43 PM
So, I revisited the 538 blog to see how the predictions worked out.

http://fivethirtyeight.com/interactives/march-madness-predictions-2015/#mens

These are their estimated probabilities of winning it all.

Kentucky - 41%
Villanova - 11%
Wisconsin - 10%
Arizona - 9%
Virginia - 8%
Duke 6%

Duke had the 6th highest probability of winning, lowest among 1 seeds and lower than two 2 seeds. Nobody else had higher than 3%.
So, basically, UK had almost 7 times the probability of winning it all than Duke. Duke was only given a 32% chance of getting out of their bracket, again, lowest among the 1 seeds. (In fact, Arizona was favored to get to the FF over Wisconsin.)

I guess there are two points:
1. Some opposing fans claim we had an easy bracket. However, we weren't given as high a chance to make it out as the other 1 seeds.
2. Duke came together at the right time, and some of the adjustments at the NCAAs should be credited to the coaching staff and the players, who seemed to step it up even another notch compared to previous games.

Pretty amazing year.

Nate Silver's predictions don't factor in Duke's decided officiating advantage... :rolleyes:

Duke95
04-16-2015, 12:47 PM
Nate Silver's predictions don't factor in Duke's decided officiating advantage... :rolleyes:

True. We didn't send out payment in on time last year, and look what happened in the first round. We learned this year. ;)

mo.st.dukie
04-16-2015, 01:02 PM
The only objective criteria used was player injuries. Everything else is subjective even geography because it can't take into account how well a fanbase will travel or how large of an alumni base a school may have in a particular area even if another school is physically closer.

Pretty good example how faulty these kinds of things are and how the results are only as good as the inputs and methodology used.

Olympic Fan
04-16-2015, 01:09 PM
The 538 projection was not far out of line with Ken Pomeroy's pre-tourney odds:

http://kenpom.com/blog/index.php/weblog/entry/ncaa_tournament_log5 (scroll; down a bit)

He had:
1. Kentucky 33.8
2. Arizona 13.9
3. Wisconsin 10.5
4. Villanova 10.0
5. Virginia 9.6
6. Gonzaga 5.4
7. Duke 4.5
8. Utah 2.0

flyingdutchdevil
04-16-2015, 01:26 PM
I see this info and come to two conclusions: Duke's % of winning was too low, and the beauty of the NCAA Tourney is the single elimination aspect of it.

I firmly believe that Kentucky was the best team all season, from beginning to end. They lost when it mattered the most, and hence they will not go down in history (yay!). Duke just won every game, and looked very impressive doing it. We won six games in a row when it mattered. That is what counts, best team in the country or not.

Many on this board believe that Duke was the best team in 1999, 2006, and 2011. I do not disagree. But we didn't win the whole thing. Does that mean we weren't the best team? In single elimination, I believe that the best team doesn't always win.

Kedsy
04-16-2015, 01:29 PM
Pretty good example how faulty these kinds of things are and how the results are only as good as the inputs and methodology used.

Why does the fact that a team with a fairly low probability of winning the tournament actually won it make the computer systems that determine the probabilities "faulty"? In three of the past five seasons, the team that won the tournament wasn't among the five most likely teams to win. And you think that means the rating systems are wrong? To me, it just means that the most likely team very often doesn't make it through a tough, one-and-done tournament. Frankly, the best team from the regular season rarely wins the NCAA tournament.

Also, the computer probability of winning it all depends very strongly on how difficult a path you have. Duke's percentage was low in large part because we had a 5-seed in our path (Utah) that the computers profiled as more of a 3-seed or even a 2-seed. Pomeroy, for example, ranked Utah as the #8 team in the country, and Gonzaga #6. Having to go through two top 10 teams is something none of the other #1 or #2 seeds had to achieve to get to the Final Four, hence Duke's chance of getting that far was lower than the other contenders. That makes perfect sense to me, not faulty at all.

mr. synellinden
04-16-2015, 01:39 PM
Why does the fact that a team with a fairly low probability of winning the tournament actually won it make the computer systems that determine the probabilities "faulty"? In three of the past five seasons, the team that won the tournament wasn't among the five most likely teams to win. And you think that means the rating systems are wrong? To me, it just means that the most likely team very often doesn't make it through a tough, one-and-done tournament. Frankly, the best team from the regular season rarely wins the NCAA tournament.

Also, the computer probability of winning it all depends very strongly on how difficult a path you have. Duke's percentage was low in large part because we had a 5-seed in our path (Utah) that the computers profiled as more of a 3-seed or even a 2-seed. Pomeroy, for example, ranked Utah as the #8 team in the country, and Gonzaga #6. Having to go through two top 10 teams is something none of the other #1 or #2 seeds had to achieve to get to the Final Four, hence Duke's chance of getting that far was lower than the other contenders. That makes perfect sense to me, not faulty at all.

Also, our ranking and hence our percentage chance of winning those 6 games was based on our season long performance in terms of offensive and defensive efficiency - and it was very obvious that Duke was playing at a level defensively that it had not been able to achieve overall and certainly not with any consistency during the regular season. I recall seeing that our defensive efficiency during the tournament was the best in the country - better than Virginia's, Arizona's, Kentucky's, or SDSU's. I think it may have even been better than those team's respective efficiency during the whole season. What would be interesting to know is what the computers would have calculated our chance of winning the tournament based on our end of season efficiency numbers OR the efficiency numbers based on just the six games during the tournament. We were a different team once Winslow and Okafor returned to near full-strength and we changed our starting lineup by replacing Jefferson with Jones.

Olympic Fan
04-16-2015, 01:46 PM
One thing I give both 538 and Pomoroy credit for -- both had Kentucky at less than 50 percent offs to win the tourney. In that sense, both projections were right ...

That's in sharp contrast to the so-called expert commentators who suggested that we should take Kentucky against the field.

I wonder if in the future, people will remember just HOW overwhelming a favorite Kentucky was going into the tournament?

ns7
04-16-2015, 02:45 PM
One thing I give both 538 and Pomoroy credit for -- both had Kentucky at less than 50 percent offs to win the tourney. In that sense, both projections were right ...

That's in sharp contrast to the so-called expert commentators who suggested that we should take Kentucky against the field.

I wonder if in the future, people will remember just HOW overwhelming a favorite Kentucky was going into the tournament?

Kentucky was obviously very good, but was simultaneously overrated. They had quite a few close calls against mediocre teams: Ole Miss, A&M, Georgia, Florida. If just one of those games had flipped, UK would have been viewed like Florida last year or Louisville in 2013: a great team that could be beaten.

The last team that was actually an overwhelming favorite was 1999 Duke. That team tore through the competition but unfortunately lost the only two one-possession games it played all year: Cincy and UConn. I believe 538's SRS system rates that team as the best ever despite losing the championship game.

Kfanarmy
04-16-2015, 02:50 PM
Why does the fact that a team with a fairly low probability of winning the tournament actually won it make the computer systems that determine the probabilities "faulty"? In three of the past five seasons, the team that won the tournament wasn't among the five most likely teams to win. And you think that means the rating systems are wrong? To me, it just means that the most likely team very often doesn't make it through a tough, one-and-done tournament. Frankly, the best team from the regular season rarely wins the NCAA tournament.

Also, the computer probability of winning it all depends very strongly on how difficult a path you have. Duke's percentage was low in large part because we had a 5-seed in our path (Utah) that the computers profiled as more of a 3-seed or even a 2-seed. Pomeroy, for example, ranked Utah as the #8 team in the country, and Gonzaga #6. Having to go through two top 10 teams is something none of the other #1 or #2 seeds had to achieve to get to the Final Four, hence Duke's chance of getting that far was lower than the other contenders. That makes perfect sense to me, not faulty at all.

What makes a good predictive model? Is it acceptable to include the winner in the top 50% most likely half the time? Does it need to have the winner in the top two most of the time...There's an awful lot of room in between to gauge success.

So maybe having the winner in your top five most of the time is a reasonable standard. I would argue that, if 60% of the time any predictive model fails to include the winner in its "5 most likely," the model is faulty. Now whether a tweak is needed or an overhaul is needed I wouldn't argue, but seems to me a "good" predictive model would include the winner in its top 5 more often than not.

I don't think rating systems necessarily must be wrong for a predictive model that uses them to be so....

Any predictive model that isn't 100% accurate is somewhat faulty right? that doesn't mean useless, but lady luck will always play a bit part. What is acceptable?

Kfanarmy
04-16-2015, 02:54 PM
I see this info and come to two conclusions: Duke's % of winning was too low, and the beauty of the NCAA Tourney is the single elimination aspect of it.

I firmly believe that Kentucky was the best team all season, from beginning to end. They lost when it mattered the most, and hence they will not go down in history (yay!). Duke just won every game, and looked very impressive doing it. We won six games in a row when it mattered. That is what counts, best team in the country or not.

Many on this board believe that Duke was the best team in 1999, 2006, and 2011. I do not disagree. But we didn't win the whole thing. Does that mean we weren't the best team? In single elimination, I believe that the best team doesn't always win.

I always thought UK was a very good team, but I never believed they were the best...for as tall and athletic as they were, their offense at times was underwhelming. I think everyone in the top 10 had a decent shot at beating them. I think 2015 Duke and Wisconsin beat them more often than not.

duke09hms
04-16-2015, 02:54 PM
One thing I give both 538 and Pomoroy credit for -- both had Kentucky at less than 50 percent offs to win the tourney. In that sense, both projections were right ...

That's in sharp contrast to the so-called expert commentators who suggested that we should take Kentucky against the field.

I wonder if in the future, people will remember just HOW overwhelming a favorite Kentucky was going into the tournament?

You mean I shouldn't start freaking out about UNC winning the title next year??

Duke95
04-16-2015, 02:55 PM
Also, the computer probability of winning it all depends very strongly on how difficult a path you have. Duke's percentage was low in large part because we had a 5-seed in our path (Utah) that the computers profiled as more of a 3-seed or even a 2-seed. Pomeroy, for example, ranked Utah as the #8 team in the country, and Gonzaga #6. Having to go through two top 10 teams is something none of the other #1 or #2 seeds had to achieve to get to the Final Four, hence Duke's chance of getting that far was lower than the other contenders. That makes perfect sense to me, not faulty at all.

That fact seems lost on those opposing fans (especially the toilet blue variety) who keep saying that Duke was granted a free ride to the Final Four. Utah was a very tough out, as evidenced by our relatively narrow margin of victory.

InSpades
04-16-2015, 03:02 PM
I would think that putting any metric on "how often the winner comes from your top 5" is a poor metric. This is very dependent on how dominant the teams at the top are. In women's college basketball if you the winner didn't come from your top 5 like 80% of the time I would say your model is probably garbage. The sport is just extremely top heavy. If say you tried to predict the winner of the World Series of Poker main event... if you had the winner came from your top 5 say 10% of the time I would say your model is amazing. I'm guessing we could come up w/ some statistical analysis to say if the projections for the NCAA tournament by KenPom are good or bad (but that's beyond my talents!).

I will say that for me the #s for this year seemed terribly off. I never thought Kentucky was anywhere close to 50/50 to win. There just isn't that much difference in the top teams. Before the final four I would have bet a lot of money on Duke or Wisconsin to win it all. You could have gotten like 9 to 5 or something on that bet. MSU had virtually no chance and I think Kentucky's chances were way too high. Turns out in this instance I was correct... but I clearly could have just been lucky. Maybe if we run that final 4 a 1000 times then maybe Kentucky does in 45% or so. Maybe Duke wins 45%. That certainly wouldn't shock anyone here.

Olympic Fan
04-16-2015, 03:47 PM
The last team that was actually an overwhelming favorite was 1999 Duke. That team tore through the competition but unfortunately lost the only two one-possession games it played all year: Cincy and UConn. I believe 538's SRS system rates that team as the best ever despite losing the championship game.

While 1999 Duke was -- I believe -- the best team that didn't win the tournament in the last quarter century (better than 1991 UNLV and 2015 Kentucky), I think you are incorrect to describe them as the "overwhelming favorite."

Yes, Duke finish the 1999 season No. 1 after a dominant run, but we forget that they were No. 1 for just eight weeks during the season, while UConn was No. 1 for 10 weeks.

Duke was 36-1 going into the title game, but UConn was 33-2 -- and both losses had come when center Jake Voskuhl was out with a sprained ankle.

Just trying to explain that Duke was favored that night in St. Pete, but the Devils weren't an overwhelming favorite -- not nearly as big a favorite as UNLV in '91 or Kentucky vs. Wisconsin in '15.

PS And I would argue that Duke's dominance of the ACC that season was a little like Kentucky's dominance of the SEC this season. The ACC in 1999 was unusually down for the ACC. Just three teams made the NCAA Tournament. Maryland was very good (better than anyone in the SEC other than UK this year) -- 28-6 and a Sweet 16 team. UNC was better than any non-UK SEC team this year -- 24-10 and No. 13 in the nation, but that was the team that lost to Weber State in the first round. The only other ACC team to win 20 was Clemson and they were 5-11 in the ACC. The fourth-place team in the league standings was Wake at 7-9 -- and they had a losing record overall.

Kedsy
04-16-2015, 03:47 PM
I would argue that, if 60% of the time any predictive model fails to include the winner in its "5 most likely," the model is faulty.

Well, then a whole lot of models have been faulty over the past five years.


Now whether a tweak is needed or an overhaul is needed I wouldn't argue, but seems to me a "good" predictive model would include the winner in its top 5 more often than not.

First of all, I picked "top 5" almost at random. Not sure what practical difference there is between top 3, top 4, top 5, top 6, etc.

Second, there wasn't a predictive model on the planet that had UConn 2011 or UConn 2014 in its top 10, much less top 5. Probably not even a poll of Jim Calhoun's living room. So over the past five years based on your definition there were very few "good" predictive models.


Any predictive model that isn't 100% accurate is somewhat faulty right?

No. Almost no predictive models of anything are 100% accurate. Doesn't make them faulty. Makes them based on probability, rather than certainty.

BobbyFan
04-16-2015, 03:54 PM
I firmly believe that Kentucky was the best team all season, from beginning to end. They lost when it mattered the most, and hence they will not go down in history (yay!). Duke just won every game, and looked very impressive doing it. We won six games in a row when it mattered. That is what counts, best team in the country or not.

I'm not sure about this. Admittedly, I agreed with your thinking through most of the tournament. But our defense was so terrific and consistent throughout the tournament, that it can't be dismissed as a transiently great stretch in a small sample size. We had become an elite defense.

Although overrated with the GOAT talk, Kentucky was rightfully considered the heavy favorite entering the tournament. Our improvement, though, significantly narrowed that gap. I wouldn't argue if Kentucky was given a very slight edge in a potential title game against us, but I'd be more inclined to flip a coin.

Des Esseintes
04-16-2015, 03:55 PM
Any predictive model that isn't 100% accurate is somewhat faulty right? that doesn't mean useless, but lady luck will always play a bit part. What is acceptable?

That's not how predictive models work. If the model labels Kentucky the favorite at 40% to win, the favorite should fall short 60% of the time. Everything is probabilistic in a predictive model. Even if the top five teams have an aggregate 80% chance of winning the title each year, there's a decent chance the 20% underdog outcome occurs two out of three years. That's how random chance works, and the models don't claim otherwise. I think you have to get a pretty big sample size of results to determine if the model's predicted percentages reasonably match or significantly deviant from actual events.

ETA: Kedsy beat me to it. I second everything he said.

Duke95
04-16-2015, 04:00 PM
No model is 100% accurate. However, absolute metaphysical certainty isn't the standard. If it were, the field of statistics would be essentially nonexistent, since we deal with uncertainty.

There are so many variables to capture, that it is simply impossible to account for all of them in any model. So, a degree of stochastic error should be expected. I actually though these projections were quite good, other than Villanova and UVa, but I can understand that some factors may not lend themselves to accurate measurement (e.g., the impact of Anderson's injury, and how recovered he was).

tbyers11
04-16-2015, 04:16 PM
While 1999 Duke was -- I believe -- the best team that didn't win the tournament in the last quarter century (better than 1991 UNLV and 2015 Kentucky), I think you are incorrect to describe them as the "overwhelming favorite."

Yes, Duke finish the 1999 season No. 1 after a dominant run, but we forget that they were No. 1 for just eight weeks during the season, while UConn was No. 1 for 10 weeks.

Duke was 36-1 going into the title game, but UConn was 33-2 -- and both losses had come when center Jake Voskuhl was out with a sprained ankle.

Just trying to explain that Duke was favored that night in St. Pete, but the Devils weren't an overwhelming favorite -- not nearly as big a favorite as UNLV in '91 or Kentucky vs. Wisconsin in '15.


There are many different ways that one could suggest that one team is better than another. However, when I see the term favored I immediately think of the Vegas line. According to statsheet (http://statsheet.com/mcb/games/1999/03/29/connecticut-77-duke-74), Duke was a 9.5 point favorite over UConn in the 99 title game. UK was only a 5 point favorite over Wisconsin this year. Duke 99 was a much bigger favorite than UK was this year. I have never been able to find a line for the 91 UNLV-Duke game (IIRC, Vegas wasn't allowed to post lines on UNLV at that time) but it would have to be double digits to be higher than Duke. For another point of reference Duke was a 7 point favorite over Butler (a 5 seed) in 2010.

UConn in 99 was a really good team, but I think Duke 99 was quite a better regardless of how the timing of their losses dictated their poll ranking. Vegas seems to agree that they were an overwhelming favorite too. However, a single game (relative to a 7 game series) greatly decreases the likelihood of the favorite coming out on top

Jarhead
04-16-2015, 04:27 PM
One thing I give both 538 and Pomoroy credit for -- both had Kentucky at less than 50 percent offs to win the tourney. In that sense, both projections were right ...

That's in sharp contrast to the so-called expert commentators who suggested that we should take Kentucky against the field.

I wonder if in the future, people will remember just HOW overwhelming a favorite Kentucky was going into the tournament?

Kentucky was clearly a superior team in the SEC where they had played a large part of their schedule. All it took for them to lose was to play against Wisconsin, a big, big step up from the teams it was accustomed to playing in the regular season. So they lost, which also partly explains the deficiencies of the systems that predicted the tourney outcome. It looks like strength of schedule was weakly considered. And don't forget the human factor. Wisconsin wanted to play against Duke, so they had to beat Kentucky. Ta Da.

Kfanarmy
04-16-2015, 04:42 PM
Well, then a whole lot of models have been faulty over the past five years.



First of all, I picked "top 5" almost at random. Not sure what practical difference there is between top 3, top 4, top 5, top 6, etc.

Second, there wasn't a predictive model on the planet that had UConn 2011 or UConn 2014 in its top 10, much less top 5. Probably not even a poll of Jim Calhoun's living room. So over the past five years based on your definition there were very few "good" predictive models.



No. Almost no predictive models of anything are 100% accurate. Doesn't make them faulty. Makes them based on probability, rather than certainty.
I think we are simply applying the word "faulty" differently. I'm using it in the sense that it means "imperfect," you seem to be interpretting my usage as "not fit for the use intended." My point was that a performance standard needs to be established before a predictive model can be deemed acceptable. So even if you pick a standard "almost at random," apply it.
At some point though the user has to understand the utility of an individual model: does it do what is is intended to do. Given a season's worth of data, what should the performance standard for a model intended to predict the winner of the NCAA tournament be?

BobbyFan
04-16-2015, 04:42 PM
There are many different ways that one could suggest that one team is better than another. However, when I see the term favored I immediately think of the Vegas line. According to statsheet (http://statsheet.com/mcb/games/1999/03/29/connecticut-77-duke-74), Duke was a 9.5 point favorite over UConn in the 99 title game. UK was only a 5 point favorite over Wisconsin this year. Duke 99 was a much bigger favorite than UK was this year. I have never been able to find a line for the 91 UNLV-Duke game (IIRC, Vegas wasn't allowed to post lines on UNLV at that time) but it would have to be double digits to be higher than Duke. For another point of reference Duke was a 7 point favorite over Butler (a 5 seed) in 2010.

UConn in 99 was a really good team, but I think Duke 99 was quite a better regardless of how the timing of their losses dictated their poll ranking. Vegas seems to agree that they were an overwhelming favorite too. However, a single game (relative to a 7 game series) greatly decreases the likelihood of the favorite coming out on top

This is spot on. UConn in 1999 was an excellent team that would be considered among the favorites most seasons. I don't they were considered the consensus #2 team though, as Michigan State was right there too. Duke, on the other hand, was a historically great team.

UConn being ranked at the top of the polls for several weeks was influenced by us losing early in the season. But there was consistently a significant gap (larger than with this year's Kentucky team) between us and the rest in the Sagarin ratings throughout the season. The same would surely hold for the RPI ratings.

Kedsy
04-16-2015, 04:44 PM
However, when I see the term favored I immediately think of the Vegas line.

But we also have to remember that what Vegas is really trying to do is even out the betting. It's possible the line was set at 9.5 in 1999 because the bookies really thought Duke was 9.5 points better, but it's also possible they set the line there because they thought it had the best chance of having half the people on either side of the bet.


Kentucky was clearly a superior team in the SEC where they had played a large part of their schedule. All it took for them to lose was to play against Wisconsin, a big, big step up from the teams it was accustomed to playing in the regular season. So they lost, which also partly explains the deficiencies of the systems that predicted the tourney outcome. It looks like strength of schedule was weakly considered. And don't forget the human factor. Wisconsin wanted to play against Duke, so they had to beat Kentucky. Ta Da.

I disagree with almost all of this. Schedule strength is very much taken into account in these rating systems. And the fact that Wisconsin beat Kentucky in one game says nothing about the "deficiencies" of any predictive system. Seriously, this year's Duke team got clobbered by Miami at Cameron. Does that mean the systems (ALL of them) that predicted Duke to win that game were deficient?

Kentucky was a better team than Wisconsin. Perhaps not as much better as people thought. And perhaps the UK players got a bit overconfident (which does reflect the human factor, unlike the idea that Wisconsin somehow played better against UK because they wanted to face Duke). But if the two teams played an extended series, I believe Kentucky would have beaten Wisconsin more than half the time.

Kedsy
04-16-2015, 04:46 PM
At some point though the user has to understand the utility of an individual model: does it do what is is intended to do. Given a season's worth of data, what should the performance standard for a model intended to predict the winner of the NCAA tournament be?

OK, I'll accept that. But I don't know that anybody, anywhere would have a decent answer to that question.

sagegrouse
04-16-2015, 04:49 PM
I had high hopes for this year's Duke team as the regular season ended. We were thumping good teams, and we had won on the road, displaying a good amount of toughness.

I had some trepidation after we lost to the Irish in the ACC's, but I was wary back then of playing a dangerous team that we had recently beaten by 30. As it turns out, we won comfortably from then on.

If 538's results, like KenPom's, are driven off of game statistics, I am not surprised it was out to lunch. As it turns out, the ACC was by far the strongest conference -- only it didn't necessarily show it in November and December, when interconference play takes place and the only available data are collected. I looked in very great detail at the all the major conference results, and there were some very "distortive" data out there from the early season. Take the Big 12, for example. No big wins from the top of the league, but Big 12 bottom-feeder Texas Tech happened to win a couple of games outside the conference. And, owing to the necessities of statistical estimation, when the Red Raiders got pummeled within the league, all of the Big 12 boats rose a few notches, or more.

Looking at the TV screen and then the court in Indianapolis, I thought the Kentucky guards -- the Harrisons -- were slow as Christmas. CBB is a guards' game, and we learned it in 2014 (UConn) as well as this year. Wisconsin, whose guards were not spectacular, carved up Kentucky but then couldn't stay with Duke. Our speed advantage against Wisconsin was overwhelming, overcoming two shaky spells -- one in each half.

CDu
04-16-2015, 04:52 PM
Given a season's worth of data, what should the performance standard for a model intended to predict the winner of the NCAA tournament be?

The problem is sample size. We have an N of 1 each year with a dichotomous measure (correct/incorrect) of a model's performance in predicting the winner of the NCAA Tournament. And the winner of the tournament is dependent upon 6 games of chance (and I'm only talking about those games for the actual winner; 57 other games not including the play-in games factor in as well). Since even the team you would think has the best chance doesn't have a 100% chance of winning every game, it's unlikely that a model will predict the ultimate winner.

For example, note that these models ALL suggest that no individual team is likely to win it. Even the most prohibitive favorites tend to have a less than 50% chance of winning. Kentucky was given a 33-40% chance. That meant they had a 60-67% chance of not winning it. Do you give the models credit for accurately predicting that Kentucky wouldn't win it? [of course not]

A better measure might be to look at what percentage of the results in a given time period the model predicted "correctly". Again, this is based on chance, so the outcome observed isn't necessarily a true reflection of the quality of the teams. But at least then we get a step closer to a sufficient sample size to measure the predictive capability of the model. And even then, I'm not sure that an N of 63 (the number of games played in the 64-team field) would be sufficient.

Basically, the issue all boils down to the fact that we have only the observed, one-time event outcome and not the probability distribution. We don't play 1000-game iterations of each matchup, which is essentially the concept on which the models are based. So the model (which predicts based on thousands of simulations) should not be expected to replicate a single outcome such as the tournament champion over six single-game eliminations.

Maybe if we had hundreds of tournaments we could get a sample size large enough to assess the performance. But that would obviously take hundreds of years. So we're sort of stuck with some degree of reliance on faith (or lack thereof) in the models.

Kfanarmy
04-16-2015, 04:56 PM
That's not how predictive models work. If the model labels Kentucky the favorite at 40% to win, the favorite should fall short 60% of the time. Everything is probabilistic in a predictive model. Even if the top five teams have an aggregate 80% chance of winning the title each year, there's a decent chance the 20% underdog outcome occurs two out of three years. That's how random chance works, and the models don't claim otherwise. I think you have to get a pretty big sample size of results to determine if the model's predicted percentages reasonably match or significantly deviant from actual events.

ETA: Kedsy beat me to it. I second everything he said.

Again...what should the standard for accuracy be? Is it acceptable that in two of three chances, a model doesn't include the winner in the top five of its most likely winners?

tbyers11
04-16-2015, 04:58 PM
But we also have to remember that what Vegas is really trying to do is even out the betting. It's possible the line was set at 9.5 in 1999 because the bookies really thought Duke was 9.5 points better, but it's also possible they set the line there because they thought it had the best chance of having half the people on either side of the bet.


Agree with this to a point. I don't know what the opening line was. If it was Duke by 7 and moved to 9.5 by game time as $$$ kept coming in on Duke then I definitely agree.

However, I anecdotally tracked point spreads vs KPom spreads for Duke games most of the year. The lines and KPom spreads were almost always within 2 points (usually Vegas liked Duke more than KPom). I'd think the same would have held true in 1999. So even if Vegas moves the line a little bit it usually isn't that far from the better publicly available predictive models and Duke would still have been a very strong favorite in 1999 against UConn

CDu
04-16-2015, 05:15 PM
Again...what should the standard for accuracy be? Is it acceptable that in two of three chances, a model doesn't include the winner in the top five of its most likely winners?

See my comment above. Yes, it should be acceptable, because the model predicted some chance for the winner. We're talking about 63 games, with probably 50 of them being nearly toss-ups. For each individual team, it amounts to 4 or 5 coin flips. If you flip a coin 4 times, how often do you expect to get heads each time [the answer is roughly 6%]? Well, that's essentially what is happening. Throughout the tournament, you have roughly 40-50 individual coin flips, each affecting the probability of the model predicting the overall winner.

More simply, any model is going to predict that the "best" (as perceived by the model) team wins. But the best team doesn't always win the tournament. Sometimes it isn't close to the best team that wins the tournament. UConn wasn't the best team in 1999 nor were they close to the best in 2011 or 2014. Villanova wasn't remotely the best team in 1985. Butler wasn't at all close to being the best team in 2010, but they were a couple inches from winning the title. Arizona wasn't nearly the best team in 1997. Syracuse wasn't the nearly best team in 2003. Duke may or may not have been the best team in 2015. Florida probably wasn't the best team in 2006. UConn was probably the best team in 2004, but they didn't play like it over the entire season. With that many coin flip games, it is highly likely that the best team won't win it. And if it is high likely that the best team won't win it, it's likely that the 2nd or 3rd or 4th best team won't win it either. Kentucky wasn't close to the best in 1998. Kansas wasn't close to the best in 1988.

Those models predicted a 20-25% chance that a team outside the top-5 would win. Is it a flawed model if that 20-25% chance happens? Not necessarily. We'd need a MUCH bigger sample size of outcomes to really assess it (which we don't have).

Wander
04-16-2015, 07:35 PM
I actually do think the models were faulty in regards to Duke's championship chances, but that the faultiness is basically just due to Utah and Virginia being substantially overrated by the computers (for Virginia, that was just because of the injury situation).

Duke95
04-16-2015, 07:50 PM
I actually do think the models were faulty in regards to Duke's championship chances, but that the faultiness is basically just due to Utah and Virginia being substantially overrated by the computers (for Virginia, that was just because of the injury situation).

I don't think that is the case with Utah. Keep in mind, these are conditional probabilities, not simple ones (at least as far as I can tell from their site). Facing a tough opponent, which we did in Utah, will lower our probability of overall victory. The game was very close, as was predicted. In fact, had Justise not missed those last two free throws in the finals, Utah would have been our closest game.

Wander
04-16-2015, 08:02 PM
I don't think that is the case with Utah. Keep in mind, these are conditional probabilities, not simple ones (at least as far as I can tell from their site). Facing a tough opponent, which we did in Utah, will lower our probability of overall victory.

I know, but I'm saying that in my opinion Utah wasn't as good as the computers thought, and if the computers had accurately rated them then Duke's championship percentage would be more in line with what we'd intuitively expect. Note that Utah went 1-5 against the other tournament teams in their conference. A good team, but overrated at 8 in kenpom's final standings.

HK Dukie
04-16-2015, 09:11 PM
The fundamental flaw in 538 and kenpom is that they equally weight all possessions throughout the season.

Tell me, does a possession in garbage time against a cupcake by walkons in december matter as much as a possession in the the last 15 games of the regular season and conference tournament FOR determining just how efficient a defense or offense is per possession in March? You are right, it doesn't.

There is no cookie cutter way to decide what data should be included. Is it the last 3 games, 10 games, 20 games, the whole season? That is why these systems default to the whole season. Obviously recent data is better but too small a sample set (like Dekker 3ppg% in the NCAA being 50% vs 30% for career indicating his three ability was overrated by the pundits even though we all like his tenacity) and you lose accuracy of prediction of your data (like a poll done on 50 instead of 500 likely voters). So while whole season data makes sense to have consistency in GENERAL, we can do better than that for specific teams when we have specific new information on the team. We can debate if the past 5 games or 15 games makes sense in predicting the future. Maybe a new player has emerged and that data is not in the season averages (no model had much data for Allen for example).

So note, when you are filling out your brackets next year, understand why these models are spitting out what they do. If a team had a big change, I don't know maybe say kicking off a player from the team, and then they go on a massive run winning 12 of the next 13 games with elite offensive and defensive efficiency (maybe better than UK during that time span), loses a game in the semifinals of a tournament thereby taking away most of the pundit herd mentality of support.....well you may have just found an UNDERVALUED team for the NCAA tournament brackets. Does this remind you of anyone???

So to conclude, Duke had much better odds than kenpom and 538 predicted based on our performance in the last 6 weeks of the season. The pundits also dropped any bandwaggoning because the team lost in the ACC tournament. So computers and people both undervalued Duke.

subzero02
04-16-2015, 09:21 PM
So, I revisited the 538 blog to see how the predictions worked out.

http://fivethirtyeight.com/interactives/march-madness-predictions-2015/#mens

These are their estimated probabilities of winning it all.

Kentucky - 41%
Villanova - 11%
Wisconsin - 10%
Arizona - 9%
Virginia - 8%
Duke 6%

Duke had the 6th highest probability of winning, lowest among 1 seeds and lower than two 2 seeds. Nobody else had higher than 3%.
So, basically, UK had almost 7 times the probability of winning it all than Duke. Duke was only given a 32% chance of getting out of their bracket, again, lowest among the 1 seeds. (In fact, Arizona was favored to get to the FF over Wisconsin.)

I guess there are two points:
1. Some opposing fans claim we had an easy bracket. However, we weren't given as high a chance to make it out as the other 1 seeds.
2. Duke came together at the right time, and some of the adjustments at the NCAAs should be credited to the coaching staff and the players, who seemed to step it up even another notch compared to previous games.

Pretty amazing year.

Defense wins championships... Our team defense improved dramatically and we also had some remarkable individual performances ( cook on pangos, matt on dekker, and amile on kaminsky are performances I won't be forgetting anytime soon). Okafor also showed improvement... Winslow was Winslow and Grayson Allen showed some quality on ball defense as well.

Kedsy
04-16-2015, 09:54 PM
That is why these systems default to the whole season.

Except I don't think the systems do use the entire season, at least not equally. I'm almost certain I've read Pomeroy saying he weights the more recent games more heavily (not certain how many games). And he doesn't really rate a possession against a cupcake the same as one against a strong team -- he adjusts the efficiencies based on expected opponent performance. Finally, I'm almost certain he uses some sort of diminishing returns principles, so garbage time possessions are devalued. Again, I don't know exactly how it works.

So I don't think your explanation is accurate. There is truth in it, but the actual truth is a whole lot more complicated.

JasonEvans
04-16-2015, 09:56 PM
I wonder if in the future, people will remember just HOW overwhelming a favorite Kentucky was going into the tournament?

I think history will look back on this year and call 2015 Duke one of the better teams of the decade. Their margin of victory in the tourney was impressive. From Feb to the end of the tourney they only lost 1 game. And, I am betting that when we look back on the NBA careers of Jahlil and Justise (and perhaps Tyus), we are going to marvel at how teams even stayed on the court with Duke. I fully expect Justise and Jahlil to be perennial all-stars and to rival guys like Kyrie, Elton, and Grant as the top Duke NBAers of all time.

Yeah... I went there.

-Jason "Luol and 'Los are also in the conversation... I'm not counting guys from decades ago who are not part of the present public consciousness" Evans

sagegrouse
04-16-2015, 10:04 PM
Except I don't think the systems do use the entire season, at least not equally. I'm almost certain I've read Pomeroy saying he weights the more recent games more heavily (not certain how many games). And he doesn't really rate a possession against a cupcake the same as one against a strong team -- he adjusts the efficiencies based on expected opponent performance. Finally, I'm almost certain he uses some sort of diminishing returns principles, so garbage time possessions are devalued. Again, I don't know exactly how it works.

So I don't think your explanation is accurate. There is truth in it, but the actual truth is a whole lot more complicated.

Of course, Kedsy, to compare teams across conferences the ONLY data are games from Nov. and Dec. -- therefore, they HAVE to have a high weight.

You can view CONFERENCE data as tightly bounds balls of string with 150-200 data points (individual games) that provides gobs of data for the analyst to evaluate teams WITHIN the same conference. But these balls of data are all floating in the ether, and the only things that put them in a relative framework are the non-conference games -- there weren't vary many significant games and they were early in the season.

Jarhead
04-16-2015, 11:00 PM
But we also have to remember that what Vegas is really trying to do is even out the betting. It's possible the line was set at 9.5 in 1999 because the bookies really thought Duke was 9.5 points better, but it's also possible they set the line there because they thought it had the best chance of having half the people on either side of the bet.



I disagree with almost all of this. Schedule strength is very much taken into account in these rating systems. And the fact that Wisconsin beat Kentucky in one game says nothing about the "deficiencies" of any predictive system. Seriously, this year's Duke team got clobbered by Miami at Cameron. Does that mean the systems (ALL of them) that predicted Duke to win that game were deficient?

Kentucky was a better team than Wisconsin. Perhaps not as much better as people thought. And perhaps the UK players got a bit overconfident (which does reflect the human factor, unlike the idea that Wisconsin somehow played better against UK because they wanted to face Duke). But if the two teams played an extended series, I believe Kentucky would have beaten Wisconsin more than half the time.

We have no way of knowing for sure that Kentucky was a better team than Wisconsin, except when they played one another in the Final Four. Strength of schedule statistics failed to consider the weakness of the SEC teams. In that game Wisconsin was the better team. The rules say that Kentucky was eliminated. When I watched that game I worried about Duke's chances two days later. There was no way the bookies, or the statisticians could have predicted the results. The bookies predicted Duke's margin of victory at 2.5. As it happened, Duke won by 5, and a good reason for that is that two of the smallest guys on the team got an emotional boost and saw to it that our team won. Why didn't the statisticians include that possibility in their calculations? They didn't have enough to go on since it was a single elimination tourney, perhaps. In the end emotions trump statistics, and predictions are a crap shoot.

uh_no
04-16-2015, 11:26 PM
The fundamental flaw in 538 and kenpom is that they equally weight all possessions throughout the season.


As others have pointed out, this is simply false...at least for kenpom. It's quite well known that recent games are given significantly more weight than far past games, and games in which you blow out an overmatched opponent are given less weight.

That doesn't counter the fact that all posessions within a game are treated the same. Kenpom has in his blog (if i recall) tried to make adjustments for end games, but generally when he doesn't adjust for something like that it's for one of 3 reasons

1) it doesn't significantly affect the ratings
2) he can't find a good way to determine what is or isn't a "garbage" posessions
3) adding in the factor decreased the overall predictive power of the model

OZ
04-16-2015, 11:28 PM
I am grateful to the sports gods, that with all the "insiders," analyses, prognosticators, stats, computers, percentages and or "premium" sights, no one can guarantee a winner. Last year Connecticut, this year Duke. Each time, I drive the 200 miles round trip to Cameron or follow them elsewhere, there is always -NO MATTER whom we play-that bit of fear (I was actually at the Wagner game) accompanied by a greater since of hope.
Before we played the first game of the tournament, there were those who filled the boards with their dire warnings of North Florida... I got that!!! Though, we were expected to win easily against Robert Morris - I was still nervous. When we played Wisconsin, most of the basketball "experts" that I heard didn't give Duke much of a chance. In fact, there were quit a few, who didn't think it would be a close game. Yes, perhaps, "statistically, Duke's championship was very unlikely;" but thankfully, stats can't predict a player's heart... or a coach's decision in critical times...nor can they foretell a lesser known having his "shinning moment" at just the right time.
And that is why I honestly try to avoid any and all analyses and predictions of games. They would not affect me any way; because I always enter the stadium the same way - a bundle of nerves with a fearful respect of the other team and a healthy dose of optimism for Duke. With all due respect, while they might be fun, amusing, interesting or informative, the moment the likes of "538 blog" is able predict Duke's basketball games is the moment I turn in my tickets and stay home.

Kedsy
04-16-2015, 11:45 PM
Of course, Kedsy, to compare teams across conferences the ONLY data are games from Nov. and Dec. -- therefore, they HAVE to have a high weight.

You can view CONFERENCE data as tightly bounds balls of string with 150-200 data points (individual games) that provides gobs of data for the analyst to evaluate teams WITHIN the same conference. But these balls of data are all floating in the ether, and the only things that put them in a relative framework are the non-conference games -- there weren't vary many significant games and they were early in the season.

Yes, of course I know your viewpoint on this. And there's a lot of validity to it. I still believe, however, that it's a lot more complicated than you make it seem.


We have no way of knowing for sure that Kentucky was a better team than Wisconsin, except when they played one another in the Final Four.

Well, first of all, that's not true. We have ways of knowing who the best team is. You just don't believe those ways. Tell me point blank, do you think Miami was a better team than Duke this year? Head-to-head is a good data point, but it doesn't tell us who the better team is. Ever.

Second, I was at the Wisconsin/Kentucky game. I was rooting for Wisconsin, I thought they were a great team. But I thought Kentucky was the better team. I thought if they played an extended series that Kentucky would win more than half the games. So at least in my case, the "eye test" coincided with the computer ratings.


Strength of schedule statistics failed to consider the weakness of the SEC teams.

Actually, I'm pretty sure the strength of schedule statistics totally considered the weakness of the SEC teams. Do you think the SOS numbers in the various rating systems were only non-conference?


The bookies predicted Duke's margin of victory at 2.5. As it happened, Duke won by 5, and a good reason for that is that two of the smallest guys on the team got an emotional boost and saw to it that our team won. Why didn't the statisticians include that possibility in their calculations? They didn't have enough to go on since it was a single elimination tourney, perhaps. In the end emotions trump statistics, and predictions are a crap shoot.

My understanding was Wisconsin was favored by 1. Duke ended up winning what was basically a toss-up game, but I don't understand your point. Are you saying that because an underdog can win a game then there's no point to figuring out who's favored? Because that doesn't make much sense to me. Are you saying statistics don't tell us anything? Are you suggesting that every team has a 50/50 chance against every other team? Because that's clearly false. I honestly have no idea what you're trying to say.

darthur
04-17-2015, 12:23 AM
What makes a good predictive model? Is it acceptable to include the winner in the top 50% most likely half the time? Does it need to have the winner in the top two most of the time...There's an awful lot of room in between to gauge success.

Has anyone compared which of Nate Silver's and Pomeroy's model was "better"? Each one assigns a probability of the tournament going the way it did. Which one assigned it the higher probability? Or even which one favored the winners most often?

It always makes me sad that we have so many ranking systems but nobody publishes data on how good they actually are. Seems like it would be straightforward to measure kenpom vs sagarin vs seeds vs AP rankings vs RPI vs BPI vs whatever year in and year out, but nobody does it. And sadly, historical records of these rankings are often not available.

Kedsy
04-17-2015, 12:31 AM
Has anyone compared which of Nate Silver's and Pomeroy's model was "better"? Each one assigns a probability of the tournament going the way it did. Which one assigned it the higher probability? Or even which one favored the winners most often?

It always makes me sad that we have so many ranking systems but nobody publishes data on how good they actually are. Seems like it would be straightforward to measure kenpom vs sagarin vs seeds vs AP rankings vs RPI vs BPI vs whatever year in and year out, but nobody does it. And sadly, historical records of these rankings are often not available.

Don't know if this gives you what you want, but here's a composite of a lot of rating systems (http://masseyratings.com/cb/arch/) dating back to 2002. Anyway, if you want to compare to actual results, there's a fair number of data points here.

YmoBeThere
04-17-2015, 05:17 AM
...is the moment I turn in my tickets and stay home.

I'll take them.

bjornolf
04-17-2015, 05:55 AM
If we had such an easy road to the FF, then how come everybody and their brother picked us not to survive our region, saying Utah would beat us, and if not, Gonzaga would?

Mtn.Devil.91.92.01.10.15
04-17-2015, 08:50 AM
It is amazing to me how many people misunderstand statistics. Silver saying Duke had a six percent chance of winning before the tournament is not necessarily wrong. It works much like predicting the weather - the only way predictive statistics can "be wrong" is if you use absolutes.

If Silver said Duke had a zero percent chance of winning, we could say without question he was incorrect. If he said Duke had a six percent chance and we won, he is not necessarily wrong. If he said we had a six percent chance and we won ten times in a row, either he is mistaken in his calculations or we are wildly fortunate.

I still don't understand why weather forecasters ever say there is either a zero percent or one hundred percent chance of rain. It is the only way to be proven wrong.

CDu
04-17-2015, 08:56 AM
It is amazing to me how many people misunderstand statistics. Silver saying Duke had a six percent chance of winning before the tournament is not necessarily wrong. It works much like predicting the weather - the only way predictive statistics can "be wrong" is if you use absolutes.

If Silver said Duke had a zero percent chance of winning, we could say without question he was incorrect. If he said Duke had a six percent chance and we won, he is not necessarily wrong. If he said we had a six percent chance and we won ten times in a row, either he is mistaken in his calculations or we are wildly fortunate.

I still don't understand why weather forecasters ever say there is either a zero percent or one hundred percent chance of rain. It is the only way to be proven wrong.

Exactly. There is just no way to know if the model was right in its estimated probabilities, because we only see one outcome and not the 1,000 (or 10,000, or whatever number of simulations were run) that the models are based on. Even if the model said Duke had a 0.000000001% chance of winning, it's possible (though roughly 0.000000001% likely, ignoring any of the other games of course) that the model could be correctly and Duke could win. It's very possible that the true probability of Duke winning this year was around 6%, and that we beat the odds. No way to know, simply because we only get to observe the one observation.

wilson
04-17-2015, 09:27 AM
Lots of nice statistical analysis and explanation here from people who know a lot more about the topic than I do...thanks to all for your insights.
Could it also be that statistical predictions, though they are often highly astute and have some strong predictive abilities, are losing their ability to confidently predict college basketball results? The rise of statistical sports analysis has been a pretty big story in the last decade or so, but so too has been the supposed decay of college hoops, owing to the high rate of turnover among top-level talent and the lack of long-term chemistry and development. As that trend continues, could the constant flux be introducing a higher level of variance into the various statistical models? Any given college basketball roster among serious contenders is almost always going to have a low number of data points, because it's so difficult to keep an entire team intact for any length of time. I think perhaps that is making it more difficult to predict the game, especially in a volatile, traditionally unpredictable environment like the NCAA Tournament.

Bostondevil
04-17-2015, 09:56 AM
"The one in a million occurrence happens with no more and no less than the expected frequency, no matter how surprised we might be when it happens to us." R.A. Fisher (the inventor of regression analysis)

538 gave Duke a 6% chance of winning at the BEGINNING of the tournament. I'd say that was about right. Granted before the tournment, Kentucky had the best odds at the beginning, but so what? The odds would change after every round.

tbyers11
04-17-2015, 10:02 AM
Has anyone compared which of Nate Silver's and Pomeroy's model was "better"? Each one assigns a probability of the tournament going the way it did. Which one assigned it the higher probability? Or even which one favored the winners most often?

It always makes me sad that we have so many ranking systems but nobody publishes data on how good they actually are. Seems like it would be straightforward to measure kenpom vs sagarin vs seeds vs AP rankings vs RPI vs BPI vs whatever year in and year out, but nobody does it. And sadly, historical records of these rankings are often not available.

Ask an ye shall receive. Sort of. 538 looks at how their forecasts did (http://fivethirtyeight.com/features/how-fivethirtyeights-ncaa-tournament-forecasts-did/#fn-2).

1) Comparisons relative to Vegas in the first rounds and later rounds and 2) against other models throughout the course of the tourney.
Interesting data but not any real clear cut answers. 538 did quite well in the first round but not as well later on.

roywhite
04-17-2015, 10:33 AM
"The one in a million occurrence happens with no more and no less than the expected frequency, no matter how surprised we might be when it happens to us." R.A. Fisher (the inventor of regression analysis)

538 gave Duke a 6% chance of winning at the BEGINNING of the tournament. I'd say that was about right. Granted before the tournment, Kentucky had the best odds at the beginning, but so what? The odds would change after every round.

At the start of the tournament, Duke came in:
ranked #4 in the country
1 loss since January
road wins @Wisconsin, Louisville (full roster for the Cards), Virginia (full roster), Syracuse, and UNC.
with no major injuries

So, a 6% chance of winning? Not in my opinion, given those facts

To each his own, but I don't see much predictive value from 538 here, and even kenpom has it's limitations.

alteran
04-17-2015, 10:46 AM
Nate Silver's predictions don't factor in Duke's decided officiating advantage... :rolleyes:

Excellent point. You'd think a genius statistician like Nate would have caught that based on numbers analysis of previous games. I wonder how it missed him.

ns7
04-17-2015, 10:51 AM
While 1999 Duke was -- I believe -- the best team that didn't win the tournament in the last quarter century (better than 1991 UNLV and 2015 Kentucky), I think you are incorrect to describe them as the "overwhelming favorite."

Yes, Duke finish the 1999 season No. 1 after a dominant run, but we forget that they were No. 1 for just eight weeks during the season, while UConn was No. 1 for 10 weeks.

Duke was 36-1 going into the title game, but UConn was 33-2 -- and both losses had come when center Jake Voskuhl was out with a sprained ankle.

Just trying to explain that Duke was favored that night in St. Pete, but the Devils weren't an overwhelming favorite -- not nearly as big a favorite as UNLV in '91 or Kentucky vs. Wisconsin in '15.

PS And I would argue that Duke's dominance of the ACC that season was a little like Kentucky's dominance of the SEC this season. The ACC in 1999 was unusually down for the ACC. Just three teams made the NCAA Tournament. Maryland was very good (better than anyone in the SEC other than UK this year) -- 28-6 and a Sweet 16 team. UNC was better than any non-UK SEC team this year -- 24-10 and No. 13 in the nation, but that was the team that lost to Weber State in the first round. The only other ACC team to win 20 was Clemson and they were 5-11 in the ACC. The fourth-place team in the league standings was Wake at 7-9 -- and they had a losing record overall.

Edit: looks like this has been answered already...

We were a 9.5 favorite over UConn in that title game. I can't remember the exact pre-tournament odds, but remember it being better than even money.

I believe UK was a ~5-6 point favorite over Wisconsin. Not sure there was a line for the UNLV game because of the special circumstances.

I agree that the ACC was down, but that 1999 team absolutely destroyed everyone. Compare that to UK this year and their multiple close wins.

ns7
04-17-2015, 10:57 AM
At the start of the tournament, Duke came in:
ranked #4 in the country
1 loss since January
road wins @Wisconsin, Louisville (full roster for the Cards), Virginia (full roster), Syracuse, and UNC.
with no major injuries

So, a 6% chance of winning? Not in my opinion, given those facts

To each his own, but I don't see much predictive value from 538 here, and even kenpom has it's limitations.

That prediction was based on Duke playing mediocre defense and excellent offense. Most people didn't think that Duke's defense would suddenly become better than Kentucky's defense. But it did and I am still super thrilled that it happened.

But I don't believe you can fault any person or model who did not predict that.

Richard Berg
04-17-2015, 11:01 AM
Statistically, Duke's championship was very unlikely - 538 blog
So?

Statistically, the chance the tourney would play out the way it did was unfathomably small. Millions of people filled out brackets, yet not a single one predicted the outcome exactly.

And yet, it happened. In fact, the chance of all of the above statements being true was extremely high. Doesn't make them meaningful.

Lar77
04-17-2015, 11:09 AM
We were a 9.5 favorite over UConn in that title game. I can't remember the exact pre-tournament odds, but remember it being better than even money.

I believe UK was a ~5-6 point favorite over Wisconsin. Not sure there was a line for the UNLV game because of the special circumstances.

I agree that the ACC was down, but that 1999 team absolutely destroyed everyone. Compare that to UK this year and their multiple close wins.

I have to agree with OF. We were favored but my recollection was that it was not a shock that they beat us.

I've enjoyed the back and forth on the "validity" of statistical analyses. Kedsy has done a good job defending it and admits that we don't have all the answers because they are proprietary models. And I am sure that Pomeroy et al will continue to refine their methodology in the pursuit of perfection. The problem is see is how ESPN, CBS, and other media seem to assume it is already perfection. And that's what gets us all wound up: "You said Duke only had a 6% chance of winning. Can't you see we're better than that!!!!" I enjoy Men's CBB because of the variables and the unpredictability that no model can pick up. As in the ACC ND game, we were not the team we had been or would be during the first 24 minutes. Did Wisconsin really expect Grayson Allen to enter the pantheon of Duke legends when he didn't play against them at the Kohl Center and was the 8th man on an 8 man rotation? But he did.

Without hijacking the thread, what is the probability of UNC sanctions coming before next season?

Kedsy
04-17-2015, 11:40 AM
Could it also be that statistical predictions, though they are often highly astute and have some strong predictive abilities, are losing their ability to confidently predict college basketball results?

The answer is either no or they never had such ability. Upsets happen in every sport and college basketball is certainly not immune. The NCAA tournament, with its one-and-done format and intertwined bracket where one upset can affect so many other matchups, is by design especially prone to unpredictable results.

In any event, this is not a new phenomenon. The very first tournament with real seeding (1979) featured a 9-seed (Penn) in the Final Four, and the next year featured a 5-seed (Iowa), a 6-seed (Purdue), and an 8-seed (UCLA) in the Final Four (8-seed UCLA made the title game). In fact, the first ten seasons of seeded NCAA tournaments contained nine years with 6-seeds or worse in the Final Four (including an 11-seed (LSU) in the Final Four in 1986), and four national champions seeded #3 or worse: Indiana (3-seed in 1981), NC State (6-seed in 1983), Villanova (8-seed in 1985), and Kansas (6-seed in 1988).

The first ten year detail:

1979: 9-seed Penn in Final Four;
1980: 8-seed UCLA in title game (as well as 5-seed Iowa and 6-seed Purdue in Final Four);
1981: 3-seed Indiana winning championship;
1982: 6-seed Houston in Final Four;
1983: 6-seed NCSU winning championship (and 4-seed Georgia in Final Four);
1984: 7-seed Virginia in Final Four;
1985: 8-seed Villanova winning championship;
1986: 11-seed LSU in Final Four;
1987: 6-seed Providence in Final Four;
1988: 6-seed Kansas winning championship.

The next ten years weren't as crazy, but still had three very unlikely champions (Michigan in 1989, Duke in 1991, and Arizona in 1997), as well as:

1989: Title game between two 3-seeds (Michigan and Seton Hall);
1990: Three teams seeded #3 or worse in the Final Four (#3 Duke, #3 Arkansas, and #4 Georgia Tech);
1991: Duke over UNLV
1992: 6-seed Michigan in title game (and 4-seed Cincinnati in Final Four);
1996: 4-seed Syracuse in title game (and 5-seed Mississippi State in Final Four);
1997: 4-seed Arizona winning championship;
1998: Two 3-seeds (Utah and Stanford) in Final Four.

In the 16 years of the new century, we've seen four champions seeded #3 or worse and 19 Final Four teams seeded #4 or worse, including:

2000: 5-seed Florida in the title game and 8-seed UNC and 8-seed Wisconsin in Final Four;
2002: 5-seed Indiana in the title game;
2003: 3-seed Syracuse winning championship;
2005: 5-seed Michigan State in Final Four;
2006: 3-seed Florida winning championship and 11-seed George Mason in Final Four (as well as 4-seed LSU);
2010: 5-seed Butler in title game (and 5-seed Michigan State in Final Four);
2011: 3-seed UConn winning a Final Four that included 4-seed UK, 8-seed Butler, and 11-seed VCU;
2012: 4-seed Louisville in Final Four;
2013: 9-seed Wichita plus two 4-seeds (Syracuse and Michigan) in Final Four;
2014: 7-seed UConn beating 8-seed Kentucky in title game;
2015: 7-seed Michigan State and "unlikely" champion Duke.

So basically the NCAA tournament has been a crapshoot at least since they started seeding and probably a lot longer than that. Anyone who expects a rating system to accurately predict the results of the tournament is generally going to be disappointed.

Mtn.Devil.91.92.01.10.15
04-17-2015, 11:42 AM
So?

Statistically, the chance the tourney would play out the way it did was unfathomably small. Millions of people filled out brackets, yet not a single one predicted the outcome exactly.

And yet, it happened. In fact, the chance of all of the above statements being true was extremely high. Doesn't make them meaningful.

I suppose at the end of the day, I agree with you here.

Duvall
04-17-2015, 11:55 AM
At the start of the tournament, Duke came in:
ranked #4 in the country
1 loss since January
road wins @Wisconsin, Louisville (full roster for the Cards), Virginia (full roster), Syracuse, and UNC.
with no major injuries

So, a 6% chance of winning? Not in my opinion, given those facts

What percentage would you normally give the #4 team in the country going into the tournament?

CDu
04-17-2015, 12:14 PM
The answer is either no or they never had such ability. Upsets happen in every sport and college basketball is certainly not immune. The NCAA tournament, with its one-and-done format and intertwined bracket where one upset can affect so many other matchups, is by design especially prone to unpredictable results.

In any event, this is not a new phenomenon. The very first tournament with real seeding (1979) featured a 9-seed (Penn) in the Final Four, and the next year featured a 5-seed (Iowa), a 6-seed (Purdue), and an 8-seed (UCLA) in the Final Four (8-seed UCLA made the title game). In fact, the first ten seasons of seeded NCAA tournaments contained nine years with 6-seeds or worse in the Final Four (including an 11-seed (LSU) in the Final Four in 1986), and four national champions seeded #3 or worse: Indiana (3-seed in 1981), NC State (6-seed in 1983), Villanova (8-seed in 1985), and Kansas (6-seed in 1988).

The first ten year detail:

1979: 9-seed Penn in Final Four;
1980: 8-seed UCLA in title game (as well as 5-seed Iowa and 6-seed Purdue in Final Four);
1981: 3-seed Indiana winning championship;
1982: 6-seed Houston in Final Four;
1983: 6-seed NCSU winning championship (and 4-seed Georgia in Final Four);
1984: 7-seed Virginia in Final Four;
1985: 8-seed Villanova winning championship;
1986: 11-seed LSU in Final Four;
1987: 6-seed Providence in Final Four;
1988: 6-seed Kansas winning championship.

The next ten years weren't as crazy, but still had three very unlikely champions (Michigan in 1989, Duke in 1991, and Arizona in 1997), as well as:

1989: Title game between two 3-seeds (Michigan and Seton Hall);
1990: Three teams seeded #3 or worse in the Final Four (#3 Duke, #3 Arkansas, and #4 Georgia Tech);
1991: Duke over UNLV
1992: 6-seed Michigan in title game (and 4-seed Cincinnati in Final Four);
1996: 4-seed Syracuse in title game (and 5-seed Mississippi State in Final Four);
1997: 4-seed Arizona winning championship;
1998: Two 3-seeds (Utah and Stanford) in Final Four.

In the 16 years of the new century, we've seen four champions seeded #3 or worse and 19 Final Four teams seeded #4 or worse, including:

2000: 5-seed Florida in the title game and 8-seed UNC and 8-seed Wisconsin in Final Four;
2002: 5-seed Indiana in the title game;
2003: 3-seed Syracuse winning championship;
2005: 5-seed Michigan State in Final Four;
2006: 3-seed Florida winning championship and 11-seed George Mason in Final Four (as well as 4-seed LSU);
2010: 5-seed Butler in title game (and 5-seed Michigan State in Final Four);
2011: 3-seed UConn winning a Final Four that included 4-seed UK, 8-seed Butler, and 11-seed VCU;
2012: 4-seed Louisville in Final Four;
2013: 9-seed Wichita plus two 4-seeds (Syracuse and Michigan) in Final Four;
2014: 7-seed UConn beating 8-seed Kentucky in title game;
2015: 7-seed Michigan State and "unlikely" champion Duke.

So basically the NCAA tournament has been a crapshoot at least since they started seeding and probably a lot longer than that. Anyone who expects a rating system to accurately predict the results of the tournament is generally going to be disappointed.

I can't give you a pitchfork, so I'll just copy and say how everyone should read this post closely to understand why it's unrealistic for any model to accurately predict the winner of a six-game single-elimination tournament.

In theory, assuming the seeding is remotely accurate, we'd think that most years the Final Four would include at least 3 if not 4 #1 or #2 seeds in the Final Four. At least, that is how any model would predict the most likely winners (though again, the probabilities of each those teams winning is going to be much lower than 50%). The evidence over the past 35 years clearly illustrates this.

But notice how rare it is the case that the Final Four is comprised entirely of the top 8 seeds. Only 9 times since 1979 has the Final Four been comprised entirely of high (#1 and #2) seeds. And more to the point (because models are typically going to predict the 1 seeds to have the highest probability of winning), only once have all four #1s made the Final Four, and only twice in the last 16 years have 3 of the four #1 seeds made the Final Four.

So by definition, any model that is predicting results based on a season's worth of data is very likely to get it wrong when considering a single-elimination tournament with at least 30 (and as many as 40 or more) coin-flip games.

blUDAYvil
04-17-2015, 12:18 PM
What percentage would you normally give the #4 team in the country going into the tournament?

13%

How do I get there? The percentage should be bounded by a minimum of 1.5% (assuming all 68 teams are of similar strength with equal likelihood of winning) and a maximum of 25% (4 teams that are equally dominant, but significantly better than the other 60). 13% falls in the middle.

A model that gave Duke a 6% chance of winning this year does not strike me as low considering Kentucky's perceived dominance (41% likelihood of winning).

pfrduke
04-17-2015, 12:21 PM
In theory, assuming the seeding is remotely accurate, we'd think that most years the Final Four would include at least 3 if not 4 #1 or #2 seeds in the Final Four. At least, that is how any model would predict the most likely winners (though again, the probabilities of each those teams winning is going to be much lower than 50%). The evidence over the past 35 years clearly illustrates this.

Just to build on this - the seeding, itself, is a "model" of sorts. The selection committee selects teams 1-68* and then seeds them in ranked order. No one says the selection committee was "wrong" just because UConn wins the title as a #7 seed, or because Kentucky doesn't win the title as the overall #1 seed. But everybody runs off and says a statistical model is "wrong" when the same thing happens. This is silly.

*of the qualifiers, since obviously several AQ teams are not in the top 68

Saratoga2
04-17-2015, 12:22 PM
While 1999 Duke was -- I believe -- the best team that didn't win the tournament in the last quarter century (better than 1991 UNLV and 2015 Kentucky), I think you are incorrect to describe them as the "overwhelming favorite."

Yes, Duke finish the 1999 season No. 1 after a dominant run, but we forget that they were No. 1 for just eight weeks during the season, while UConn was No. 1 for 10 weeks.

Duke was 36-1 going into the title game, but UConn was 33-2 -- and both losses had come when center Jake Voskuhl was out with a sprained ankle.

Just trying to explain that Duke was favored that night in St. Pete, but the Devils weren't an overwhelming favorite -- not nearly as big a favorite as UNLV in '91 or Kentucky vs. Wisconsin in '15.

PS And I would argue that Duke's dominance of the ACC that season was a little like Kentucky's dominance of the SEC this season. The ACC in 1999 was unusually down for the ACC. Just three teams made the NCAA Tournament. Maryland was very good (better than anyone in the SEC other than UK this year) -- 28-6 and a Sweet 16 team. UNC was better than any non-UK SEC team this year -- 24-10 and No. 13 in the nation, but that was the team that lost to Weber State in the first round. The only other ACC team to win 20 was Clemson and they were 5-11 in the ACC. The fourth-place team in the league standings was Wake at 7-9 -- and they had a losing record overall.

I thought Duke lost due to the way the game was officiated. That is another variable to add to the mix when trying to guess the probability of winning. If you are allowed to play aggressive defense without worrying about fouling out key players, the results can be skewed.

CDu
04-17-2015, 12:31 PM
Just to build on this - the seeding, itself, is a "model" of sorts. The selection committee selects teams 1-68* and then seeds them in ranked order. No one says the selection committee was "wrong" just because UConn wins the title as a #7 seed, or because Kentucky doesn't win the title as the overall #1 seed. But everybody runs off and says a statistical model is "wrong" when the same thing happens. This is silly.

*of the qualifiers, since obviously several AQ teams are not in the top 68

I can't pitchfork you either, but this is a very nice and simple way of saying it. Upsets happen. Sometimes, the committee seeds improperly, but more often than not the seeds are pretty close to what people and models would have predicted. And just because an upset happened doesn't necessarily mean the seed was wrong. Upsets happen. And if the seeding is appropriate, no model is going to predict something other than "chalk" with less than 50% probability. Otherwise, it wouldn't be an upset. So the model is inherently going to pick probabilities that are most in line with the seeds (which are, as you said, basically a model in themselves).

Basically, the models weren't saying Duke was a bad team; just that Duke got a raw deal in getting Utah (a top-10 team by most models) as a Sweet-16 team rather than an elite-8 team. Most models predicted that Duke was a toss-up with Utah and Gonzaga. So instead of playing 3 toss-up games (which would have suggested closer to a 10-12% probability of winning, we had 4 toss-up games. And that ignores any risk of losing the first two games. So it doesn't seem all that unreasonable to me to say Duke had a ~6% chance of winning the title. 50%*50%*50%*50% = 6.25%, and that ignores the probability of losing in the first weekend and the idea that UK was supposedly better than the other #1s.

sagegrouse
04-17-2015, 12:42 PM
At the start of the tournament, Duke came in:
ranked #4 in the country
1 loss since January
road wins @Wisconsin, Louisville (full roster for the Cards), Virginia (full roster), Syracuse, and UNC.
with no major injuries

So, a 6% chance of winning? Not in my opinion, given those facts

To each his own, but I don't see much predictive value from 538 here, and even kenpom has it's limitations.

Hi, Roy, broken record here. Statistical models have their uses -- God knows, I made a living doing that stuff for more than a decade.

This was not a good year to evaluate college hoops teams using only game results (and their various intra-game stats and measures). And it really doesn't matter very much which methods are used or how much the model weights recent results:

a. The only way to align conferences is to use data all the way back in November and December -- and it really doesn't matter whether you weight those games as 1.0x or 0.1x -- It's all the data there is that enable one to line up the teams BETWEEN conferences.

b. The Big 12 is way over-rated given those results. There weren't many inter-conference games, but the Big 12 teams did win a few.

c. The ACC was way under-rated. There were a few more games (thanks to the Big Ten-ACC match-ups), but the ACC teams early on seemed to stub their toes, losing the series to the decidedly weaker Big Ten and having some other embarrassing losses -- by Miami, State, and Notre Dame. Thus, using the data from November and December to the hilt, the five power teams in the ACC -- Duke, UNC, Louisville, Notre Dame and Virginia -- may have seemed weaker than they actually were.

d. The evidence for the ACC is how well the conference did in the NCAA's: an unprecedented five teams in the Sweet Sixteen -- only Virginia lost, which was to a strong Michigan State team that should have been a four seed rather than a seven seed. And Louisville went to OT with Michigan State in the Elite Eight.

e. I thought Duke was potentially the strongest team in the field, assuming we could duck the freshman-itis that seemed to affect us in some of our losses. Our road record was spectacular, and winning tough games on the road is a good measure of strength. There were other reasons, and I don't truly trust my judgment when I care so much about the outcome.

Kfanarmy
04-17-2015, 12:53 PM
perhaps we could theorize a better mathematical model. I'm not sure Pythagorus would base an analysis of something so complex on a two dimensional three sided shape, and I'm confident Vegas is more concerned with establishing betting lines that make them money regardless of the actual outcome. ehh but maybe I don't want to know who is going to win...

Mtn.Devil.91.92.01.10.15
04-17-2015, 12:58 PM
13%

How do I get there? The percentage should be bounded by a minimum of 1.5% (assuming all 68 teams are of similar strength with equal likelihood of winning) and a maximum of 25% (4 teams that are equally dominant, but significantly better than the other 60). 13% falls in the middle.

A model that gave Duke a 6% chance of winning this year does not strike me as low considering Kentucky's perceived dominance (41% likelihood of winning).

This is pretty sensible analysis. I don't know about picking 13% but your assumption of somewhere between 25% and 1.5% makes plenty of sense. Skewing due to UK's favoritism leaves us somewhere on the lower end, and the matchup analysis with Utah pushes it down further.

6% isn't all that absurd or disrespectful.

johnb
04-17-2015, 01:21 PM
538 was awesome at predicting elections.

Predicting sporting events is a different beast, whether it's predicting who will win the whole tournament or what the odds are that a team will win during the course of a game.

Richard Berg
04-17-2015, 01:22 PM
Could it also be that statistical predictions, though they are often highly astute and have some strong predictive abilities, are losing their ability to confidently predict college basketball results?
No, I think they're getting better. Stats are Darwinian, almost by definition -- models that don't do as well get revised or discarded.

Of course, part of getting "better" is having a more accurate picture of their own uncertainty. Greater parity means more games will be statistical toss-ups, and even games where one side is strongly favored will have a wider error bars. (Maybe that's what you meant by "predictions losing confidence"?)

As for 2015 in particular, I felt like this tourney had an unusually low number of upsets. Especially if you go by dork polls (where Sparty & Utah were stronger than their seeds, Nova was weaker than two #2s, and so on), there weren't any significant Cinderella stories after the round of 64.

The chance that the good teams would all perform as well as they "should" was still very low, of course, but that's the nature of joint probability. It doesn't say anything about the model, really; any model is going to produce low %s once you start multiplying the individual events together.

DukieinSoCal
04-17-2015, 01:30 PM
Does anyone have a link to the in-game odds of winning as the game progressed? I'm curious to know how low our odds were when we were down 9 in the 2nd half. I wonder if the percentages factor in foul trouble as well, although I would guess not. I bet our odds of winning the title game bottomed out around 5% or less.

Duke95
04-17-2015, 01:44 PM
As a statistician, I can appreciate the difficulty of creating such a model. I do think that Duke's probability of winning the tournament at 6% is a bit low. It does appear, based on examining each step, that a lot of that is, as has been pointed out, due to the Utah game. Our chances of continuing to play dropped considerably because of that SS game.

It would be interesting, at least for me, to see some confidence bounds around those probability estimates.

wilson
04-17-2015, 01:51 PM
...Of course, part of getting "better" is having a more accurate picture of their own uncertainty. Greater parity means more games will be statistical toss-ups, and even games where one side is strongly favored will have a wider error bars. (Maybe that's what you meant by "predictions losing confidence"?)This is precisely what I was getting at, but you've said it more adeptly, with a stronger understanding of statistical modeling than I have. Thanks for your response.
Again, some really nice insights and commentary from you math nerds, I mean people, in this thread.

Duke95
04-17-2015, 02:07 PM
No, I think they're getting better. Stats are Darwinian, almost by definition -- models that don't do as well get revised or discarded.

Of course, part of getting "better" is having a more accurate picture of their own uncertainty. Greater parity means more games will be statistical toss-ups, and even games where one side is strongly favored will have a wider error bars. (Maybe that's what you meant by "predictions losing confidence"?)

As for 2015 in particular, I felt like this tourney had an unusually low number of upsets. Especially if you go by dork polls (where Sparty & Utah were stronger than their seeds, Nova was weaker than two #2s, and so on), there weren't any significant Cinderella stories after the round of 64.

The chance that the good teams would all perform as well as they "should" was still very low, of course, but that's the nature of joint probability. It doesn't say anything about the model, really; any model is going to produce low %s once you start multiplying the individual events together.

Interesting that the 538 model did not agree with these "dork" polls. Villanova was given a higher chance to win than Duke.
MSU was given basically no chance.

Where I do think the models were perhaps deficient, and maybe through no fault of their own, is that I don't think they did (or could) pick up the effects of Anderson at UVa or Jones at Louisville.
There are a lot of qualitative factors that simply do not translate well into quantitative measurements.

CDu
04-17-2015, 02:14 PM
Interesting that the 538 model did not agree with these "dork" polls. Villanova was given a higher chance to win than Duke.
MSU was given basically no chance.

Where I do think the models were perhaps deficient, and maybe through no fault of their own, is that I don't think they did (or could) pick up the effects of Anderson at UVa or Jones at Louisville.
There are a lot of qualitative factors that simply do not translate well into quantitative measurements.

To be fair, I think MSU was dead in the water had they faced any of Duke, Kentucky, Wisconsin, Arizona, etc. So their chances of winning the title probably really were next to nothing. They happened to get a good draw in that they were in the one bracket with multiple "overseeded" teams (Villanova, UVa due to Anderson's injury, Louisville due to the Jones dismissal). Villanova lost early, which opened up the top of the bracket for an overseeded Louisville and UVa wasn't the same UVa that built such a strong early-season resume. So games in which MSU should have been substantial underdogs (UVa, a "true" #1 seed in the Elite-8) were essentially toss-up games. And MSU won the coin toss in those games. Beating Oklahoma was the most impressive thing they did in the tournament, and even that wasn't much of an upset.

So in terms of title hopes, MSU's chances were next to nill. But in terms of Final Four hopes, they were probably not unreasonable (I didn't pick them, but it wasn't completely outlandish to do so).

mr. synellinden
04-17-2015, 02:19 PM
Does anyone have a link to the in-game odds of winning as the game progressed? I'm curious to know how low our odds were when we were down 9 in the 2nd half. I wonder if the percentages factor in foul trouble as well, although I would guess not. I bet our odds of winning the title game bottomed out around 5% or less.

I remember reading once that a team's chances of winning are about 80% if they can get a lead that equals the number of minutes remaining. I'm sure odds go up or down with some mathematical formula based on how much more or less than a "minutes-left" lead you have. If you assume their chances of winning would have been 50% with a tie game and 80% with a 13 point lead, then with a 9 point lead and 13 minutes left, I would guess Wisconsin's chances of winning were approximately 65-70% and nowhere near 95%.

gcashwell
04-17-2015, 02:21 PM
Does anyone have a link to the in-game odds of winning as the game progressed? I'm curious to know how low our odds were when we were down 9 in the 2nd half. I wonder if the percentages factor in foul trouble as well, although I would guess not. I bet our odds of winning the title game bottomed out around 5% or less.

About 11% at worst: http://blogs.wsj.com/dailyfix/2015/04/06/wisconsin-vs-duke-ncaa-live-blog/

Kedsy
04-17-2015, 02:23 PM
I remember reading once that a team's chances of winning are about 80% if they can get a lead that equals the number of minutes remaining.

I remember hearing that sort of estimate before the 3-point shot. My guess is it's a lot less than that now. My guess is to get up to 80% or 90% you'd need to have 1.5x or even 2x points per number of minutes left. But that's just a guess.


About 11% at worst: http://blogs.wsj.com/dailyfix/2015/04/06/wisconsin-vs-duke-ncaa-live-blog/

But after looking at that, maybe my guess above is wrong. Very surprising to me that a team with a 9-point lead and 13 minutes to go would win more than 88% of the time. I mean, a three, a stop, and a three and it's a three-point game with 12 minutes to play and basically a toss up.

tbyers11
04-17-2015, 02:30 PM
Does anyone have a link to the in-game odds of winning as the game progressed? I'm curious to know how low our odds were when we were down 9 in the 2nd half. I wonder if the percentages factor in foul trouble as well, although I would guess not. I bet our odds of winning the title game bottomed out around 5% or less.

KenPom was a win probability graph for the game, but it's behind his paywall. Because of that I won't post it here but here is some key info. Our lowest probability to win was 13.7% at the 13:25 mark of the 2nd half when were down 48-39. Our win % at the start of the game was 45.1%.

Other key points and big plays which swung the momentum:
-Grayson's personal 8 point barrage in 90 secs brought our WP back to about 30%
-Tyus 3 to make 59-58 at the 4 min mark moved our WP from 35% to 52%
-Our WP was at least 75% from the Jah and 1 where Kaminsky grabbed him that made it 61-58

Our most unlikely pulling-out-of-a-game all season was the UVa game. Our WP% was less than 10% for nearly all of the 2nd half from 15:00 to 3:00. It was about 4% when Matt hit the 3 to make it 63-61 UVa at the 2:47 mark. It got better quickly after that :D

CDu
04-17-2015, 02:39 PM
But after looking at that, maybe my guess above is wrong. Very surprising to me that a team with a 9-point lead and 13 minutes to go would win more than 88% of the time. I mean, a three, a stop, and a three and it's a three-point game with 12 minutes to play and basically a toss up.

Obviously a lot of it has to do with (a) the pace of play and (b) who has possession when the lead in question is achieved.

A 9-point lead in a game between UNLV and Duke cerca 1991? Not so significant. A 9-point lead in a game between Wisconsin and UVa last year? Huge.

Still, I agree that a 9-point lead with 13 minutes to go doesn't SEEM like it should lead to an 88% likelihood of victory.

That said, the fact that we came back and made it a one-possession game in literally the least amount of time possible was very improbable. I would guess the odds of us getting back-to-back 3pt plays and them not scoring would be around 0.5-2% (~10-20% chance of scoring 3 points on any single possession for us; ~50% chance of Wisconsin scoring on any possession). So while it's true that "a 3, a stop, and a 3" gets us to within one possession, that scenario was a really unlikely scenario. And even in that scenario, it's a one-possession game with the leading team having possession and a roughly 50% chance of making it a multiple-possession game again. So we still would be less than a 50% chance of winning, even in that extremely unlikely scenario (which happened to play out for us).

So maybe 12% isn't as unreasonable as we both think.

Bostondevil
04-17-2015, 02:51 PM
At the start of the tournament, Duke came in:
ranked #4 in the country
1 loss since January
road wins @Wisconsin, Louisville (full roster for the Cards), Virginia (full roster), Syracuse, and UNC.
with no major injuries

So, a 6% chance of winning? Not in my opinion, given those facts

To each his own, but I don't see much predictive value from 538 here, and even kenpom has it's limitations.

And all of those things were factored into the statistical model. If all teams are equal, each team has 1/64 chance of winning. Anything above that is added on because of things like quality wins, major injuries would be subtracted.

It's not like the model said they had no chance of winning.

We can look back after Duke wins and say, wow, the team that ended up winning was only given a 6% chance to do so! They were wrong! But they weren't wrong. They gave Duke a better chance of winning than all but a few teams. Most of those teams had quality wins and no major injuries too.

Just out of curiosity, if 538 had a model that said heads has a 50% chance of winning a coin toss and tails came up, would you say their model has no predictive value?

Kedsy
04-17-2015, 02:57 PM
So maybe 12% isn't as unreasonable as we both think.

Yeah, I guess, for all the reasons you state. Especially in light of the following:


Grayson's personal 8 point barrage in 90 secs brought our WP back to about 30%

So being down 4 with almost 12 minutes to play still only gave us a 30% chance of winning? That's just amazing to me.

bedeviled
04-17-2015, 02:58 PM
It always makes me sad that we have so many ranking systems but nobody publishes data on how good they actually are.tbyers11 (http://forums.dukebasketballreport.com/forums/showthread.php?35914-Statistically-Duke-s-championship-was-very-unlikely-538-blog&p=802643#post802643) replied with a great link. You may also be interested in an earlier post of mine (http://forums.dukebasketballreport.com/forums/showthread.php?34409-The-Dork-Polls-2015&p=790997#post790997).


At some point though the user has to understand the utility of an individual model: does it do what is is intended to do. Given a season's worth of data, what should the performance standard for a model intended to predict the winner of the NCAA tournament be?

It seems like I often see things differently than others, so, in case it helps, I'll take the time to attempt an overly wordy answer/viewpoint to your question. I could be horribly wrong, but here's what I think:

First, there is a concept of primary importance which must be discussed. In my understanding, the given percentages (eg Kentucky has a 99.999% chance of winning this game) are NOT truly predictions of Team A's chance to win the game. Rather, they are the probability that the particular model accurately predicts which team will win the game. Those are certainly related concepts, and the media and even the model makers sometimes equate the two. But, they are not equivalent. (More on this later)

How are probabilities of winning even calculated?? IDK, but here's a guess based on things I've read.
Here's a simplified path of how I *think* the probabilities are established:
1. Calculate/assign each team a rating/ranking
2. Predict the winner of the game. I suspect that, for most models, this is the same as "Predict that the team with the higher rating/ranking wins the game." However, some models could include additional tricks.
3. Open up your system's modeled or historical data. Look at the data for which the teams had similar ratings/rankings to the teams in the current game. For what percentage of those games did this model accurately predict the winner?

This step is likely decently complex. But, for illustrative purposes, here is a quick graph of one way the modeled/historical data could look. For this basic predictive model, the winner is predicted to be the team with the higher rank and the independent variable is the difference between teams' ranks. Let's say Team A is ranked #6 and Team B is ranked #36 (a difference of 30 units). We look at the compiled data to see that our model is right only about 75% of the time when we say that the higher ranked team will win.
5036

4. Tell everyone, "In modeling/historical data similar to Team A's and Team B's rating/ranking, we are correct 75% of the time when predicting that Team A is the winner." Or, "According to our model, team A has an 75% chance of winning." Or, just "Team A has an 75% chance of winning."

How are a model's predictions validated?
This gets more to your question about what the performance standards for a model should be. But, before talking about the tournament, let's consider the full season. How do these predictive models evaluate their "Percent chance of winning" performance over the season. From what I can piece together, the evaluation is along the lines of "with regard to the games the model predicted as having a specific probability, what was the model's actual accuracy in determining the winner?"

For example, we could look at all the games for which we predicted the favored team had a 50-55% "chance of winning" (In the graph above, this is essentially the same as looking at all the games in which the difference in teams' rankings was something like between 1 and 13 units - because those are the games in which we gave the favored team a 50-55% chance based on our modeled or historical data). So, how frequently did our model determine the correct winner? If it determined correctly somewhere around 50-55%, we would conclude that our prediction is valid.

Another example, in games we say we can predict the winner with 80-85% certainty, we should expect that we accurately determined the winner 80-85% of the time. Sooooo, the models are not faulty for only predicting the winner 80-85% of the time in those games. Indeed, the models have done exactly what they predicted!...just not what we wanted them to do :)

What should the performance standard be for predicting the winner of the tournament?
Finally, on to your question:
the utility of an individual model: does it do what is is intended to do. Given a season's worth of data, what should the performance standard for a model intended to predict the winner of the NCAA tournament be?

Well, these models were NOT built to predict the winner of the NCAA tournament. As mentioned above, they are indeed doing what they are intended to do....just not what we want them to do or what is being pushed upon them. The models predict individual games, not the tournament, and not the Champion. The probability of compound events (each individual game) is used to predict who the champion will be. Thus, the test to see if the model does what it is intended to do should not be how frequently the model predicts the Champion, but, rather, how well the model predicts individual games. It does not necessarily matter that 2014 UConn kept winning its way to a Championship despite having, say, a "15% chance" each game. What matters to the model is if, in all the games that the model gave the higher ranked team an 85% chance of winninng, did the higher ranked team win 85% of the time?

I *think* the models are doing what they are designed to do and are probably meeting that performance standard - they do not treat the Championship game as distinct from all the other games with similar opponents. Could a model be designed specifically for the tournament? IDK. Would it have to take into account the specific rounds of the tournament? Maybe not. Maybe the current models work fine for the tournament (ie even in the tournament, they do what they say they are capable of doing) but just need refinement to help increase their capability for tournament-type of games. For instance, I would think a tournament model would have to be based on how teams play against Top 50 or so opponents rather than comparing teams based on how well they would do against the NCAA average opponent. I mean, shouldn't a tournament model attempt to tease apart what separates a #2 team from a #12 team, rather than declaring the game a toss-up? As it is, the model IS correct in that the model is saying, "I can't predict who will win this game," and, sure enough, it does a bad job of predicting such a game, lol. :D

That's the rub. These aren't really predictions about a team's chances of winning. They are predictions about how well the model can predict who is going to win!!

My severely extreme analogy:
In reality, Duke has a 100% chance of beating East Chapel Hill High School and Kentucky has a 100% chance of beating Jumbo's Allstars (that's our DBR team!)
However, a model (we'll call it 'Mopnek') uses the following criteria to predict winners: there is a 50% likelihood of a team beating an opponent whose name starts with the next letter of the alphabet.

When the games D vs E and K vs J are played out, the winners are (D)uke and (K)entucky.
Mopnek predicted the winner 50% of the time, just as Mopnek said it would!
The fact that the Mopnek prediction was equal to the outcome in the sample is used to validate the system - the system predicts as accurately as it says it predicts.
BUT, that does not mean that a specific team's chances against another team are the same as the chances of the system predicting that game correctly.

The real meaning of that 50% prediction is "In those games, the model has a 50% chance of accurate prediction when choosing its team." It does NOT mean that the team actually has a 50% chance of winning. Put another way, a game predicted by Mopnek as a toss-up does not mean that the game could go either way, it just means that Penkom doesn't know which way the game will go.

Does it matter? Are there cases in which a team actually has a good chance to win a game but the models don't know that (ie declare it a toss-up game)?
The misinterpretations in the crazy analogy probably apply to real world scenarios, too. I agree with Wander in saying that Utah was overrated (because I desperately want to use KenPom to tell me who can beat whom :p). In the Dork Polls thread, I tried to complain that KenPom wasn't good at predicting "who is the best team" the way that I view "best team."
http://forums.dukebasketballreport.com/forums/showthread.php?34409-The-Dork-Polls-2015&p=784309#post784309

Before Duke's 2nd win over UNC and Utah's loss to Washington, KenPom had Utah ranked #6 and Duke ranked #8. Yet, here were their average unadjusted efficiency margins versus the top KenPom teams (efficiency margin is offensive efficiency minus defensive efficiency....like, do you score more points than your opponent).



Avg Per Game Efficiency Margin Against Top Teams in Kenpom Ratings



vs Top 10
vs Top 25
vs Top 50
vs Top 100
vs Top 150
vs Top 200
All Games


UTAH
-13.276

-11.167

1.768
9.047
15.823
18.913
25.917


DUKE
13.369
15.211
11.975
15.876
16.058
15.709
22.523



I actually held off on posting that data at the time, in part because I feared Utah would prove me wrong (and 'cause the story was more complicated than this chart, with blowouts, recency effects, etc). Well, it turns out that we *played* Utah and beat them. Now, I look at that chart and am certain who I would pick in a battle between two Top 10 teams! :)

The rating of Utah (and Texas) made me consider that, while KenPom may do a good job of rating which teams are good according to certain criteria, it might not do the best job at deciding which top teams will beat other top teams. Most the time we don't notice this because
1. Good teams, according to many different criteria, tend to win
2. In games between two good teams, the predictions state that the game could go either way. So, the model looks correct when winning or losing.
Actually, in reality, the models ARE correct, we are just misinterpreting them. Sure enough, the models aren't good at predicting the games they say they aren't good at predicting (ie they predict the winner only 55% of the time in games where the model believes it has a 55% chance of predicting the winner).

Again, it does not mean that the team actually had a 55% chance of winning. Notably, Duke was 11-2 vs KenPom Top 25 teams (final ratings). Wisconsin was 10-2. Maybe it's just "luck" that KenPom can't predict their wins. Or, maybe, just maybe, there actually is an uncaptured something about certain teams that make them winners.

Anyway, I'm really, really sorry for the long post. And, again, I could totally be wrong, but that's how I see the "Chance of winning" topic.

pfrduke
04-17-2015, 03:10 PM
Yeah, I guess, for all the reasons you state. Especially in light of the following:



So being down 4 with almost 12 minutes to play still only gave us a 30% chance of winning? That's just amazing to me.

You've got to go from the baseline, though. If, with 40 minutes left to play and tied, we had a 45% chance of winning, we're necessarily going to be lower than 45% when we're down 4 with 12 minutes to play. Wisconsin has spent 28 minutes of game time improving its initial position. So, basically, the conclusion is that a team that was favored 55/45 at the tip will be favored 70/30 when it's up by 4 with 12 to play. That doesn't seem too unreasonable to me.

Kedsy
04-17-2015, 03:10 PM
Anyway, I'm really, really sorry for the long post. And, again, I could totally be wrong, but that's how I see the "Chance of winning" topic.

Don't be sorry. This is fabulous stuff.

Mtn.Devil.91.92.01.10.15
04-17-2015, 03:13 PM
Yeah, I guess, for all the reasons you state. Especially in light of the following:



So being down 4 with almost 12 minutes to play still only gave us a 30% chance of winning? That's just amazing to me.

Well, if you figured we started out the game at 45% chance, it stands to reason that regardless of the momentum, being behind would further decrease the chances of victory...

Oh, or what pfrduke said just before I posted.

Mtn.Devil.91.92.01.10.15
04-17-2015, 03:14 PM
TONS of good analysis excerpted

Wow - I can't spork you, but thanks for all this.

Just

wow.

COYS
04-17-2015, 03:17 PM
tbyers11 (http://forums.dukebasketballreport.com/forums/showthread.php?35914-Statistically-Duke-s-championship-was-very-unlikely-538-blog&p=802643#post802643) replied with a great link. You may also be interested in an earlier post of mine (http://forums.dukebasketballreport.com/forums/showthread.php?34409-The-Dork-Polls-2015&p=790997#post790997).



It seems like I often see things differently than others, so, in case it helps, I'll take the time to attempt an overly wordy answer/viewpoint to your question. I could be horribly wrong, but here's what I think:

First, there is a concept of primary importance which must be discussed. In my understanding, the given percentages (eg Kentucky has a 99.999% chance of winning this game) are NOT truly predictions of Team A's chance to win the game. Rather, they are the probability that the particular model accurately predicts which team will win the game. Those are certainly related concepts, and the media and even the model makers sometimes equate the two. But, they are not equivalent. (More on this later)

How are probabilities of winning even calculated?? IDK, but here's a guess based on things I've read.
Here's a simplified path of how I *think* the probabilities are established:
1. Calculate/assign each team a rating/ranking
2. Predict the winner of the game. I suspect that, for most models, this is the same as "Predict that the team with the higher rating/ranking wins the game." However, some models could include additional tricks.
3. Open up your system's modeled or historical data. Look at the data for which the teams had similar ratings/rankings to the teams in the current game. For what percentage of those games did this model accurately predict the winner?

This step is likely decently complex. But, for illustrative purposes, here is a quick graph of one way the modeled/historical data could look. For this basic predictive model, the winner is predicted to be the team with the higher rank and the independent variable is the difference between teams' ranks. Let's say Team A is ranked #6 and Team B is ranked #36 (a difference of 30 units). We look at the compiled data to see that our model is right only about 75% of the time when we say that the higher ranked team will win.
5036

4. Tell everyone, "In modeling/historical data similar to Team A's and Team B's rating/ranking, we are correct 75% of the time when predicting that Team A is the winner." Or, "According to our model, team A has an 75% chance of winning." Or, just "Team A has an 75% chance of winning."

How are a model's predictions validated?
This gets more to your question about what the performance standards for a model should be. But, before talking about the tournament, let's consider the full season. How do these predictive models evaluate their "Percent chance of winning" performance over the season. From what I can piece together, the evaluation is along the lines of "with regard to the games the model predicted as having a specific probability, what was the model's actual accuracy in determining the winner?"

For example, we could look at all the games for which we predicted the favored team had a 50-55% "chance of winning" (In the graph above, this is essentially the same as looking at all the games in which the difference in teams' rankings was something like between 1 and 13 units - because those are the games in which we gave the favored team a 50-55% chance based on our modeled or historical data). So, how frequently did our model determine the correct winner? If it determined correctly somewhere around 50-55%, we would conclude that our prediction is valid.

Another example, in games we say we can predict the winner with 80-85% certainty, we should expect that we accurately determined the winner 80-85% of the time. Sooooo, the models are not faulty for only predicting the winner 80-85% of the time in those games. Indeed, the models have done exactly what they predicted!...just not what we wanted them to do :)

What should the performance standard be for predicting the winner of the tournament?
Finally, on to your question:

Well, these models were NOT built to predict the winner of the NCAA tournament. As mentioned above, they are indeed doing what they are intended to do....just not what we want them to do or what is being pushed upon them. The models predict individual games, not the tournament, and not the Champion. The probability of compound events (each individual game) is used to predict who the champion will be. Thus, the test to see if the model does what it is intended to do should not be how frequently the model predicts the Champion, but, rather, how well the model predicts individual games. It does not necessarily matter that 2014 UConn kept winning its way to a Championship despite having, say, a "15% chance" each game. What matters to the model is if, in all the games that the model gave the higher ranked team an 85% chance of winninng, did the higher ranked team win 85% of the time?

I *think* the models are doing what they are designed to do and are probably meeting that performance standard - they do not treat the Championship game as distinct from all the other games with similar opponents. Could a model be designed specifically for the tournament? IDK. Would it have to take into account the specific rounds of the tournament? Maybe not. Maybe the current models work fine for the tournament (ie even in the tournament, they do what they say they are capable of doing) but just need refinement to help increase their capability for tournament-type of games. For instance, I would think a tournament model would have to be based on how teams play against Top 50 or so opponents rather than comparing teams based on how well they would do against the NCAA average opponent. I mean, shouldn't a tournament model attempt to tease apart what separates a #2 team from a #12 team, rather than declaring the game a toss-up? As it is, the model IS correct in that the model is saying, "I can't predict who will win this game," and, sure enough, it does a bad job of predicting such a game, lol. :D

That's the rub. These aren't really predictions about a team's chances of winning. They are predictions about how well the model can predict who is going to win!!

My severely extreme analogy:
In reality, Duke has a 100% chance of beating East Chapel Hill High School and Kentucky has a 100% chance of beating Jumbo's Allstars (that's our DBR team!)
However, a model (we'll call it 'Mopnek') uses the following criteria to predict winners: there is a 50% likelihood of a team beating an opponent whose name starts with the next letter of the alphabet.

When the games D vs E and K vs J are played out, the winners are (D)uke and (K)entucky.
Mopnek predicted the winner 50% of the time, just as Mopnek said it would!
The fact that the Mopnek prediction was equal to the outcome in the sample is used to validate the system - the system predicts as accurately as it says it predicts.
BUT, that does not mean that a specific team's chances against another team are the same as the chances of the system predicting that game correctly.

The real meaning of that 50% prediction is "In those games, the model has a 50% chance of accurate prediction when choosing its team." It does NOT mean that the team actually has a 50% chance of winning. Put another way, a game predicted by Mopnek as a toss-up does not mean that the game could go either way, it just means that Penkom doesn't know which way the game will go.

Does it matter? Are there cases in which a team actually has a good chance to win a game but the models don't know that (ie declare it a toss-up game)?
The misinterpretations in the crazy analogy probably apply to real world scenarios, too. I agree with Wander in saying that Utah was overrated (because I desperately want to use KenPom to tell me who can beat whom :p). In the Dork Polls thread, I tried to complain that KenPom wasn't good at predicting "who is the best team" the way that I view "best team."
http://forums.dukebasketballreport.com/forums/showthread.php?34409-The-Dork-Polls-2015&p=784309#post784309

Before Duke's 2nd win over UNC and Utah's loss to Washington, KenPom had Utah ranked #6 and Duke ranked #8. Yet, here were their average unadjusted efficiency margins versus the top KenPom teams (efficiency margin is offensive efficiency minus defensive efficiency....like, do you score more points than your opponent).



Avg Per Game Efficiency Margin Against Top Teams in Kenpom Ratings



vs Top 10
vs Top 25
vs Top 50
vs Top 100
vs Top 150
vs Top 200
All Games


UTAH
-13.276

-11.167

1.768
9.047
15.823
18.913
25.917


DUKE
13.369
15.211
11.975
15.876
16.058
15.709
22.523



I actually held off on posting that data at the time, in part because I feared Utah would prove me wrong (and 'cause the story was more complicated than this chart, with blowouts, recency effects, etc). Well, it turns out that we *played* Utah and beat them. Now, I look at that chart and am certain who I would pick in a battle between two Top 10 teams! :)

The rating of Utah (and Texas) made me consider that, while KenPom may do a good job of rating which teams are good according to certain criteria, it might not do the best job at deciding which top teams will beat other top teams. Most the time we don't notice this because
1. Good teams, according to many different criteria, tend to win
2. In games between two good teams, the predictions state that the game could go either way. So, the model looks correct when winning or losing.
Actually, in reality, the models ARE correct, we are just misinterpreting them. Sure enough, the models aren't good at predicting the games they say they aren't good at predicting (ie they predict the winner only 55% of the time in games where the model believes it has a 55% chance of predicting the winner).

Again, it does not mean that the team actually had a 55% chance of winning. Notably, Duke was 11-2 vs KenPom Top 25 teams (final ratings). Wisconsin was 10-2. Maybe it's just "luck" that KenPom can't predict their wins. Or, maybe, just maybe, there actually is an uncaptured something about certain teams that make them winners.

Anyway, I'm really, really sorry for the long post. And, again, I could totally be wrong, but that's how I see the "Chance of winning" topic.

Wish I could spork this post, but thanks for this lengthy analysis. You bring up a point that I have wondered about, and that is that the models perhaps have the most room for improvement in picking the winner between two SPECIFIC teams . . . particularly two top 10 opponents. This would require quite a bit more data mining and, given how short the college basketball season is and how unbalanced schedules are across conferences, it might prove to be impossible. However, perhaps the model could look at things such as the style of play, the effective height, or the measured athleticism of the teams against which one top team either consistently over-performed or under-performed against. I know John Hollinger put together Steal %, FT Rate, and Rebound % as athletic "marker" stats for projecting how a player would fare after making the jump to the pros. In addition, other stats like the turnover percentage of opposing guards or the usage rate for post players might shed some light on an individual matchup.

Maybe it would show that Wisconsin struggled a bit more against teams capable of getting to the line consistently, particularly guards like Tyus who have strong free throw rates. This might indicate that they struggle against quick guards and could have lent some insight on how they could lose to Duke twice.

I know that some models already try to do this, especially at the pro level. But I haven't seen any details as to how advanced models like 538 and Kenpom are when predicting specific matchups.

On the other hand, it could be that the models are already just about as good as they can be and Duke had the unlikely pleasure of beating a slightly "better" Badger team twice this year.

freshmanjs
04-17-2015, 03:17 PM
tbyers11 (http://forums.dukebasketballreport.com/forums/showthread.php?35914-Statistically-Duke-s-championship-was-very-unlikely-538-blog&p=802643#post802643) replied with a great link. You may also be interested in an earlier post of mine (http://forums.dukebasketballreport.com/forums/showthread.php?34409-The-Dork-Polls-2015&p=790997#post790997).



It seems like I often see things differently than others, so, in case it helps, I'll take the time to attempt an overly wordy answer/viewpoint to your question. I could be horribly wrong, but here's what I think:

i don't think there is any mathematical difference between

A. team x has a 55% chance of beating team y
B. team x will beat team y and that statement has a 55% chance of being correct

CDu
04-17-2015, 03:19 PM
bedeviled, cool post. One question: why does the "all games" column look so weird in the efficiency margin data for Utah and Duke? Shouldn't it be a weighted average of the other categorical results rather than greater than any of those other results?

Still, your table (assuming it is otherwise accurate) shows something interesting and potentially important. And it gets back to another point others have hinted at. The thing with Duke this year is that, on the season, we had a tendency to lose our focus. As such, we rarely pummeled the teams we were expected to pummel, but we also rarely lost to top teams. In fact, we lost only twice to a team in the top-25 (Notre Dame twice), and both were potentially examples of us taking them too lightly (we blew them out when they had our full attention in Cameron). So it is quite possible that these models (which have to rely on a season's worth of data) actually did slightly underrate us, because they were based on a season's worth of data in which we didn't always play at or near our best.

Conversely, Utah appeared to be good at beating up on the teams they should beat up on, but struggled against top teams. So they may have outkicked their coverage a bit in getting a top-10 rating by virtue of taking every game (and every possession) more seriously than most. So whereas some of the teams in the 10-20 range may have slipped more often and played down to their competition, Utah may not have.

If both of the above are true, then models based on average performance of the course of a season are going to naturally consider that matchup more of a toss-up than perhaps it should be. So it may be that we really should have been a ~10% chance of winning. Or more. Or it could be that the model is assuming we might (like we did during the regular season and even during the ACC tournament) fall asleep at the wheel again thinking we could coast, and get upset.

Of course, none of this changes the overall point that none of these models are designed to accurately predict the champion. They are designed to, with a certain level of accuracy given the particular game considered, estimate the probability of correctly predicting the winner. So the best one can do is to compare the probability of correctly predicting the winner amongst games with a similar probability favorite/underdog.

In other words, they really aren't at all suited to predict the tournament champion, because all of them are going to predict that the #1 team to have the best chance, and with almost certainty even that team will have much less than a 50% chance of winning the title.

CDu
04-17-2015, 03:28 PM
You've got to go from the baseline, though. If, with 40 minutes left to play and tied, we had a 45% chance of winning, we're necessarily going to be lower than 45% when we're down 4 with 12 minutes to play. Wisconsin has spent 28 minutes of game time improving its initial position. So, basically, the conclusion is that a team that was favored 55/45 at the tip will be favored 70/30 when it's up by 4 with 12 to play. That doesn't seem too unreasonable to me.

That's an excellent point. These models are taking into account not just the time and score but also the perceived difference in quality of teams. Had we not been a 45% probability of winning, the probability of coming back from down 9 would have been slightly higher (though not much - the difference between 45% and 50% at baseline is small). So still a 30% chance of winning down 4 with 12 minutes to go given a 45% starting point is around a 34% chance of winning if we assume a coin-flip game at tipoff, which seems low.

I guess the argument is that it requires, at some point, a score/stop/score combo to win, which is unlikely even given a lot of opportunities. In a 50/50 game, the probability that you'll go two full possessions better over a span of roughly 24 possessions must be lower than I would have thought.

Kedsy
04-17-2015, 03:30 PM
bedeviled, cool post. One question: why does the "all games" column look so weird in the efficiency margin data for Utah and Duke? Shouldn't it be a weighted average of the other categorical results rather than greater than any of those other results?

I thought the same thing when I read his post, but then I realized the 200+ category isn't in his table. Both Duke and Utah had a big efficiency margin against sub-200 teams.

Also, we only beat Utah by 6, so it's not like the ratings were crazy wrong about our chances against them.

CDu
04-17-2015, 03:36 PM
I thought the same thing when I read his post, but then I realized the 200+ category isn't in his table. Both Duke and Utah had a big efficiency margin against sub-200 teams.

Also, we only beat Utah by 6, so it's not like the ratings were crazy wrong about our chances against them.

Oh, yeah, I totally forgot about 200+ games. Oops! Still, maybe "All Other Games" is what the last column means? No way our and Utah's efficiency margins for all games was 20+ given our results in the other categories.

Though I feel compelled to note that, for the same reasons the model isn't intended to predict the champion, saying that the 6-point game suggests the model wasn't crazy wrong isn't accurate. Pretty much outside of the model saying there was a zero percent chance of a 6-point Duke win, the game results are going to indicate the model wasn't crazy wrong. (okay, done with the nitpicking!)

The main point is that the post by bedeviled was a REALLY good one. Lots of interesting stuff to discuss in there.

bedeviled
04-17-2015, 03:53 PM
I thought the same thing when I read his post, but then I realized the 200+ category isn't in his table. Both Duke and Utah had a big efficiency margin against sub-200 teamsKedsy's right. "All Games" does mean all games, not all others. You said, "No way our and Utah's efficiency margins for all games was 20+ given our results in the other categories," to which I respond, "WAY!!"

Sorry. I didn't much care about things after Utah passed us in performance against average opponents! :p I really thought the BIG games should have been weighted more.
Interesting to me (though, again, the story is more complex with adjustments and recency effects):
- We had a better Efficiency Margin for games in Top 150 (though Utah was steadily catching up as the ranks got worse). They caught and passed us after 150.
- Yet, we also had a better Efficiency Margin for games in the 250-300 and 300+ stratifications
- In the 150-250 segment, where the final nail was placed, we only had 1 game - the close win over Virginia Tech, while Utah slayed their opponents in that range.

COYS
04-17-2015, 03:54 PM
bedeviled, cool post. One question: why does the "all games" column look so weird in the efficiency margin data for Utah and Duke? Shouldn't it be a weighted average of the other categorical results rather than greater than any of those other results?

Still, your table (assuming it is otherwise accurate) shows something interesting and potentially important. And it gets back to another point others have hinted at. The thing with Duke this year is that, on the season, we had a tendency to lose our focus. As such, we rarely pummeled the teams we were expected to pummel, but we also rarely lost to top teams. In fact, we lost only twice to a team in the top-25 (Notre Dame twice), and both were potentially examples of us taking them too lightly (we blew them out when they had our full attention in Cameron). So it is quite possible that these models (which have to rely on a season's worth of data) actually did slightly underrate us, because they were based on a season's worth of data in which we didn't always play at or near our best.

Conversely, Utah appeared to be good at beating up on the teams they should beat up on, but struggled against top teams. So they may have outkicked their coverage a bit in getting a top-10 rating by virtue of taking every game (and every possession) more seriously than most. So whereas some of the teams in the 10-20 range may have slipped more often and played down to their competition, Utah may not have.

If both of the above are true, then models based on average performance of the course of a season are going to naturally consider that matchup more of a toss-up than perhaps it should be. So it may be that we really should have been a ~10% chance of winning. Or more. Or it could be that the model is assuming we might (like we did during the regular season and even during the ACC tournament) fall asleep at the wheel again thinking we could coast, and get upset.

Of course, none of this changes the overall point that none of these models are designed to accurately predict the champion. They are designed to, with a certain level of accuracy given the particular game considered, estimate the probability of correctly predicting the winner. So the best one can do is to compare the probability of correctly predicting the winner amongst games with a similar probability favorite/underdog.

In other words, they really aren't at all suited to predict the tournament champion, because all of them are going to predict that the #1 team to have the best chance, and with almost certainty even that team will have much less than a 50% chance of winning the title.

CDu, this is an interesting analysis. I know Kedsy and other have also documented how our guys really did seem to focus in all of our biggest games. I wonder if this will become more common in the OAD era for objectively talented but young teams to have better than expected tournament runs. Duke started three NBA caliber freshman. Kentucky in 2011 scuffled for parts of the season before putting together a Final Four run late in year while also starting mostly freshman and sophomores. Then, UK made the Final Four last year despite struggling earlier in the season. For a while, Duke was actually known as a team that came ready to play early in the season but (supposedly) struggled later in the season as other teams caught up in terms of cohesion and preparation. I'm not sure if I buy that just yet, but there might be something to the idea that it is less likely for a team of talented freshman to play up to their potential night in and night out during the grind of the season but that when the big lights are on, that lack of consistent focus evaporates.

CDu
04-17-2015, 04:07 PM
Kedsy's right. Sorry. I didn't much care about things after Utah passed us in performance against average opponents! :p I really thought the BIG games should have been weighted more.

Interesting to me (though, again, the story is more complex with adjustments and recency effects):
- We had a better Efficiency Margin for games in Top 150 (though Utah was steadily catching up as the ranks got worse). They caught and passed us after 150.
- Yet, we also had a better Efficiency Margin for games in the 250-300 and 300+ stratifications
- In the 150-250 segment, where the final nail was placed, we only had 1 game - the close win over Virginia Tech, while Utah slayed their opponents in that range.

Oh wow, I completely misread the table altogether. Please disregard my previous critiques of the table. The lesson here: I'm occasionally a dufus who can't read good.

Main point still remains (now even moreso): awesome article, bedeviled.

tbyers11
04-17-2015, 04:30 PM
Yeah, I guess, for all the reasons you state. Especially in light of the following:



So being down 4 with almost 12 minutes to play still only gave us a 30% chance of winning? That's just amazing to me.

One nitpick about this point that you, CDu, and pfrduke (suggesting that initial WP% plays a role) discussed eloquently above. By the play-by-play, we are actually only down 3 (48-45), not 4, after Grayson's barrage at the 12:10 mark. I agree that from casual observation it seems way too low. So now we can all be a little more incredulous :D

I also went back and found other points in the game when we were down 3 and the WP% off the KPom graph


Score Time Duke WP%
WI lead 12-9 13:28, 1st half 39%
WI lead 36-33 19:01, 2nd half 36%
WI lead 42-39 16:14, 2nd half 33%
WI lead 48-45 12:10, 2nd half 30%

If we started the game at 45% and were down 3 after 6 and half minutes, 39% seems about right. The resultant decreases in our WP% as the time remaining decreases also seem to make sense.

bjornolf
04-17-2015, 05:06 PM
Never tell me the odds.


My dad always said, "You only have a 1 in 3000 chance of being struck by lightning, unless you're that one guy."
The unspoken ending to that was that then your chance is 100%. I always liked that one.

Mtn.Devil.91.92.01.10.15
04-17-2015, 05:59 PM
Never tell me the odds.


My dad always said, "You only have a 1 in 3000 chance of being struck by lightning, unless you're that one guy."
The unspoken ending to that was that then your chance is 100%. I always liked that one.

Likewise we have a 100% probability of being the 2015 NCAA Champs for the rest of time.

That's my favorite stat.

DukieinSoCal
04-17-2015, 06:02 PM
KenPom was a win probability graph for the game, but it's behind his paywall. Because of that I won't post it here but here is some key info. Our lowest probability to win was 13.7% at the 13:25 mark of the 2nd half when were down 48-39. Our win % at the start of the game was 45.1%.

I wonder what factors KenPom uses to come up with these numbers. I'm sure he somehow factors in Wisconsin's historically efficient offense but how do you account for the fact that they had upperclassman leadership that had been to the Final Four the previous year and the fact that we were in serious foul trouble with Jah and Justise? To me, it felt like our chances of coming back to win in this particular game and situation were very, very low. Hence, the amazing euphoria that overtook me as we stormed back to take home the natty. I'm still on could nine! :)

Kedsy
04-17-2015, 07:17 PM
To me, it felt like our chances of coming back to win in this particular game and situation were very, very low.

I was at the game, and I wasn't particularly worried at all. I felt that if we had an empty possession and they hit a bucket to go up 11 or 12 then I'd start to worry. Instead, Grayson hit a three and it was a 6-point game. Like I said earlier, all we needed (down 9) was three-stop-three and it was a one-possession game. And you see three-stop-three all the time, right? It didn't occur to me until this discussion how unlikely that was.

So I guess different folks feel differently in situations like that.

uh_no
04-17-2015, 10:54 PM
d. The evidence for the ACC is how well the conference did in the NCAA's: an unprecedented five teams in the Sweet Sixteen

Minor quibble...this statement is false
2009 Big east:

16: uconn, UL, Pitt, Nova, Cuse
8: uconn, Ul, Pitt, Nova
4: Uconn, Nova

NOthing against the rest of it. I thought far and away that the ACC was the best conference, having NCSU, UL, Duke, ND, and UNC all going as far as they did (big miss on UVA....but what are you going to do!). There is much too much evidence to not consider the ACC as far and away the best at the top this year.

ice-9
04-17-2015, 11:46 PM
Why does the fact that a team with a fairly low probability of winning the tournament actually won it make the computer systems that determine the probabilities "faulty"? In three of the past five seasons, the team that won the tournament wasn't among the five most likely teams to win. And you think that means the rating systems are wrong? To me, it just means that the most likely team very often doesn't make it through a tough, one-and-done tournament. Frankly, the best team from the regular season rarely wins the NCAA tournament.

Also, the computer probability of winning it all depends very strongly on how difficult a path you have. Duke's percentage was low in large part because we had a 5-seed in our path (Utah) that the computers profiled as more of a 3-seed or even a 2-seed. Pomeroy, for example, ranked Utah as the #8 team in the country, and Gonzaga #6. Having to go through two top 10 teams is something none of the other #1 or #2 seeds had to achieve to get to the Final Four, hence Duke's chance of getting that far was lower than the other contenders. That makes perfect sense to me, not faulty at all.

I do think that this year, KenPom either greatly underestimated Duke or overestimated Duke's opponents in the NCAA tournament. I haven't looked up numbers, but my hunch is that Duke outperformed the predicted score in most of their games in the tournament.

The reason why Duke's initial odds were so low is because Utah and Gonzaga were seen as so good. San Diego State wasn't too bad either. Now that we've played the games, I think we can all agree Duke was clearly better than those two -- much better than what KenPom would have predicted.

uh_no
04-18-2015, 01:13 AM
I do think that this year, KenPom either greatly underestimated Duke or overestimated Duke's opponents in the NCAA tournament. I haven't looked up numbers, but my hunch is that Duke outperformed the predicted score in most of their games in the tournament.

The reason why Duke's initial odds were so low is because Utah and Gonzaga were seen as so good. San Diego State wasn't too bad either. Now that we've played the games, I think we can all agree Duke was clearly better than those two -- much better than what KenPom would have predicted.

and more importantly because duke's tournament defense far outstripped almost anything it had done in the regular season.

I do think utah/zaga might have been slightly overrated in kenpom...but i would have had them top 15

sagegrouse
04-18-2015, 06:43 AM
I do think that this year, KenPom either greatly underestimated Duke or overestimated Duke's opponents in the NCAA tournament. I haven't looked up numbers, but my hunch is that Duke outperformed the predicted score in most of their games in the tournament.

The reason why Duke's initial odds were so low is because Utah and Gonzaga were seen as so good. San Diego State wasn't too bad either. Now that we've played the games, I think we can all agree Duke was clearly better than those two -- much better than what KenPom would have predicted.

The ACC was systematically underrated due to a fairly mundane conference performance in November and December. In fact, the ACC was very, very good this year.

And also, Utah and Gonzaga were probably overrated: West Coast teams did not play many games against teams in the other power conferences.

bedeviled
04-18-2015, 09:21 AM
i don't think there is any mathematical difference between
A. team x has a 55% chance of beating team y
B. team x will beat team y and that statement has a 55% chance of being correctPerhaps you are calling attention to something I misstated. If you are speaking more generally, though, consider that the problem is that the model's statements and validations (B) are based on groups, not the singular game (A) in question. Thus, it should read:
A. team x has a 55% chance of beating team y
B. a team like x beats a team like y, and that statement has been true 55% of the time when looking at data from a group of such teams.
Unless the model has accounted for every possible factor, it cannot jump from point B to point A. There is a difference.

Declaring group odds (probabilities from a set of modeled/historical games) as if they were the odds of a singular event is a type of false equivalency unless all variables are equal. Moreover, the odds are also somewhat arbitrary if not all variables are factored in. That is, the odds depend on what criteria are used in the model. Different models will give different odds.

For example, say you have a history of flipping the following 4 coins and recording their outcomes. These represent your databank of games.


Denomination

Weighted?
Odds of Heads


Quarter
Yes
100%


Quarter
No
50%


Nickel
Yes
25%


Penny
Yes
0%



Now, you have a new coin (Duke vs Wisconsin), and you want to predict the odds of tossing a heads (Duke wins). The new coin is a weighted quarter. But, you don't know how it is weighted until you toss it...at which point you would find out it comes up heads 100% of the time. So, what are your pre-toss predictions? Well, it depends on what model you are using - various models think different things are important about coins, and arrange their data accordingly.


Model

Criterion Important to the Model
Silly Analogous Bball Criterion
Odds of Heads if Compared to Similar Coins from Databank




Model 1
"Coins are not distinguishable"
Cuban says, "MBB is an ish-show. Who cares?"
44%


Model 2
"Weighted coins are special"
Teams from Big12 win
42%


Model 3
"I've seen silver coins win more in the past. Silver is for winners!"
Teams with high Efficiency Margins win
58%


Model 4
"Quarters are worth more"
Teams w/ 5+ McD's
75%




Qtrpounder All-Americans



Model 5
"This is a weighted, silver quarter"
2015 Duke reminds me of 1991
100%



So, we see that the stated prediction varies with whatever the model thinks is important, while the true odds remain constant. And, obviously, the stated predictions and the true probability are not the same....except for Model 5 which both considers all the variables in our simple example and includes an event exactly like our unknown event.

Hopefully, anyone who has a loved one with a tragic disease is aware that the group odds and the singular odds are not equal. Indeed, medicine is always working to refine their groupings/comparisons to give more accurate predictions.
(identifying biomarkers to separate out which patients will respond to a certain medication)
(better characterizing a disease to differentiate who has a progressive form vs protracted form)

A logical question, then, is 'How do we better refine our tournament basketball prediction?'
Because I find it interesting, I'd like to include one consideration for refinement. I have previously linked an Ohio State term paper (http://kb.osu.edu/dspace/bitstream/handle/1811/48883/EmilyToutkoushian.pdf?sequence=1) on developing a model specifically for predicting the tournament (data was from 1986-2009 and tested on 2010,11). In developing her model, the author disovered that "% of seasons coach has gone to the Final Four" was a significant factor in predicting tourney results. I don't think it's a great leap to think that it would have been a factor in predicting 2015 results, too! Regarding the variable, she had this to say,
The very small number of coaches that this variable is meaningful makes it almost useless, but does imply that it is worth recruiting, and paying for, coaches who have made it to the Final Four and/or Championships.For future considerations, I wonder if "Coach previously played in or was Assistant Coach in Final Four" would also be a significant factor.

bedeviled
04-18-2015, 09:39 AM
I just went to the Kaggle March Madness competition site (https://www.kaggle.com/c/march-machine-learning-mania-2015/forums) to see how those crazy smart folks did. Per Wikipedia,
Kaggle is a platform for predictive modelling and analytics competitions on which companies and researchers post their data and statisticians and data miners from all over the world compete to produce the best models

In one of the first posts in their forum, someone was bragging about being top 1% of the ESPN bracket pool....just before he mentioned the news that a 12-year-old ties for 1st in ESPN bracket challenge (http://www.wralsportsfan.com/12-year-old-ties-for-1st-in-espn-bracket-challenge/14566519/).

Duke95
04-18-2015, 11:53 PM
The ACC was systematically underrated due to a fairly mundane conference performance in November and December. In fact, the ACC was very, very good this year.

And also, Utah and Gonzaga were probably overrated: West Coast teams did not play many games against teams in the other power conferences.

I would agree somewhat on Gonzaga. However, if you remember, quite a few people picked Gonzaga to beat us. With regard to Utah, I've seen them play a bit. They played Arizona tough. With Kansas, they had horrible first half jitters, then stormed back. They were a very strong team. Honestly, I would say they were slightly underrated.

subzero02
04-19-2015, 12:12 AM
I would agree somewhat on Gonzaga. However, if you remember, quite a few people picked Gonzaga to beat us. With regard to Utah, I've seen them play a bit. They played Arizona tough. With Kansas, they had horrible first half jitters, then stormed back. They were a very strong team. Honestly, I would say they were slightly underrated.

Utah was probably under-seeded but they were not underrated by several of the computer models.

jv001
04-19-2015, 06:55 AM
The ACC was systematically underrated due to a fairly mundane conference performance in November and December. In fact, the ACC was very, very good this year.

And also, Utah and Gonzaga were probably overrated: West Coast teams did not play many games against teams in the other power conferences.

SEC (minus UK) and West coast teams very much over rated and Duke was under rated at the end of the season. Duke's defense picked up towards the end of the season and made Duke one of the nations two or three best teams. Matter of fact we'll never know how good Kentucky was because of their very soft SEC schedule. GoDuke!

Kedsy
04-19-2015, 10:50 AM
Matter of fact we'll never know how good Kentucky was because of their very soft SEC schedule. GoDuke!

What I think is overrated is the weakness of Kentucky's schedule. KenPom's final numbers rank UK's schedule as the 31st best in the country.

Jarhead
04-19-2015, 11:36 AM
What I think is overrated is the weakness of Kentucky's schedule. KenPom's final numbers rank UK's schedule as the 31st best in the country.

When was that last calculated?

CDu
04-19-2015, 11:40 AM
What I think is overrated is the weakness of Kentucky's schedule. KenPom's final numbers rank UK's schedule as the 31st best in the country.

To be fair, that is partially inflated by facing West Virginia, Notre Dame, and Wisconsin to end the tournament. It also may reflect some overvaluing of the SEC.

UK certainly did play and wallop some good teams: Louisville and UNC for example. But on the whole I do think their schedule was pretty soft.

bedeviled
04-19-2015, 12:16 PM
To be fair, that is partially inflated by facing West Virginia, Notre Dame, and Wisconsin to end the tournament. It also may reflect some overvaluing of the SEC. UK certainly did play and wallop some good teams: Louisville and UNC for example. But on the whole I do think their schedule was pretty soft.Overvaluing of Texas, as well.

What I find overvalued is the BPI. I hated hearing ESPN talk about it this year, and will be even more disgruntled next year.
Duke had the 67th hardest SOS according to the BPI.....despite a full third of our games (13) being against the top 20 teams in the BPI! With such an easy schedule, it's no wonder we were able to finish 6th, lol.

Kedsy
04-19-2015, 10:23 PM
To be fair, that is partially inflated by facing West Virginia, Notre Dame, and Wisconsin to end the tournament. It also may reflect some overvaluing of the SEC.

UK certainly did play and wallop some good teams: Louisville and UNC for example. But on the whole I do think their schedule was pretty soft.

It was definitely inflated by UK's tournament games. But before the tournament started, Kentucky's schedule was still rated 41st by Pomeroy. Not as strong as Duke's (16th pre-tourney), but a statement saying UK was overrated because they played such a poor schedule wouldn't reasonably seem to be speaking about the 41st best schedule in the country.

subzero02
04-19-2015, 10:56 PM
To be fair, that is partially inflated by facing West Virginia, Notre Dame, and Wisconsin to end the tournament. It also may reflect some overvaluing of the SEC.

UK certainly did play and wallop some good teams: Louisville and UNC for example. But on the whole I do think their schedule was pretty soft.

Although Kansas fans are probably happy that you didn't list them amongst the good teams that Kentucky stomped; rock chalk should probably be listed first after the beat down they received in the champions classic.( UK 72... KU 40)

ice-9
04-19-2015, 11:57 PM
I would agree somewhat on Gonzaga. However, if you remember, quite a few people picked Gonzaga to beat us. With regard to Utah, I've seen them play a bit. They played Arizona tough. With Kansas, they had horrible first half jitters, then stormed back. They were a very strong team. Honestly, I would say they were slightly underrated.

Now that you've seen the games, in a series of 10, how many times do you think Gonzaga and Utah would win against Duke?

I'd put it at two, maybe three. Duke would win 80% of the time against those teams. Gonzaga and Utah aren't on Duke's level, despite what KenPom thinks.

(And really, maybe 90%. The more times they play against each other the more apparent the talent differential.)

CDu
04-20-2015, 09:19 AM
Now that you've seen the games, in a series of 10, how many times do you think Gonzaga and Utah would win against Duke?

I'd put it at two, maybe three. Duke would win 80% of the time against those teams. Gonzaga and Utah aren't on Duke's level, despite what KenPom thinks.

(And really, maybe 90%. The more times they play against each other the more apparent the talent differential.)

The way Duke played in the tournament (most specifically, the way Duke played DEFENSE in the tournament), Pomeroy would have absolutely agreed with this sentiment. The "problem" (from a modelling perspective) is that Duke didn't play defense like that during the season. They showed flashes of it later in the season (most notably in the first half against Notre Dame in Cameron), but throughout the year they just seemed to do enough defensively to let their offense win by a comfortable margin.

Pomeroy (and all of the models) probably had Duke rated appropriately based on their body of work; they just didn't have a way of predicting that Duke's defense would come to life so brilliantly for 6 straight games in the NCAA tournament.

Troublemaker
04-20-2015, 09:23 AM
Now that you've seen the games, in a series of 10, how many times do you think Gonzaga and Utah would win against Duke?

I'd put it at two, maybe three. Duke would win 80% of the time against those teams. Gonzaga and Utah aren't on Duke's level, despite what KenPom thinks.

(And really, maybe 90%. The more times they play against each other the more apparent the talent differential.)

I'm not sure the eye test is reliable in one-game samples.

If Duke would beat Miami 8 out of 10 times, but all you saw was the game in Cameron, you might think it was actually Miami that would beat Duke 8 out of 10.

And, maybe Duke really does only beat Utah 6 out of 10 times but because we were fortunate to witness one of the 6 the first and only time the two teams played, we now perhaps mistakenly believe Duke would win 8 out of 10.

CDu
04-20-2015, 09:34 AM
I'm not sure the eye test is reliable in one-game samples.

If Duke would beat Miami 8 out of 10 times, but all you saw was the game in Cameron, you might think it was actually Miami that would beat Duke 8 out of 10.

And, maybe Duke really does only beat Utah 6 out of 10 times but because we were fortunate to witness one of the 6 the first and only time the two teams played, we now perhaps mistakenly believe Duke would win 8 out of 10.

Agreed. It certainly helped that Utah's star got in foul trouble early. It also helped that Utah (a very good shooting team) couldn't hit anything from 3pt range. And it is worth noting that, despite those two facts, Utah only lost by 6. If either (or both) of the two facts above goes differently, we may have been in even more of a dogfight than we were in anyway.

Wander
04-20-2015, 11:22 AM
I'm not sure the eye test is reliable in one-game samples.


Then don't use the eye test and look at Utah's wins and losses. Utah went from Dec 11 to March 18 with one win against an NCAA tournament team, despite playing in a power conference. They finished the regular season 3-7 against NCAA tournament teams - the best of those wins was a one point overtime victory at home against a 7 seed. They deserve credit for crushing mediocre opponents, but IMO looking at the wins/losses, it's apparent they were overrated by the computers.

Saratoga2
04-20-2015, 11:31 AM
Agreed. It certainly helped that Utah's star got in foul trouble early. It also helped that Utah (a very good shooting team) couldn't hit anything from 3pt range. And it is worth noting that, despite those two facts, Utah only lost by 6. If either (or both) of the two facts above goes differently, we may have been in even more of a dogfight than we were in anyway.

You make good points as you normally do. My take away is that it is very difficult to make a predictive model that yields good results for a single game, or even a group of games, due to the sheer number of variables. Even if one makes very complex models, like those governing weather prediction, it is likely to miss fairly often. The wonder of it was that the selection committee did a very good job of seeding. Duke was seeded as a 1 and won through despite the pundits picking against them often for reasons discussed on this thread. Wiscoonsin was elevated to a 1 and certainly deserved it. Kentucky earned their 1 but did so without playing really tough competition to get there. Villanova also earned their 1 and was hit but a physical ACC team and was prepared for that kind of struggle. While it is fun to talk about the statisical nature of the game the predictive capability should be taken with a grain of salt.

sagegrouse
04-20-2015, 11:45 AM
Then don't use the eye test and look at Utah's wins and losses. Utah went from Dec 11 to March 18 with one win against an NCAA tournament team, despite playing in a power conference. They finished the regular season 3-7 against NCAA tournament teams - the best of those wins was a one point overtime victory at home against a 7 seed. They deserve credit for crushing mediocre opponents, but IMO looking at the wins/losses, it's apparent they were overrated by the computers.

Hi, broken record here: You are exactly right. I went through the inter-conference schedules for teams in all the power conferences. The PAC-12 had about one-half as many per team as those in the other conferences. And, of course, those games were all in November and December.

Computer models without meaningful data are fairly useless.

hughgs
04-20-2015, 01:25 PM
You make good points as you normally do. My take away is that it is very difficult to make a predictive model that yields good results for a single game, or even a group of games, due to the sheer number of variables. Even if one makes very complex models, like those governing weather prediction, it is likely to miss fairly often. The wonder of it was that the selection committee did a very good job of seeding. Duke was seeded as a 1 and won through despite the pundits picking against them often for reasons discussed on this thread. Wiscoonsin was elevated to a 1 and certainly deserved it. Kentucky earned their 1 but did so without playing really tough competition to get there. Villanova also earned their 1 and was hit but a physical ACC team and was prepared for that kind of struggle. While it is fun to talk about the statisical nature of the game the predictive capability should be taken with a grain of salt.

I keep seeing this idea that somehow the predictive power of the models are bad.

Y'all are missing the point.

The models are statistical models. Statistical models don't predict anything, they simply tell you the probability of a given outcome. To ask a statistical model to predict something is akin to asking the theory of evolution to explain how the universe was created. In both cases you would be asking the model/theory to do something that it is not designed to do.

Duvall
04-20-2015, 01:28 PM
The ACC was systematically underrated due to a fairly mundane conference performance in November and December. In fact, the ACC was very, very good this year.

Well, it's probably not an accident that the ACC started looking better when the teams that were pulling it down stopped playing games.

ice-9
04-20-2015, 11:23 PM
I keep seeing this idea that somehow the predictive power of the models are bad.

Y'all are missing the point.

The models are statistical models. Statistical models don't predict anything, they simply tell you the probability of a given outcome. To ask a statistical model to predict something is akin to asking the theory of evolution to explain how the universe was created. In both cases you would be asking the model/theory to do something that it is not designed to do.

The problem is a few/some/many keep using these statistical models to make definitive statements of how team x is better than team y -- as an example, observe the premise of the 538 article, which used the same statistics to predict tournament outcomes.

As if any observation contrary to those models is your own personal fallacy, a victim of randomness, because obviously KenPom's probabilities are the arbiter of truth. KenPom is just one tool among many.

hughgs
04-20-2015, 11:57 PM
The problem is a few/some/many keep using these statistical models to make definitive statements of how team x is better than team y -- as an example, observe the premise of the 538 article, which used the same statistics to predict tournament outcomes.

As if any observation contrary to those models is your own personal fallacy, a victim of randomness, because obviously KenPom's probabilities are the arbiter of truth. KenPom is just one tool among many.

I looked at the article cited in the original post and didn't see where it was saying that their model predicted anything. Can you point out where 538 is using statistics to predict something? Thanks.

BobbyFan
04-21-2015, 12:11 AM
This is spot on. UConn in 1999 was an excellent team that would be considered among the favorites most seasons. I don't they were considered the consensus #2 team though, as Michigan State was right there too. Duke, on the other hand, was a historically great team.

UConn being ranked at the top of the polls for several weeks was influenced by us losing early in the season. But there was consistently a significant gap (larger than with this year's Kentucky team) between us and the rest in the Sagarin ratings throughout the season. The same would surely hold for the RPI ratings.

To expound, I did some digging:

RPI (http://www.basketballprospectus.com/rpi.php?y=1999) We had an RPI entering the tournament of .693, far ahead of Michigan State and UConn. Since then, there hasn't been as large a discrepancy between the #1 and #2 team. The only comparable years in the modern era I could find prior to that (I couldn't find all seasons, but did get 91 and 92) were Kentucky 1996 which was a great, loaded team (and better than this year's UK team), and UNC 1984 which was one of the great teams to not win a title.

Kenpom (http://www.basketballprospectus.com/rate.php?y=1999) Well, it says Pomeroy. I'm sure the methodology has since been adjusted, but the dominance is clear.

I couldn't find Sagarin's full ratings that season. But I did find our numbers in, of all places, a UK forum (http://www.kysportsreport.com/forums/showthread.php?22471-Sagarin-UK-s-Rating-101-58-4th-Highest-Since-1975-%28Behind-1999-Duke-96-UK-76-IU%29) which were provided in the excitement of their ultimately and sadly unfulfilled season. We had the highest Sagarin rating since 1975, as per the post:



1. 1999 Duke 104.88
2. 1996 Kentucky 103.26
3. 1976 Indiana 102.36
4. 2015 Kentucky 101.58
5. 1991 UNLV 101.14
6. 1975 Indiana 101.06

Some other teams of interest:

2012 UK 95.75
2001 Duke 99.52
2007 Florida 96.14

In retrospect, the 99 team has become underrated, which is a bit odd considering that the hype that year was off the charts. Maybe it's because of the perceived lack of NBA success of its individual players, or perhaps that it lacked the flair and brashness of 1991 UNLV to make it memorable. But that was an incredible team which fit together awfully well, both on offense and, often overlooked, on defense.

Highlander
04-21-2015, 11:17 AM
In retrospect, the 99 team has become underrated, which is a bit odd considering that the hype that year was off the charts. Maybe it's because of the perceived lack of NBA success of its individual players, or perhaps that it lacked the flair and brashness of 1991 UNLV to make it memorable. But that was an incredible team which fit together awfully well, both on offense and, often overlooked, on defense.

The perceived lack of NBA success has always kind of rubbed me the wrong way. I went back and looked at all the NBA champions from 1980 through 2014, so the last 35 years. Fair warning that this is all based on Wikipedia (http://en.wikipedia.org/wiki/List_of_NBA_champions) and my sometimes faulty memory. At any rate, multiple titles won by superstars break down like this:


Michael Jordan - 6
Tim Duncan -5
Kobe Bryant - 5
Magic Johnson - 5
Shaq - 4 (3 with Kobe)
Larry Bird - 3
Lebron James - 2
Hakeem Olajuwon - 2
Isaiah Thomas - 2


By my count, there were only 4 titles won without one of these 9 people on the team - Dallas 2011, Boston 2008, Detroit 2004, and Philadelphia 1983. That means, unless you played on a team with one of these 9 Hall of Famers, there was an 88% chance you were never going to win a title. If you played between 1980 and 2003, your odds of a title were a paltry 4%. That era overlaps with Duke's first two National Championships and 7 Final Fours in 9 years and includes greats like Dawkins, Alarie, Ferry, Laettner, Hill, and Hurley. They all had the bad luck of having to match up against Magic, Bird, Hakeem, and Jordan led teams. The only Duke players to win titles were Ferry and Battier, both of which were complimentary players at the end of their careers playing with HOF level talent for the first time. The NBA has certainly become less star driven over the last 15 years, but still the best way to win a title in the NBA is to play with a hall of famer on your team, or be said Hall of Famer. The criticism of Duke I think is more that they have never produced a HOF caliber player capable of leading his team to multiple titles the way Jordan, Duncan, Shaq, and Kobe have.

Duke's '99 team certainly bodes that true. I mean, Battier won two NBA titles with the Heat after a 13 year career, while Brand was an All-Star and All NBA 2nd team on a similar 14 year career. Corey Maggette also hung around the league for a solid 14 years, but never played on a great team, spending 8 of those years on the Sterling owned Clippers. Only Will Avery and Trajan Langdon could be seen as disappointments, really, and Trajan had a fairly normal NBA career followed by lots of success overseas. By most accounts, 2 titles, an NBA All Star, and three 14 year NBA veterans off of one NCAA team would be pretty successful. But because none of them were Michael Jordan, it is perceived that they were a disappointment.

Bostondevil
04-21-2015, 01:13 PM
By most accounts, 2 titles, an NBA All Star, and three 14 year NBA veterans off of one NCAA team would be pretty successful. But because none of them were Michael Jordan, it is perceived that they were a disappointment.

Something I always say to folks who want to rag on Duke players in the NBA - take away Michael Jordan and who does Carolina have?

Michael Jordan went to Carolina in 1981. Coach K was not Coach K in 1981. Who has Carolina produced since, oh, let's say 1986 that has had a better NBA career than Elton Brand? I don't really follow the NBA so, you're gonna have to tell me. I do watch the Celtics sometimes. I'd be a bigger Celtics fan if they had at least one Duke guy on their team, although I did love the '08 Celtics.

Highlander
04-21-2015, 02:00 PM
Something I always say to folks who want to rag on Duke players in the NBA - take away Michael Jordan and who does Carolina have?

Michael Jordan went to Carolina in 1981. Coach K was not Coach K in 1981. Who has Carolina produced since, oh, let's say 1986 that has had a better NBA career than Elton Brand? I don't really follow the NBA so, you're gonna have to tell me. I do watch the Celtics sometimes. I'd be a bigger Celtics fan if they had at least one Duke guy on their team, although I did love the '08 Celtics.

Take your pick...

Rasheed Wallace was a key piece of the Detroit Pistons who won in 2004. He was a 4x All Star. That's better than Elton by any objective measure.

If you just go via titles...
Scott Williams won 3 titles with Jordan in 91-93.
Kenny Smith won 2 titles with the Rockets and Hakeem in 1994 and 1995.
Pete Chilcutt was also on the 1995 Rockets team.
Rick Fox won 3 titles as a 6th man for the Lakers from 2001-2003.
Brendan Haywood won a title with the Mavericks in 2011.

In that same vein, Duke has 3 titles - 2 by Battier, 1 by Ferry.

sagegrouse
04-21-2015, 02:06 PM
Take your pick...

Rasheed Wallace was a key piece of the Detroit Pistons who won in 2004. He was a 4x All Star. That's better than Elton by any objective measure.

If you just go via titles...
Scott Williams won 3 titles with Jordan in 91-93.
Kenny Smith won 2 titles with the Rockets and Hakeem in 1994 and 1995.
Pete Chilcutt was also on the 1995 Rockets team.
Rick Fox won 3 titles as a 6th man for the Lakers from 2001-2003.
Brendan Haywood won a title with the Mavericks in 2011.

In that same vein, Duke has 3 titles - 2 by Battier, 1 by Ferry.

James Worthy entered the NBA in 1982 and won three titles -- as a starter in 1985, 1987, and 1988. He is in the Naismith HOF.

Duke95
04-21-2015, 02:31 PM
I keep seeing this idea that somehow the predictive power of the models are bad.

Y'all are missing the point.

The models are statistical models. Statistical models don't predict anything, they simply tell you the probability of a given outcome. To ask a statistical model to predict something is akin to asking the theory of evolution to explain how the universe was created. In both cases you would be asking the model/theory to do something that it is not designed to do.

If you are speaking in general terms, then what you are saying is not correct. Statistical models do, in fact, generate predicted values. Econometric analyses are often concerned with forecasting the dependent variable generating predicted values in a world where, for example, an alleged wrongdoing (e.g., price-fixing) does not exist.

If you are speaking just with regard to the 538 model, yes, I think that's right, all they are are doing is generating probabilities.

hughgs
04-21-2015, 02:37 PM
If you are speaking in general terms, then what you are saying is not correct. Statistical models do, in fact, generate predicted values. Econometric analyses are often concerned with forecasting the dependent variable generating predicted values in a world where, for example, an alleged wrongdoing (e.g., price-fixing) does not exist.

If you are speaking just with regard to the 538 model, yes, I think that's right, all they are are doing is generating probabilities.

Knowing nothing about econometric analyses I would say forecasting using statistics is very different than a statistical model. I've used statistics in models but have never considered them statistical models. In my mind, and this may be where the wheels fall off, a statistical model is trying to predict the statistics of a situation, not predict or forecast an outcome.

sagegrouse
04-21-2015, 02:38 PM
If you are speaking in general terms, then what you are saying is not correct. Statistical models do, in fact, generate predicted values. Econometric analyses are often concerned with forecasting the dependent variable generating predicted values in a world where, for example, an alleged wrongdoing (e.g., price-fixing) does not exist.

If you are speaking just with regard to the 538 model, yes, I think that's right, all they are are doing is generating probabilities.

Correctimundo! Statistical models are very often used for forecasts or predictions. The statistics of forecasts can be a bit different IIRC (and there is always the first time), especially if there is some extrapolation beyond the existing data set.

Duke95
04-21-2015, 02:46 PM
Knowing nothing about econometric analyses I would say forecasting using statistics is very different than a statistical model. I've used statistics in models but have never considered them statistical models. In my mind, and this may be where the wheels fall off, a statistical model is trying to predict the statistics of a situation, not predict or forecast an outcome.

Well, not quite. Remember, when saying "statistical models", you are really looking at a wide array of tools available. For example, a structural model would be considered an econometric model, which is just another way of saying you're applying statistical principles to economic problems.

Statistical models, including time series models such as ARIMA, etc., can be atheoretical and dedicated entirely to generating accurate forecasts. Forecasting and prediction are often the main goal of time series models (though not always, sometimes you just want to determine the influence of certain factors.)

There are also models that generate a predicted probability, such as logit/probit models, used with limited dependent variables (e.g., binary or categorical outcomes). For example, if you have a bunch of variables and you want to determine how they influence whether teams won or lost (1/0 binary outcome), then you would likely use such a model.

Long story short, the term "statistical model" doesn't imply a particular type of statistic any more than Phylum Chordata implies Class Mammalia.

Bostondevil
04-21-2015, 02:52 PM
Take your pick...

Rasheed Wallace was a key piece of the Detroit Pistons who won in 2004. He was a 4x All Star. That's better than Elton by any objective measure.

If you just go via titles...
Scott Williams won 3 titles with Jordan in 91-93.
Kenny Smith won 2 titles with the Rockets and Hakeem in 1994 and 1995.
Pete Chilcutt was also on the 1995 Rockets team.
Rick Fox won 3 titles as a 6th man for the Lakers from 2001-2003.
Brendan Haywood won a title with the Mavericks in 2011.

In that same vein, Duke has 3 titles - 2 by Battier, 1 by Ferry.

Pete Chilcutt? How to say this as nicely as possible - but really? You're giving me Pete Chilcutt as a better NBA player than Elton Brand?

As you so aptly point out, you don't win titles unless you play on a team with an alpha dog title winner, so, no, you can't compare careers based on titles. When comparing all the non alpha dogs - titles are the last thing I would consider.

But OK - I'll give you Rasheed Wallace. Since Jordan, the best NBA player to come out of Carolina is Rasheed Wallace.

Alpha Dogs are rare, Carolina had one. Who else? Michigan State, Indiana State, Houston, Wake Forest, Indiana and LSU. If you go by titles - then the best NBA player producing NCAA teams are UNC, Wake Forest, and Michigan State, plus wherever Kobe went to high school.

Kedsy
04-21-2015, 03:15 PM
...plus wherever Kobe went to high school.

Lower Merion high school in suburban Philadelphia. Same high school my son is going to attend next fall. (Though my son has a zero percent chance of playing on an NBA championship team, I'm sad to say.)

CDu
04-21-2015, 03:22 PM
Pete Chilcutt? How to say this as nicely as possible - but really? You're giving me Pete Chilcutt as a better NBA player than Elton Brand?

As you so aptly point out, you don't win titles unless you play on a team with an alpha dog title winner, so, no, you can't compare careers based on titles. When comparing all the non alpha dogs - titles are the last thing I would consider.

But OK - I'll give you Rasheed Wallace. Since Jordan, the best NBA player to come out of Carolina is Rasheed Wallace.

Alpha Dogs are rare, Carolina had one. Who else? Michigan State, Indiana State, Houston, Wake Forest, Indiana and LSU. If you go by titles - then the best NBA player producing NCAA teams are UNC, Wake Forest, and Michigan State, plus wherever Kobe went to high school.

Others include Vince Carter, Jerry Stackhouse, Brad Daugherty, and Antawn Jamison, all of whom are/were as good or better than Brand in the NBA.

sagegrouse
04-21-2015, 03:27 PM
Others include Vince Carter, Jerry Stackhouse, Brad Daugherty, and Antawn Jamison, all of whom are/were as good or better than Brand in the NBA.

I'm too busy to look the rest up, but Elton Brand has more points and three times as many rebounds as Jerry Stackhouse. Moreover, Elton was a great player to have on your team (still is!) -- not necessarily so about Stackhouse.

hughgs
04-21-2015, 03:46 PM
Well, not quite. Remember, when saying "statistical models", you are really looking at a wide array of tools available. For example, a structural model would be considered an econometric model, which is just another way of saying you're applying statistical principles to economic problems.

Statistical models, including time series models such as ARIMA, etc., can be atheoretical and dedicated entirely to generating accurate forecasts. Forecasting and prediction are often the main goal of time series models (though not always, sometimes you just want to determine the influence of certain factors.)

There are also models that generate a predicted probability, such as logit/probit models, used with limited dependent variables (e.g., binary or categorical outcomes). For example, if you have a bunch of variables and you want to determine how they influence whether teams won or lost (1/0 binary outcome), then you would likely use such a model.

Long story short, the term "statistical model" doesn't imply a particular type of statistic any more than Phylum Chordata implies Class Mammalia.

I don't think my response implied a particular type of statistic. I'm mostly trying to integrate my ideas of what a statistical model does with the types of models I've used in the past.

In my mind, whether the data is generated stochastically or generated deterministically, it is the output of the model that defines whether the model is a statistical model. While not the arbiter of everything, Wikipedia seems to agree with me:

"In mathematical terms, a statistical model is usually thought of as a pair (S, \mathcal{P}), where S is the set of possible observations, i.e. the sample space, and \mathcal{P} is a set of probability distributions on S.[2]"

In other words, a statistical model uses observations to generate probabilities.

In my mind that would be different than using statistical data to generate forecasts or predictions. Tell me where our differences lie. Thanks.

Highlander
04-21-2015, 03:48 PM
Pete Chilcutt? How to say this as nicely as possible - but really? You're giving me Pete Chilcutt as a better NBA player than Elton Brand?

As you so aptly point out, you don't win titles unless you play on a team with an alpha dog title winner, so, no, you can't compare careers based on titles. When comparing all the non alpha dogs - titles are the last thing I would consider.

But OK - I'll give you Rasheed Wallace. Since Jordan, the best NBA player to come out of Carolina is Rasheed Wallace.

Alpha Dogs are rare, Carolina had one. Who else? Michigan State, Indiana State, Houston, Wake Forest, Indiana and LSU. If you go by titles - then the best NBA player producing NCAA teams are UNC, Wake Forest, and Michigan State, plus wherever Kobe went to high school.

I did use the qualifier for Chilcutt of "If you go by titles..." The reason I included him is that his title is no more or less significant than Danny Ferry's. And Ferry was, until 3 years ago, the only Duke player in the K era to ever play on an NBA championship team. Titles are what people view as "perceived NBA success"

But for the sake of argument, let's throw titles out the window and see if any other UNC players since Jordan had equal or better careers than Elton Brand. I'll stick to players who are retired or close to it, as the verdict is still out on active players. These are in addition to the athletes not named Pete in my list below:


Vince Carter was an 8x NBA All Star and 2x All NBA team. He had a streak of 23 straight games where he scored over 20 points. He's 25th in career scoring, right between Robert Parrish and Charles Barkley. I'd say that's objectively better than Elton.
Jerry Stackhouse was a 2x NBA All Star, averaging close to 30 ppg in 2001. He played for 18 years.
Antawn Jamison was also a 2x NBA All Star. He is 40th in NBA career scoring (Brand isn't ranked).


So the best NBA players to come out of UNC since Michael Jordan who had equal or better careers to Elton Brand are Rasheed Wallace, Vince Carter, Jerry Stackhouse, Antawn Jamison, and Rick Fox. None of those players were Michael Jordan for sure, but it's not accurate to say UNC has produced no one of consequence since MJ.

As SageGrouse says, James Worthy was also an "Alpha Dog" from UNC who won 3 titles and was a starter on those teams. So UNC's produced two, not one, since 1980. No other school can claim that.

I hate the fact that UNC players have had so much success in the NBA, especially when compared head to head with our guys. Some of that is due to luck - we've had lots of bad (JWill, Hurley, Hill with injuries) and they have had their share of good (Fox, Chilcutt, Haywood). But even if you take out Jordan/Worthy and only look 2000 and beyond, their cupboard is hardly bare.

CDu
04-21-2015, 04:06 PM
I did use the qualifier for Chilcutt of "If you go by titles..." The reason I included him is that his title is no more or less significant than Danny Ferry's. And Ferry was, until 3 years ago, the only Duke player in the K era to ever play on an NBA championship team. Titles are what people view as "perceived NBA success"

But for the sake of argument, let's throw titles out the window and see if any other UNC players since Jordan had equal or better careers than Elton Brand. I'll stick to players who are retired or close to it, as the verdict is still out on active players. These are in addition to the athletes not named Pete in my list below:


Vince Carter was an 8x NBA All Star and 2x All NBA team. He had a streak of 23 straight games where he scored over 20 points. He's 25th in career scoring, right between Robert Parrish and Charles Barkley. I'd say that's objectively better than Elton.
Jerry Stackhouse was a 2x NBA All Star, averaging close to 30 ppg in 2001. He played for 18 years.
Antawn Jamison was also a 2x NBA All Star. He is 40th in NBA career scoring (Brand isn't ranked).


So the best NBA players to come out of UNC since Michael Jordan who had equal or better careers to Elton Brand are Rasheed Wallace, Vince Carter, Jerry Stackhouse, Antawn Jamison, and Rick Fox. None of those players were Michael Jordan for sure, but it's not accurate to say UNC has produced no one of consequence since MJ.

As SageGrouse says, James Worthy was also an "Alpha Dog" from UNC who won 3 titles and was a starter on those teams. So UNC's produced two, not one, since 1980. No other school can claim that.

I hate the fact that UNC players have had so much success in the NBA, especially when compared head to head with our guys. Some of that is due to luck - we've had lots of bad (JWill, Hurley, Hill with injuries) and they have had their share of good (Fox, Chilcutt, Haywood). But even if you take out Jordan/Worthy and only look 2000 and beyond, their cupboard is hardly bare.

Totally agreed. I would also note that injuries derailed the career of one of UNC's best: Brad Daugherty.

The tide is certainly changing though: since Carter and Jamison, the advantage is decidedly in favor of Duke. We have Brand, Battier, Boozer, Dunleavy, Deng, Irving, and Maggette who have been clearly better than UNC's best in that time (Lawson or Marvin Williams), guys like Redick and Henderson are comparable, and guys like Parker, Okafor, and Winslow should pass those guys too. But it is really hard to argue against UNC's NBA success among players leaving college up through 1998.

InSpades
04-21-2015, 04:41 PM
Curious as to why we are looking at who was better than Brand and the cut-off being 1994 or so.

If we say post-Jordan then shouldn't we be saying "who does UNC have that was better than Grant Hill?". That makes things a lot more favorable for Duke. No?

Grant has 7 all-star appearances, a rookie of the year, 4 2nd team all-nbas and a 1st team all-nba. He's the best UNC/Duke NBA player since Jordan.

Next year, Kyrie will match Rasheed's all-star game appearances (4).

Obviously if you go back to Jordan/Worthy then UNC has everyone covered. They were all-timers. Since then? You could make the case that Duke has or will very soon surpass Carolina. If you just look at say... all-star game appearances I bet it's close. Duke is probably even ahead. And the future looks bright w/ Jabari, Okafor and Winslow all potential studs.

Duke95
04-21-2015, 04:44 PM
I don't think my response implied a particular type of statistic. I'm mostly trying to integrate my ideas of what a statistical model does with the types of models I've used in the past.

In my mind, whether the data is generated stochastically or generated deterministically, it is the output of the model that defines whether the model is a statistical model. While not the arbiter of everything, Wikipedia seems to agree with me:

"In mathematical terms, a statistical model is usually thought of as a pair (S, \mathcal{P}), where S is the set of possible observations, i.e. the sample space, and \mathcal{P} is a set of probability distributions on S.[2]"

In other words, a statistical model uses observations to generate probabilities.

In my mind that would be different than using statistical data to generate forecasts or predictions. Tell me where our differences lie. Thanks.

No, I think you are misinterpreting things. To start, statistical models are NOT deterministic. That is a distinguishing factor from many mathematical models. We generally use a statistical model in order to estimate the data-generating process, nearly always through some simplified or idealized form. The reason for this is because we do not have metaphysical certainty regarding the actual data generating process in nature. There is always uncertainty. We seek to understand how the outcome was generated, so we make some assumptions. What were the factors that determined, or generated, the win for a particular team? Was it offensive efficiency, defensive efficiency, was it both, what else was it? We can hypothesize these make a difference and formally test that. In this case, a model can generate a probability of victory for a team.

In other cases, we seek to understand what determined the prices for a company's product. There are supply factors, demand factors, capacity factors, competitive factors, etc. We take these into account and generate a model to explain prices. Then we can change the underlying data, for example, to see how prices would change if there were, say, one fewer competitors. This is often done in merger analysis to see if a particular merger would be potentially anti-competitive.

Different models for different purposes.

MChambers
04-21-2015, 04:49 PM
I'm too busy to look the rest up, but Elton Brand has more points and three times as many rebounds as Jerry Stackhouse. Moreover, Elton was a great player to have on your team (still is!) -- not necessarily so about Stackhouse.
Living in Washington DC, I'm painfully aware of Stackhouse's limitations, such as when he thought a one week beach house rental meant you could stay in the house for 8 days:

http://usatoday30.usatoday.com/sports/basketball/nba/wizards/2003-07-14-stackhouse-assault-charge_x.htm

And of course, the GM who traded a young Rip Hamilton for an old Stackhouse: Michael Jordan.

Maybe we should be discussing which school turns out better GMs? UNC has Kupchak; Duke has Ferry and King. Any others?

NSDukeFan
04-21-2015, 05:35 PM
I did use the qualifier for Chilcutt of "If you go by titles..." The reason I included him is that his title is no more or less significant than Danny Ferry's. And Ferry was, until 3 years ago, the only Duke player in the K era to ever play on an NBA championship team. Titles are what people view as "perceived NBA success"

But for the sake of argument, let's throw titles out the window and see if any other UNC players since Jordan had equal or better careers than Elton Brand. I'll stick to players who are retired or close to it, as the verdict is still out on active players. These are in addition to the athletes not named Pete in my list below:


Vince Carter was an 8x NBA All Star and 2x All NBA team. He had a streak of 23 straight games where he scored over 20 points. He's 25th in career scoring, right between Robert Parrish and Charles Barkley. I'd say that's objectively better than Elton.
Jerry Stackhouse was a 2x NBA All Star, averaging close to 30 ppg in 2001. He played for 18 years.
Antawn Jamison was also a 2x NBA All Star. He is 40th in NBA career scoring (Brand isn't ranked).


So the best NBA players to come out of UNC since Michael Jordan who had equal or better careers to Elton Brand are Rasheed Wallace, Vince Carter, Jerry Stackhouse, Antawn Jamison, and Rick Fox. None of those players were Michael Jordan for sure, but it's not accurate to say UNC has produced no one of consequence since MJ.

As SageGrouse says, James Worthy was also an "Alpha Dog" from UNC who won 3 titles and was a starter on those teams. So UNC's produced two, not one, since 1980. No other school can claim that.

I hate the fact that UNC players have had so much success in the NBA, especially when compared head to head with our guys. Some of that is due to luck - we've had lots of bad (JWill, Hurley, Hill with injuries) and they have had their share of good (Fox, Chilcutt, Haywood). But even if you take out Jordan/Worthy and only look 2000 and beyond, their cupboard is hardly bare.

I'm not sure that Rasheed had a better career than Brand unless you value who he played with above all else. Elton has more points, rebounds and blocks in fewer games. I don't think Rasheed made an all-NBA team, but Brand was 2nd team all-NBA. And Rick Fox? Come on. I guess Robert Horry had a better career than Karl Malone. Stackhouse was a 2 time all-star, like Brand and had the two big scoring seasons, but I don't know if that trumps Brand's career excellence. Although Stackhouse played 18 years, that hasn't stopped Brand from having more career points. I agree with your overall point that UNC has had better NBA players than Duke until now, but remember Brand averaged close to 20 and 10 the first 8 years of his career. It's not his fault he didn't always have great teammates.

Bostondevil
04-21-2015, 06:48 PM
Others include Vince Carter, Jerry Stackhouse, Brad Daugherty, and Antawn Jamison, all of whom are/were as good or better than Brand in the NBA.

OK.

Next question - isn't the knock on Duke players that they don't have the NBA careers that everybody expects? Did Vince Carter or Jerry Stackhouse or Antawn Jamison have the NBA careers that everybody expected? (I do think that Daugherty had the career that everybody expected.)

hughgs
04-21-2015, 09:09 PM
No, I think you are misinterpreting things. To start, statistical models are NOT deterministic. That is a distinguishing factor from many mathematical models. We generally use a statistical model in order to estimate the data-generating process, nearly always through some simplified or idealized form. The reason for this is because we do not have metaphysical certainty regarding the actual data generating process in nature. There is always uncertainty. We seek to understand how the outcome was generated, so we make some assumptions. What were the factors that determined, or generated, the win for a particular team? Was it offensive efficiency, defensive efficiency, was it both, what else was it? We can hypothesize these make a difference and formally test that. In this case, a model can generate a probability of victory for a team.

In other cases, we seek to understand what determined the prices for a company's product. There are supply factors, demand factors, capacity factors, competitive factors, etc. We take these into account and generate a model to explain prices. Then we can change the underlying data, for example, to see how prices would change if there were, say, one fewer competitors. This is often done in merger analysis to see if a particular merger would be potentially anti-competitive.

Different models for different purposes.

I don't see how what I said is misinterpreting anything that you've said. I never stated that statistical models are deterministic. I simply said that I believed that statistical models are based on their output, not on how the output was generated. You may disagree with that statement, which is why I asked what were our differences.

However, you haven't answered my question. I understand that different models perform different functions and internally produce data in different ways. What I'm trying to understand is the definition of a statistical model.

My understanding has always been that a statistical model is a model that generates probabilities of events. I quoted the Wikipedia article to show that my understanding is not in left field. Furthermore, I've always distinguished statistical models from other models by saying that other models generate predictions (or scientific hypotheses).

So, what is your definition of a statistical model? Is it any model that uses statistics to generate data? I've created plenty of engineering models that use statistics and would never claim they were statistical models. And given your definition of a statistical model is it something that is specific to your industry? The models I've created in engineering have used plenty of statistics but no one would ever refer to them as statistical models. Thanks.

ice-9
04-21-2015, 11:06 PM
I looked at the article cited in the original post and didn't see where it was saying that their model predicted anything. Can you point out where 538 is using statistics to predict something? Thanks.

It's right at the top of the page.


FiveThirtyEight’s men’s and women’s NCAA tournament forecasting models calculate the chance of each team reaching each round, taking into account a composite of power rankings, pre-season rankings, the team’s placement on the NCAA’s S-curve, player injuries and geography, where data is available.

There's also the title of the article.


2015 March Madness Predictions

Duke95
04-21-2015, 11:47 PM
I don't see how what I said is misinterpreting anything that you've said. I never stated that statistical models are deterministic. I simply said that I believed that statistical models are based on their output, not on how the output was generated. You may disagree with that statement, which is why I asked what were our differences.

However, you haven't answered my question. I understand that different models perform different functions and internally produce data in different ways. What I'm trying to understand is the definition of a statistical model.

My understanding has always been that a statistical model is a model that generates probabilities of events. I quoted the Wikipedia article to show that my understanding is not in left field. Furthermore, I've always distinguished statistical models from other models by saying that other models generate predictions (or scientific hypotheses).

So, what is your definition of a statistical model? Is it any model that uses statistics to generate data? I've created plenty of engineering models that use statistics and would never claim they were statistical models. And given your definition of a statistical model is it something that is specific to your industry? The models I've created in engineering have used plenty of statistics but no one would ever refer to them as statistical models. Thanks.

First, a statistical model is not defined by its output. The output can be predictions, but those predictions can be generated by just randomly guessing just as easily as they can by a statistical model. A statistical model is a procedure that involves the application of probability distributions on the sample space. In other words, what we do is make a series of assumptions regarding these probability distribution in order to explain, generally, how the data generation process works. We may make normality assumptions, etc.. Note that what I mean is not that the OUTPUT is a probability (although it can, and often is), but rather that the assumptions are probabilistic in nature.

Peter McCullough has a pretty good article on this, appropriately titled "What is a statistical model?". If you go to the examples, you will see that, among the examples, he gives the logic and classical regression examples I offered in the previous post. They're among the most commonly used tools, so they make quite good examples.

Link: https://galton.uchicago.edu/~pmcc/pubs/AOS023.pdf

So, a model does not generate a hypothesis. A model can be used to test a hypothesis. If I want to test how well the data conform to a certain hypothesis, I can do that with a t-test, Wald, etc., etc. I can estimate a model to see the probability that I will see as large or larger a test statistic given my null is true (the p-value). Long story short, a statistical model that generates probabilities of events is only one type of such a model. There are many others that generate predicted values (e.g., predicted prices, etc.) not just probabilities. The definition is in the methodology, not the output.

hughgs
04-22-2015, 07:47 AM
It's right at the top of the page.


FiveThirtyEight’s men’s and women’s NCAA tournament forecasting models calculate the chance of each team reaching each round, taking into account a composite of power rankings, pre-season rankings, the team’s placement on the NCAA’s S-curve, player injuries and geography, where data is available.


The above quote doesn't say that the model is predicting the winner. It says it the model calculates "... the chance of each team ..." As I said previously, those are completely different things.

BobbyFan
04-22-2015, 08:03 AM
Others include Vince Carter, Jerry Stackhouse, Brad Daugherty, and Antawn Jamison, all of whom are/were as good or better than Brand in the NBA.

I'll give you Carter, but not the rest. Daugherty was an excellent offensive player, but soft on defense and had a short career. Stackhouse and, to some degree, Jamison compiled points on poor shooting throughout their careers; neither was in Brand's class.

Rasheed Wallace was a unique player that could be the perfect complement in the right situation. But he couldn't be relied upon to be as good a scoring option as Brand was. I wouldn't argue either way here.

ice-9
04-22-2015, 08:21 AM
The above quote doesn't say that the model is predicting the winner. It says it the model calculates "... the chance of each team ..." As I said previously, those are completely different things.

I'm not sure what you're trying to say, but I know what I'm trying to say, and I'm pretty sure 538's article is consistent with it.

hughgs
04-22-2015, 09:00 AM
First, a statistical model is not defined by its output. The output can be predictions, but those predictions can be generated by just randomly guessing just as easily as they can by a statistical model. A statistical model is a procedure that involves the application of probability distributions on the sample space. In other words, what we do is make a series of assumptions regarding these probability distribution in order to explain, generally, how the data generation process works. We may make normality assumptions, etc.. Note that what I mean is not that the OUTPUT is a probability (although it can, and often is), but rather that the assumptions are probabilistic in nature.

Peter McCullough has a pretty good article on this, appropriately titled "What is a statistical model?". If you go to the examples, you will see that, among the examples, he gives the logic and classical regression examples I offered in the previous post. They're among the most commonly used tools, so they make quite good examples.

Link: https://galton.uchicago.edu/~pmcc/pubs/AOS023.pdf

So, a model does not generate a hypothesis. A model can be used to test a hypothesis. If I want to test how well the data conform to a certain hypothesis, I can do that with a t-test, Wald, etc., etc. I can estimate a model to see the probability that I will see as large or larger a test statistic given my null is true (the p-value). Long story short, a statistical model that generates probabilities of events is only one type of such a model. There are many others that generate predicted values (e.g., predicted prices, etc.) not just probabilities. The definition is in the methodology, not the output.

First, thanks for pointing out that a model doesn't generate a hypotheses. I was trying to say what you said, but clearly got it wrong.

As for the article, I had browsed the article before I sent my first question about the definition of a statistical model. So, maybe I missed it, but I didn't see an example in the article where the output wasn't a probability. For example, when generating regression parameters, which would seem to be a non-statistical output, the models can generate a range for the regression parameter. However, that range is contingent on some probability. And so, you're not generating true prediction, you're really generating a probability that the parameter is within some range. Statistical software, such as R and JMP, readily calculate slope and intercept ranges, but those ranges are in terms of some probability (typically 95%).

Let me give you an example of a model that uses statistics but would never be considered a statistical model by my previous co-workers. We were trying to detect failure modes of a rotating piece of hardware. We threw together a model of the hardware along with additive noise and tried to predict the speed of the piece from position data. Generating the noise certainly requires some statistical assumptions, but no one ever thought we were generating a statistical model. BTW, don't try and predict speed from position. It doesn't work very well :).

So, I have the same predicament. I don't understand why my definition of a statistical model is wrong. Is it wrong because of the context? My context is in engineering. Are statistical models defined differently in econometrics? If they are, then how is my definition wrong.

I'm sure everyone else has bailed on this conversation, so thanks for sticking with it.

CDu
04-22-2015, 09:23 AM
OK.

Next question - isn't the knock on Duke players that they don't have the NBA careers that everybody expects? Did Vince Carter or Jerry Stackhouse or Antawn Jamison have the NBA careers that everybody expected? (I do think that Daugherty had the career that everybody expected.)

I would absolutely say that Carter exceeded his expectations. The guy went 5th in the draft and yet went on to score over 23,000 points (25th all time). Honestly, I can't see a reasonable argument as to how he didn't exceed expectations. He didn't win a championship until very late in his

Similar thing (but to a lesser degree) with Jamison. He went 4th in the draft with a lot of questions about his position in the NBA (he was a tweener: PF game in a SF body). Yet he managed to score over 20,000 points in the league.

Daugherty was putting together a really nice careeer on a perennial playoff team before back injuries ended his career much too early. Like Grant Hill, it is unfair to say he didn't meet expectations in the pros considering he was a 5-time All-Star and one-time All-NBA player before the back injuries forced him to retire. Basically, it's the same sort of issue that derailed Christian Laettner's career (though Laettner wasn't quite as good as Daugherty in the NBA prior to his injuries) and (though a different body part) Grant Hill's career (though Hill was decidedly better than Daugherty before his injuries).

Stackhouse is the only one that one could remotely reasonably argue may not have lived up to expectations. But even he had 16,000 points and had a couple of really good years including a 29.8ppg season in 2001. It's still a reach to say that Stackhouse didn't live up to expectations as he had a really long and productive career. Compare Stackhouse to all-time Duke greats like Dawkins and Ferry and it's hard to say he didn't do pretty darn well for himself.

Now, a fair argument can be made (and I'm making it now) that, since Carter and Jamison, none of UNC's players have done much of anything in the NBA. In other words, since Dean Smith stopped recruiting, UNC either hasn't gotten NBA-level talent or hasn't developed talent into NBA stars. But there really isn't a reasonable argument prior to Dean Smith's retirement that UNC didn't consistently produce better pros than Duke (for whatever that is worth).

Things have changed dramatically since then. Brand, Maggette, Boozer, Deng, and Irving have all had (or are in the process of having) stellar careers, while Battier, Dunleavy, Henderson, and Redick had or are putting together solid complementary careers. And Parker, Okafor, and Winslow look like they'll have very promising careers as well.

Duke95
04-22-2015, 09:41 AM
First, thanks for pointing out that a model doesn't generate a hypotheses. I was trying to say what you said, but clearly got it wrong.

As for the article, I had browsed the article before I sent my first question about the definition of a statistical model. So, maybe I missed it, but I didn't see an example in the article where the output wasn't a probability. For example, when generating regression parameters, which would seem to be a non-statistical output, the models can generate a range for the regression parameter. However, that range is contingent on some probability. And so, you're not generating true prediction, you're really generating a probability that the parameter is within some range. Statistical software, such as R and JMP, readily calculate slope and intercept ranges, but those ranges are in terms of some probability (typically 95%).

Let me give you an example of a model that uses statistics but would never be considered a statistical model by my previous co-workers. We were trying to detect failure modes of a rotating piece of hardware. We threw together a model of the hardware along with additive noise and tried to predict the speed of the piece from position data. Generating the noise certainly requires some statistical assumptions, but no one ever thought we were generating a statistical model. BTW, don't try and predict speed from position. It doesn't work very well :).

So, I have the same predicament. I don't understand why my definition of a statistical model is wrong. Is it wrong because of the context? My context is in engineering. Are statistical models defined differently in econometrics? If they are, then how is my definition wrong.

I'm sure everyone else has bailed on this conversation, so thanks for sticking with it.

To start off, regression parameters are indeed statistical output. These are just the slope parameters associated with each independent variable. In addition to those parameters, the model will usually generate the confidence bounds associated with each parameter at some researcher-specified level (e.g, 95%). The idea there is that, in repeated samples, the interval will contain the actual population parameter 95% of the time. So, the parameter itself is not calculated "in terms of the confidence interval". The interval is the range around the parameter. The parameter itself remains the same if you extend the confidence interval from 95% to 90%, for example.

What you were doing is indeed a statistical model (as I understood from what you were saying). By generating noise, I assume you mean some sort of Monte Carlo analysis or maybe just simple random number generation from a certain distribution. Statistical models don't have to be complex. They can be exceedingly simple in many cases.

CDu
04-22-2015, 10:08 AM
I'm not sure what you're trying to say, but I know what I'm trying to say, and I'm pretty sure 538's article is consistent with it.

With the caveat that I only think I know what you're trying to say, I think you are misinterpreting the article. The 538 model predicts the probability that each team has of winning. It does not predict the winner; just the chance that each team has of winning. And based on the model's predictions of Duke's chances, Duke was an unlikely winner (as was nearly every team in the tournament).

The title "2015 March Madness Predictions" is not saying that they are predicting the champion. They are predicting the chance of each team winning (as stated directly below the title).

There is a difference between saying that the 538 model predicts Team X will win (which is NOT what the 538 models do, nor is it what 538 purports it to do) and saying that the 538 model predicts the chances that Team X will win (which IS what the 538 model does and what 538 says it does).

The 538 model predicted that Kentucky had the best chance of winning. It did not predict that Kentucky would win, nor did they say it predicted Kentucky would win. It just predicted that, if the tournament was played 1,000 times, Kentucky was likely to win more times than any other individual team. Now, if you were to use that model to predict a winner, you'd pick Kentucky (because the model predicted Kentucky had the best chance of winning). But you'd do so with the understanding that even Kentucky was unlikely (<50% chance) to win it.

Some readers may have taken it wrong, but that's the case. The model didn't predict a winner. It predicted the chances each team had of winning. And based on those predictions, no team was a likely winner (all were unlikely, and all but Kentucky were very unlikely).

ice-9
04-22-2015, 11:30 AM
With the caveat that I only think I know what you're trying to say, I think you are misinterpreting the article. The 538 model predicts the probability that each team has of winning.

In my (and 538's) mind, it's the same thing. Probability-based predictions of winning are still that, predictions of winning.

CDu
04-22-2015, 11:33 AM
In my (and 538's) mind, it's the same thing. Probability-based predictions of winning are still that, predictions of winning.

I think it is presumptuous of you to say what is in 538's mind.

And your second sentence simply isn't true. Predicted probability of winning is not the same thing as predicting the winner. They didn't predict the winner. They predicted the chances each team had of winning. Nowhere do they say "we predict that Kentucky (or any other team) will win". Nor do they even appear to imply that.

Duke95
04-22-2015, 11:43 AM
I think it is presumptuous of you to say what is in 538's mind.

And your second sentence simply isn't true. Predicted probability of winning is not the same thing as predicting the winner. They didn't predict the winner. They predicted the chances each team had of winning. Nowhere do they say "we predict that Kentucky (or any other team) will win". Nor do they even appear to imply that.

If I can interject for a second, that is true. I don't know what technique 538 is using, but what you have here is a limited dependent outcome: 1=win, 0=loss. Two commonly used ways to model that outcome are logit and probit models. Both will generate a probability outcome, that is, a value lying between 0 and 1. They will not generate just 1 or 0.

sagegrouse
04-22-2015, 11:48 AM
I think it is presumptuous of you to say what is in 538's mind.

And your second sentence simply isn't true. Predicted probability of winning is not the same thing as predicting the winner. They didn't predict the winner. They predicted the chances each team had of winning. Nowhere do they say "we predict that Kentucky (or any other team) will win". Nor do they even appear to imply that.

Easy there, big guy, easy! Predicting the probability of winning (and losing) is a more general form of predicting the winner (and loser). It provides more information to the recipient.

CDu
04-22-2015, 11:57 AM
If I can interject for a second, that is true. I don't know what technique 538 is using, but what you have here is a limited dependent outcome: 1=win, 0=loss. Two commonly used ways to model that outcome are logit and probit models. Both will generate a probability outcome, that is, a value lying between 0 and 1. They will not generate just 1 or 0.

Which is precisely why I'm saying that 538's model isn't predicting the winner, nor are they saying that the model predicted Kentucky would win. They predict the probability of several outcomes (3rd round appearance, Sweet-16, Elite-8, Final Four, championship game, champion) for every team. They predict that there was a small chance Kentucky could lose very early. Their model predicts that Kentucky was the most likely of any of the teams to win it, but that even Kentucky was unlikely to win it. That is very different from predicting that Kentucky would win it.

Note: given that there are a series of games with binary outcomes being played here, each dependent on other games, I suspect they are not using a simple logit or probit model. If it were just one game, then yes, a logit or probit model would be fine. But I suspect, given that the model is dependent on 63 games (well, I guess technically 67, but realistically 63) they are using a decision-tree modelling approach and running simulations to get their predicted probabilities. But that's just a guess. And it could be that they use a logit/probit for each game. But there are a ton of different potential matchups that they'd have to run a logit/probit on (increasing in number for each subsequent round), which makes me think they probably didn't go that route.

CDu
04-22-2015, 12:06 PM
Easy there, big guy, easy! Predicting the probability of winning (and losing) is a more general form of predicting the winner (and loser). It provides more information to the recipient.

The model "predicts the winner" only in the sense that it says the winner (with a sample size of 1) will be partly Kentucky, partly Wisconsin, partly Villanova, part... etc. It predicts the probability that a given "roll of a 68-sided die" turns out a certain way. If the model had said one team would win the tournament with greater than 50% probability, then maybe you could argue that the model predicted the winner to be that team. Though I wouldn't argue that with much confidence unless the predicted probability was WAY higher than 50%, which is essentially a coin flip.

Let's take the coin flip example. Roughly speaking, a coin flip has a 50% chance of being heads and 50% tails. So any reasonable model is going to predict that the probable outcome of a coin flip will be heads ~50% and tails ~50%. You then flip a coin and it lands heads. Does that mean the model was wrong for saying that tails had a 50% chance of being the winner? Of course not.

More importantly, the model didn't predict that either heads or tails would win that flip. It "knows" that each iteration has some random chance of favoring one side or the other. So the model just says that, if you flipped a sufficiently large number of times (so as to normalize away the effects of random chance), you'd likely see ~50% of the outcomes being heads and ~50% being tails. Same thing applies to the 538 (and other models). The model says that, if we ran the same tournament 1,000 times (or more), we'd likely see Kentucky win it ~400 times, and a couple of other teams win it ~100 times, still more teams would win it ~50-75 times, and a scattering of other teams from there.

Duke95
04-22-2015, 12:20 PM
Which is precisely why I'm saying that 538's model isn't predicting the winner, nor are they saying that the model predicted Kentucky would win. They predict the probability of several outcomes (3rd round appearance, Sweet-16, Elite-8, Final Four, championship game, champion) for every team. They predict that there was a small chance Kentucky could lose very early. Their model predicts that Kentucky was the most likely of any of the teams to win it, but that even Kentucky was unlikely to win it. That is very different from predicting that Kentucky would win it.

Note: given that there are a series of games with binary outcomes being played here, each dependent on other games, I suspect they are not using a simple logit or probit model. If it were just one game, then yes, a logit or probit model would be fine. But I suspect, given that the model is dependent on 63 games (well, I guess technically 67, but realistically 63) they are using a decision-tree modelling approach and running simulations to get their predicted probabilities. But that's just a guess. And it could be that they use a logit/probit for each game. But there are a ton of different potential matchups that they'd have to run a logit/probit on (increasing in number for each subsequent round), which makes me think they probably didn't go that route.

Well, rather than use a simulation, I expect they've looked at each team's entire season, modeled their 1/0 win/loss outcome on a series of variables, both theirs and their opponents' (and others), and now, they entered in the data for each opponent (or prospective opponent) in the tournament and predicted the outcome. Then, they calculate the probabilities at each step, which are conditional on those in the previous step.

That's what I expect they did, but I don't know.

CDu
04-22-2015, 12:53 PM
Well, rather than use a simulation, I expect they've looked at each team's entire season, modeled their 1/0 win/loss outcome on a series of variables, both theirs and their opponents' (and others), and now, they entered in the data for each opponent (or prospective opponent) in the tournament and predicted the outcome. Then, they calculate the probabilities at each step, which are conditional on those in the previous step.

That's what I expect they did, but I don't know.

Certainly possible, but that seems like a lot of individual models to run given the number of different matchups that can occur in each round of the tournament. It would seem much easier to just build it as a decision-tree model in which you have an attribute weight (with distributional assumption) and an attribute weight (with distributional assumption) for the location of the game being played, and just run it as a simulation model. Way less work than to run all the different logit/probit models for each possible matchup in each round and then going back and estimating the probability of reaching round X for each team based on all the possible matchups for that team.

All that said, I think either is possible. Just that if I were doing it, I definitely wouldn't go the route of the logit/probit model approach.

Bostondevil
04-22-2015, 01:08 PM
I would absolutely say that Carter exceeded his expectations. The guy went 5th in the draft and yet went on to score over 23,000 points (25th all time). Honestly, I can't see a reasonable argument as to how he didn't exceed expectations.


You know what, I can make one.

Vince Carter went 5th in the 1998 draft. I went and looked up that draft. Yes, he did much better than the #1 overall - Michael Olowokandi and, like I said before, I don't really follow the NBA so I can't comment on Mike Bibby (#2) or Raef LaFrentz (#3). Antawn Jamison went 4th. Then Vince. (Even I know Olowokandi was one of the all time #1 overall busts.)

But do you know who went 9th? Dirk Nowitzki. And 10th? Paul Pierce. (I do know their careers, both are future Hall of Famers. Plus Celtics!)

Both of those guys had way better careers than Vince Carter. And they went lower in the draft. Now, sure, the draft is a bit of a crapshoot. But based on where he went in the draft, experts thought he was going to be a better player than either Nowitzki or Pierce. He was not.

The expectations, fair or not, on Vince Carter were that he was the next Michael Jordan (anybody remember him being called Baby Jordan, something like that?) and he wasn't. So even though he had a very good NBA career, he didn't live up to the expectations.

Indoor66
04-22-2015, 01:31 PM
Maybe we should just give back the trophy.

CDu
04-22-2015, 02:00 PM
You know what, I can make one.

Vince Carter went 5th in the 1998 draft. I went and looked up that draft. Yes, he did much better than the #1 overall - Michael Olowokandi and, like I said before, I don't really follow the NBA so I can't comment on Mike Bibby (#2) or Raef LaFrentz (#3). Antawn Jamison went 4th. Then Vince. (Even I know Olowokandi was one of the all time #1 overall busts.)

But do you know who went 9th? Dirk Nowitzki. And 10th? Paul Pierce. (I do know their careers, both are future Hall of Famers. Plus Celtics!)

Both of those guys had way better careers than Vince Carter. And they went lower in the draft. Now, sure, the draft is a bit of a crapshoot. But based on where he went in the draft, experts thought he was going to be a better player than either Nowitzki or Pierce. He was not.

Carter was, at worst, the 3rd best player in the draft (one could certainly argue for him over Paul Pierce; that's at least debatable). He went 5th. So he was a value pick at #5. Doesn't matter that the two possible guys ahead of him went later; he more than met expectations.


The expectations, fair or not, on Vince Carter were that he was the next Michael Jordan (anybody remember him being called Baby Jordan, something like that?) and he wasn't. So even though he had a very good NBA career, he didn't live up to the expectations.

Pretty sure this is revisionist history. When Carter came out, he was most certainly not considered the next Michael Jordan. He was talented, but he hadn't exactly dominated in college. If he was considered the next Jordan, he wouldn't have gone #5. It wasn't until he was in the NBA and started putting up monster highlights that the Jordan comparisons started to come up. And "Baby Jordan" was Harold Miner, not Vince Carter.

Duke95
04-22-2015, 02:27 PM
Certainly possible, but that seems like a lot of individual models to run given the number of different matchups that can occur in each round of the tournament. It would seem much easier to just build it as a decision-tree model in which you have an attribute weight (with distributional assumption) and an attribute weight (with distributional assumption) for the location of the game being played, and just run it as a simulation model. Way less work than to run all the different logit/probit models for each possible matchup in each round and then going back and estimating the probability of reaching round X for each team based on all the possible matchups for that team.

All that said, I think either is possible. Just that if I were doing it, I definitely wouldn't go the route of the logit/probit model approach.

Well, I think we disagree here. I would not run a decision tree approach. Why simulate the data, when you have already so much data collected from previous games? And you would not have to run the logit/probit over and over. You would just have the parameters from the 64 teams, for which yes, you would have to run a model that includes the data from their previous games this season: one model for each team.

Once you run the models, all you need are the inputs for the independent variables. You don't run the model again, because you already have the parameters.

Bostondevil
04-22-2015, 02:28 PM
Carter was, at worst, the 3rd best player in the draft (one could certainly argue for him over Paul Pierce; that's at least debatable). He went 5th. So he was a value pick at #5. Doesn't matter that the two possible guys ahead of him went later; he more than met expectations.



Pretty sure this is revisionist history. When Carter came out, he was most certainly not considered the next Michael Jordan. He was talented, but he hadn't exactly dominated in college. If he was considered the next Jordan, he wouldn't have gone #5. It wasn't until he was in the NBA and started putting up monster highlights that the Jordan comparisons started to come up. And "Baby Jordan" was Harold Miner, not Vince Carter.

Kinda can't believe we're arguing about this - I mean really, it's Vince Carter.

Michael Jordan, if you recall, went 3rd, so sure, the next Michael Jordan could go 5th! ;-)

Debatable that Carter was better than Pierce is (still active, btw)!?! NO IT IS NOT! (Am I wearing my Celtic green goggles today? Apparently so.) It is debatable as to whether Carter is a Hall of Famer. Pierce is a Hall of Famer. So is Nowitzki. No debate necessary.

I also think that you and I define meeting expectations differently. I maintain that Carter had a very good NBA career but he was supposed to have a great one ergo, expectations not met. Carter was called the next Jordan at one point although I believe you that it was not right out of college. Whenever it was, the comparisons did not pan out. Expectations not met. But I will agree to disagree with you on this one.

But I hope you will agree with me that Antawn Jamison who went 4th in the 1998 draft did not meet expectations. He went one spot ahead of Carter. He wasn't better than Carter! (And also no real debate on his HOF worthiness, he's not getting in.)

I found this article on Vince Carter which is both relevant to the argument at hand and kinda on the spooky side how much so. http://grantland.com/features/the-case-vinsanity-hall-famer-lakers-defensive-mess-nba-loudest-arena-jeremy-lin-doing-right/

Bostondevil
04-22-2015, 02:47 PM
I found this too from 3 years ago.

Still a fun read.

http://thegloriousextrapass.blogspot.com/2012/08/assessing-hall-of-fame-chances-of.html

Bostondevil
04-22-2015, 02:52 PM
And to further my googling - Vince Carter is still playing? Who knew? Not me!

Kedsy
04-22-2015, 03:06 PM
Kinda can't believe we're arguing about this - I mean really, it's Vince Carter.

Michael Jordan, if you recall, went 3rd, so sure, the next Michael Jordan could go 5th! ;-)

Debatable that Carter was better than Pierce is (still active, btw)!?! NO IT IS NOT! (Am I wearing my Celtic green goggles today? Apparently so.) It is debatable as to whether Carter is a Hall of Famer. Pierce is a Hall of Famer. So is Nowitzki. No debate necessary.

I also think that you and I define meeting expectations differently. I maintain that Carter had a very good NBA career but he was supposed to have a great one ergo, expectations not met. Carter was called the next Jordan at one point although I believe you that it was not right out of college. Whenever it was, the comparisons did not pan out. Expectations not met. But I will agree to disagree with you on this one.

But I hope you will agree with me that Antawn Jamison who went 4th in the 1998 draft did not meet expectations. He went one spot ahead of Carter. He wasn't better than Carter! (And also no real debate on his HOF worthiness, he's not getting in.)

I found this article on Vince Carter which is both relevant to the argument at hand and kinda on the spooky side how much so. http://grantland.com/features/the-case-vinsanity-hall-famer-lakers-defensive-mess-nba-loudest-arena-jeremy-lin-doing-right/

I wasn't going to weigh in on this debate, but I can't stop myself.

First of all, Carter and Jamison were drafted 4th and 5th and then immediately traded for each other. To say that Jamison didn't meet expectations because his career wasn't better than Carter's is kind of silly.

Second, just because someone picked later in the draft (e.g., Nowitzki) wildly exceeded expectations doesn't mean everyone drafted before him failed to meet expectations. That's a non-sequitur.

Finally, when an NBA team picks 4th or 5th in the NBA draft, the expectation is the pick will provide an NBA starter for a few years. Anyone who makes an All Star team or All NBA, or who scores 29+ points a game is exceeding expectations. By a lot. You don't need to have a "great career" to meet expectations, quite frankly if a player taken in that slot has a 10+ year NBA career as a productive player (as both Carter and Jamison did) then that 4th or 5th pick exceeded expectations. And it's ridiculous even to imply any player would need a solid Hall of Fame resume to meet expectations.

For comparison's sake, I give you the 4th and 5th picks in the last five NBA drafts:



Year 4th 5th
2014 Aaron Gordon Dante Exum
2013 Cody Zeller Alex Len
2012 Dion Waiters Thomas Robinson
2011 Tristan Thompson Jonas Valanciunas
2010 Wesley Johnson DeMarcus Cousins


There's one star (although I'm far from confident Cousins could have a better career than either Carter or Jamison), one solid starter, a couple part-time starters, and a couple potential starters in a few years. Compared to this list, Carter and Jamison greatly exceeded any rational expectation one might have for that pick. It's not even close.

CDu
04-22-2015, 03:20 PM
Kinda can't believe we're arguing about this - I mean really, it's Vince Carter.

Michael Jordan, if you recall, went 3rd, so sure, the next Michael Jordan could go 5th! ;-)

Debatable that Carter was better than Pierce is (still active, btw)!?! NO IT IS NOT! (Am I wearing my Celtic green goggles today? Apparently so.) It is debatable as to whether Carter is a Hall of Famer. Pierce is a Hall of Famer. So is Nowitzki. No debate necessary.

Vince Carter is one of the top 25 scorers of all time. The only guys ahead of him that aren't in the Hall of Fame simply aren't eligible yet and will get there. He's going to get in the Hall of Fame.


I also think that you and I define meeting expectations differently. I maintain that Carter had a very good NBA career but he was supposed to have a great one ergo, expectations not met. Carter was called the next Jordan at one point although I believe you that it was not right out of college. Whenever it was, the comparisons did not pan out. Expectations not met. But I will agree to disagree with you on this one.

Now you are moving the bar. We are talking about meeting expectations out of college, no? Seems unfair to shift to "expectations at some point in the future after college". I mean, the fact that Carter went from "not expected to be the next Michael Jordan" after college to "expected to be the next Michael Jordan" some time a few years later says he improved upon where people thought he was coming out of college.

In any case, there is just no reasonable argument that a guy who scored over 23,000 points and is almost certainly going to go to the Hall of Fame did not meet expectations coming out of college.


But I hope you will agree with me that Antawn Jamison who went 4th in the 1998 draft did not meet expectations. He went one spot ahead of Carter. He wasn't better than Carter! (And also no real debate on his HOF worthiness, he's not getting in.)

I won't agree actually. If you are setting the bar of meeting expecations at "did he or didn't he make the Hall of Fame?", then I absolutely argue that your definition of meeting expectations is pretty wildly skewed. I think Jamison absolutely met expectations. I don't think it is fair to say that, simply because he didn't have a Hall of Fame career like 3 of the guys in his draft class did that he somehow didn't meet expectations. Jamison has had a terrific career. There were big questions as to whether he could translate to the NBA given that he was essentially a center in college but would have to transition from there to SF/PF in the pros. How did that work out for a guy like Joe Smith, for example?

Jamison certainly didn't exceed expectations wildly in the same way that guys like Carter, Nowitzki, and Pierce did. But he absolutely met expectations and I'd argue exceeded them some too. He was a terrific player for a very long time in the league despite the questions about his position. He developed a fairly reliable perimeter shot (this was a guy who made his living shooting 5-8 foot jump-hooks faster than you could blink in college).

subzero02
04-22-2015, 03:38 PM
You know what, I can make one.

Vince Carter went 5th in the 1998 draft. I went and looked up that draft. Yes, he did much better than the #1 overall - Michael Olowokandi and, like I said before, I don't really follow the NBA so I can't comment on Mike Bibby (#2) or Raef LaFrentz (#3). Antawn Jamison went 4th. Then Vince. (Even I know Olowokandi was one of the all time #1 overall busts.)

But do you know who went 9th? Dirk Nowitzki. And 10th? Paul Pierce. (I do know their careers, both are future Hall of Famers. Plus Celtics!)

Both of those guys had way better careers than Vince Carter. And they went lower in the draft. Now, sure, the draft is a bit of a crapshoot. But based on where he went in the draft, experts thought he was going to be a better player than either Nowitzki or Pierce. He was not.

The expectations, fair or not, on Vince Carter were that he was the next Michael Jordan (anybody remember him being called Baby Jordan, something like that?) and he wasn't. So even though he had a very good NBA career, he didn't live up to the expectations.

Vince Carter easily surpassed any realistic expectations placed upon him before the draft. His spectacular play early in his NBA career led to excessively lofty expectations amongst fans and some analysts. He's a great player and was truly a dominant force for several years. In their primes, Dirk was the best amongst the 3 but an argument can be made that Carter was better than Pierce. Pierce is better overall due to a higher level of play over a longer period of time.

CDu
04-22-2015, 03:53 PM
Vince Carter easily surpassed any realistic expectations placed upon him before the draft. His spectacular play early in his NBA career led to excessively lofty expectations amongst fans and some analysts. He's a great player and was truly a dominant force for several years. In their primes, Dirk was the best amongst the 3 but an argument can be made that Carter was better than Pierce. Pierce is better overall due to a higher level of play over a longer period of time.

I'd agree with this and would say that all three of these players (to varying degrees) greatly exceeded their expectations coming into the draft. I'd say that Antawn Jamison met or exceeded his expectations coming into the draft.

Just to make it clear that I'm not on some UNC lean here, I'd say that Elton Brand has certainly met (if not exceeded) expectations coming into the draft, even given that he went #1. I'd say that Boozer wildly exceeded expectations coming into the draft. Same for Deng. Same for Maggette. Irving is well on his way to exceeding expectations. Dunleavy has met expectations. Redick has met expectations (perhaps even exceeded). Henderson has met them. Battier met expectations. Avery did not. Rivers has not (so far).

In fact, most of the Duke superstars who didn't meet expectations failed to do so mainly because of injury rather than performance. Dawkins, Hill, Hurley, and Williams certainly had their careers derailed early by injury (and Hill was still a really great player for several years prior to the injuries). Laettner had injuries later that kept him from really reaching expectations, but he wasn't wildly far off. For me, only Danny Ferry really falls in the category of a Duke superstar in the Coach K who failed to live up to NBA expectations without major injury being the culprit. Your mileage may vary in how you regard Shane Battier, Shelden Williams and JJ Redick. I'm disappointed that Williams didn't stick longer as a role player in the NBA, but I certainly wasn't surprised that none of these guys became an NBA star (given their relative size/athleticism for their positions).

CDu
04-22-2015, 04:41 PM
As a final comment around the "expectations topic", I'll add this olive branch:
- Aside from a handful of players (Michael Jordan, James Worthy, Bob McAdoo, and Vince Carter), I would say that UNC's players have had fairly similar NBA success as individual players to that of Duke players. So while the old argument that UNC players fare better than Duke players in the NBA was true, it was almost entirely based on a handful of players (and most largely based on just two players).
- The tide has absolutely shifted since the 1999 draft. Since that draft, there is absolutely no question that Duke players have outperformed UNC players. And there is no end to that trend in sight. And that is even with the fact that one of our best talents had a horrific motorcycle injury before he could become a star in the NBA.
- As such, I don't think even UNC fans (at least not reasonable ones) are trotting that argument out much anymore, as it is becoming more and more hollow. Jordan has been retired for over a decade. Worthy hasn't played in two decades. McAdoo hasn't played in 3 decades. And Carter is on his last legs. If UNC are trotting out that argument, the retort is pretty simple: what have you done for me lately?

As an aside, I think that pretty soon we're going to hit a point where the topic is meaningless. For almost a decade, the best talent wasn't going to the NBA at all, and if they did most only stayed one year. So a lot of the Hall of Famers coming up were straight from high school guys (James, Garnett, Bryant). And now with the one-and-done era upon us, that's becoming even more true. Guys like Anthony Davis, Irving, Rose, Durant, Love, Bosh, etc. are the most likely candidates. There are exceptions (Harden, Westbrook, Curry, Wade, etc.). But I think as we get further and further into the one-and-done world, those exceptions will become less and less common.

As such, I really don't buy the idea that colleges are developing players the way they were when guys stayed 3-4 years. Now, it is about working to mesh the talent of the players while you have them. There is certainly some room for development, but I'd argue that for the most part these guys are developing into NBA players outside of their time at college. Calipari didn't make Anthony Davis or Derrick Rose a superstar; Davis and Rose did that on their own. Ben Howland didn't make Kevin Love a superstar, Love did. Coach K didn't make Irving a superstar, Irving did. Rick Barnes didn't make Kevin Durant great, Boeheim didn't make Carmelo great, and so on.

One could make the same argument about the olden days too: that it has always been up to the player first and foremost, not the coach. And honestly, I buy into this argument. I don't think Dean Smith made Michael Jordan a great player. I think Michael Jordan made Michael Jordan a great player. But I think it is a lot harder to dispel that argument now than it was when college coaches were responsible for some of the most formative years of these players.

Bostondevil
04-22-2015, 06:31 PM
Vince Carter easily surpassed any realistic expectations placed upon him before the draft. His spectacular play early in his NBA career led to excessively lofty expectations amongst fans and some analysts. He's a great player and was truly a dominant force for several years. In their primes, Dirk was the best amongst the 3 but an argument can be made that Carter was better than Pierce. Pierce is better overall due to a higher level of play over a longer period of time.

Emphasis mine.

Pierce has had a better NBA career than Carter. (That's what I mean when I say Pierce is a better player.) Since we are debating it, I guess it is debatable. You can give me all the stats you want to and I'll admit that I didn't watch Carter play in the pros nearly as much as Pierce but, I have seen Carter play and I've seen him play in games where he just quit on teams. It's not really a debate.

And as to expectations, OK, I concede. They lived up to their athletic promise. However, I expected Carter and Jamison to continue the Worthy/Jordan dominance of UNC players in the NBA (and I dreaded it). So, they did not live up to MY expectations, but, I suspect that will keep neither of them awake at night.

I do notice though, that we pretty quickly zeroed in on Carter and to a lesser extent Jamison - what about Rick Fox? :)

CDu
04-22-2015, 06:44 PM
Emphasis mine.

Pierce has had a better NBA career than Carter. (That's what I mean when I say Pierce is a better player.) Since we are debating it, I guess it is debatable. You can give me all the stats you want to and I'll admit that I didn't watch Carter play in the pros nearly as much as Pierce but, I have seen Carter play and I've seen him play in games where he just quit on teams. It's not really a debate.

And as to expectations, OK, I concede. They lived up to their athletic promise. However, I expected Carter and Jamison to continue the Worthy/Jordan dominance of UNC players in the NBA (and I dreaded it). So, they did not live up to MY expectations, but, I suspect that will keep neither of them awake at night.

I do notice though, that we pretty quickly zeroed in on Carter and to a lesser extent Jamison - what about Rick Fox? :)

You will get no argument from me that Fox was a lesser player than Brand and others. I think the only reason Fox came up was when someone listed titles won. But Fox was one of the poster children for riding the coattails of Hall of Famers in getting those titles.

Neals384
04-22-2015, 09:07 PM
As such, I really don't buy the idea that colleges are developing players the way they were when guys stayed 3-4 years. Now, it is about working to mesh the talent of the players while you have them. There is certainly some room for development, but I'd argue that for the most part these guys are developing into NBA players outside of their time at college. Calipari didn't make Anthony Davis or Derrick Rose a superstar; Davis and Rose did that on their own. Ben Howland didn't make Kevin Love a superstar, Love did. Coach K didn't make Irving a superstar, Irving did. Rick Barnes didn't make Kevin Durant great, Boeheim didn't make Carmelo great, and so on.

One could make the same argument about the olden days too: that it has always been up to the player first and foremost, not the coach. And honestly, I buy into this argument. I don't think Dean Smith made Michael Jordan a great player. I think Michael Jordan made Michael Jordan a great player. But I think it is a lot harder to dispel that argument now than it was when college coaches were responsible for some of the most formative years of these players.

I think it depends. Justise grew so much in this one season with Duke - his dominating offense was always there, but his defense, his ability to stay out of foul trouble (or play with it), his ability to handle adversity, and his determination to play every possession as hard as he can really improved during the season. Justise is going to have a great NBA career and his one year with Duke will help make it happen sooner.

Jahlil had a fully-developed post game, but did improve in other areas during the year as well. He will also have a great NBA career, and the year with Duike helped, but not as much as it did Justise.

Tyus was our only OAD this season who came in a pretty much a finished product.

Henderson
04-22-2015, 09:18 PM
I think it depends. Justise grew so much in this one season with Duke - his dominating offense was always there, but his defense, his ability to stay out of foul trouble (or play with it), his ability to handle adversity, and his determination to play every possession as hard as he can really improved during the season. Justise is going to have a great NBA career and his one year with Duke will help make it happen sooner.

Jahlil had a fully-developed post game, but did improve in other areas during the year as well. He will also have a great NBA career, and the year with Duike helped, but not as much as it did Justise.

Tyus was our only OAD this season who came in a pretty much a finished product.

Agree that both Justise and Jahlil benefited from their years. But I thought Tyus also improved immensely during his time at Duke -- in the mental aspect of the game. He got a lot stronger mentally, tougher, more confident. And I have to believe that the Duke coaching staff played a huge role in that.

ice-9
04-22-2015, 10:13 PM
I think it is presumptuous of you to say what is in 538's mind.

And your second sentence simply isn't true. Predicted probability of winning is not the same thing as predicting the winner. They didn't predict the winner. They predicted the chances each team had of winning. Nowhere do they say "we predict that Kentucky (or any other team) will win". Nor do they even appear to imply that.


I can just see the editorial process at 538. "Hey guys, let's make an article where we don't predict winners and title it 'March Madness Predictions,' where we predict the chances of a team winning. But of course that's not actually predicting a winner. Because assigning a percentage between 0 and 1 to the chances of winning, instead of actually assigning 0 or 1, means we're not actually predicting a winner. Because to assign 0 or 1 probability is dumb and way too declarative, whereas something between 0 and 1 is sophistication."

---

Bob: "Hey Jeff, who do you think will win the Bulls-Bucks game?"

Me: "Hmm, the Bulls have a 90% chance of winning, based on...stuff."

Now what the hell did I just do? Did I not just predict a winner, albeit with a major qualifier?

Come on guys. It's not that difficult.

---

Sorry if I'm sounding a little cranky. Rough few days.

freshmanjs
04-22-2015, 10:23 PM
Bob: "Hey Jeff, who do you think will win the Bulls-Bucks game?"

Me: "Hmm, the Bulls have a 90% chance of winning, based on...stuff."

Now what the hell did I just do? Did I not just predict a winner, albeit with a major qualifier?

Come on guys. It's not that difficult.

---

I can also see the editorial process at 538. "Hey guys, let's make an article where we don't predict winners and title it 'March Madness Predictions,' of which said predictions refer to what we predict to be the chances of a team winning. Because of course that's not actually predicting a winner. Because we put a percentage between 0 and 1 to the chances of winning, instead of actually putting in 0 or 1. No way the former is predicting a winner."

---

Sorry if I'm sounding a little cranky. Rough few days.

correct. you did not predict a winner. you described the probabilities of outcomes.

CDu
04-22-2015, 10:30 PM
correct. you did not predict a winner. you described the probabilities of outcomes.

Exactly! Now, if you say "well the Bulls appear to have a 90% chance of winning, so I predict that the Bulls will win", then you have predicted a winner. Otherwise, you have just described the chances of the Bulls winning.

Compare that with the 538 models of the election. In that case, the models so strongly predicted Obama would win that Silver felt confident in predicting that Obama would win the actual election. No such claims were made with the tournament, because that would be folly (as no team is a favorite to win it against the field).

ice-9
04-23-2015, 03:37 AM
correct. you did not predict a winner. you described the probabilities of outcomes.

I guess you know what I meant better than I do.

This seems like yet another meaningless argument about semantics (to me) so I'll just bow out.

sagegrouse
04-23-2015, 06:05 AM
correct. you did not predict a winner. you described the probabilities of outcomes.


Exactly! Now, if you say "well the Bulls appear to have a 90% chance of winning, so I predict that the Bulls will win", then you have predicted a winner. Otherwise, you have just described the chances of the Bulls winning.

Compare that with the 538 models of the election. In that case, the models so strongly predicted Obama would win that Silver felt confident in predicting that Obama would win the actual election. No such claims were made with the tournament, because that would be folly (as no team is a favorite to win it against the field).

Laying out a set of probabilities is more powerful than predicting the winner. All you need is a decision-operator to convert the probability into a prediction (or a bet or a set of bets). The distinctions here are more than semantics, but saying that specifying a set of tournament probabilities for 64 or 68 teams is not the same thing as predicting a tournament winner is true -- but a bit vacuous, since the former provides so much more detail.

CDu
04-23-2015, 07:55 AM
Laying out a set of probabilities is more powerful than predicting the winner. All you need is a decision-operator to convert the probability into a prediction (or a bet or a set of bets). The distinctions here are more than semantics, but saying that specifying a set of tournament probabilities for 64 or 68 teams is not the same thing as predicting a tournament winner is true -- but a bit vacuous, since the former provides so much more detail.

Agreed 100%. Though I would add that the reason for the distinction in this thread is that it seems some in this thread are using those probabilities to suggest that the model predicted Kentucky would win, and are thus questioning the model's value. That is the main reason I have been so steadfast in arguing this point: firstly, the model didn't predict a winner, and secondly, the model didn't predict that Kentucky was an odds-on favorite. As such nothing about the tourney results calls into question the "correctness" of the model. So while I agree that it should be a meaningless point, in this context it seems necessary to make.

freshmanjs
04-23-2015, 08:28 AM
Laying out a set of probabilities is more powerful than predicting the winner. All you need is a decision-operator to convert the probability into a prediction (or a bet or a set of bets). The distinctions here are more than semantics, but saying that specifying a set of tournament probabilities for 64 or 68 teams is not the same thing as predicting a tournament winner is true -- but a bit vacuous, since the former provides so much more detail.

right. the point is you cannot determine, after one trial, whether that model was "right" or "wrong." it's way too little data to evaluate it's validity.

Mtn.Devil.91.92.01.10.15
04-23-2015, 08:58 AM
Three statisticians are out hunting in the woods. They spot a buck 75 yards away. The first statistician takes account for the wind speed and direction, the angle of the shot, and trajectory of the bullet and fires. He misses four yards to the left. The buck spooks a bit, so the second statistician quickly takes into account the same factors and gets a shot off. He also misses, this time four yards to the right.

The third statistician suddenly jumps up and down "We got him!"

CameronBornAndBred
04-23-2015, 09:35 AM
Three statisticians are out hunting in the woods. They spot a buck 75 yards away. The first statistician takes account for the wind speed and direction, the angle of the shot, and trajectory of the bullet and fires. He misses four yards to the left. The buck spooks a bit, so the second statistician quickly takes into account the same factors and gets a shot off. He also misses, this time four yards to the right.

The third statistician suddenly jumps up and down "We got him!"
They played that one on NPR last week. (It was performed in front of a live audience with a resounding "thud".)

yancem
04-23-2015, 10:07 AM
correct. you did not predict a winner. you described the probabilities of outcomes.

Seems to me that the math guys arguing that someone stating probabilities of an outcome are not predicting a winner may be technically correct, but it sounds an awful like either building in an excuse for being wrong or not having the guts to make an actual prediction.

freshmanjs
04-23-2015, 10:09 AM
Seems to me that the math guys arguing that someone stating probabilities of an outcome are not predicting a winner may be technically correct, but it sounds an awful like either building in an excuse for being wrong or not having the guts to make an actual prediction.

if i have a model that says, when rolling dice, i have a 97.2% chance of rolling something other than a 12. then you roll dice and get a 12. is my model wrong? do i need an "excuse" for you getting a 12?

Duke95
04-23-2015, 10:34 AM
Seems to me that the math guys arguing that someone stating probabilities of an outcome are not predicting a winner may be technically correct, but it sounds an awful like either building in an excuse for being wrong or not having the guts to make an actual prediction.

Math isn't about "guts". It's about brains.

You do realize that there is uncertainty in the world around us, right? Statistical models incorporate that uncertainty into the results. Then, a person can look at those probability outcomes and draw a conclusion.

CDu
04-23-2015, 11:23 AM
if i have a model that says, when rolling dice, i have a 97.2% chance of rolling something other than a 12. then you roll dice and get a 12. is my model wrong? do i need an "excuse" for you getting a 12?

This. Similarly, a model says that a coin flip has a 50% chance of heads winning. You flip the coin and it lands tails. The model wasn't wrong. Just that particular outcome was tails.

Tournaments aren't deterministic. There is a chance that any team can win it. The team most likely to win it usually doesn't simply by virtue of it being unlikely that any particular team wins all six games. Someone will do it, but the likelihood of any particular team doing it is small.

Just like your dice example. Basically the tourney is the toss of a 68-sided, unevenly-weighted, die. The models can estimate the probability of that die landing with a particular side (team) face up. But once the die is rolled, anything can happen. If we roll that die enough times, in theory the results should look like the model probabilities. Unfortunately (or fortunately, as it adds drama), we only get to toss the die once.

So it isn't an excuse. It is just a fact. The tourney doesn't necessarily reflect the true best team. It just reflects the team that happened to prevail in the only observed chance we see. Often the winner really is the best team, often (probably more often) it is not. We don't get to observe enough observations (tourneys) to know if the model is right or wrong; you just have to accept or reject the model based on your belief in its assumptions/theoretical basis.

Troublemaker
04-23-2015, 11:56 AM
Essentially, if you re-played the tournament a billion times and Kentucky comes out the winner ~40% of those times, then the model was correct.

A model can NOT be wrong just because this ONE time the tournament was played Duke happened to be the winner (thankfully).

Incidentally, if the tournament were played a billion times, even Robert Morris would've won a few tournaments in there.

Mtn.Devil.91.92.01.10.15
04-23-2015, 12:04 PM
They played that one on NPR last week. (It was performed in front of a live audience with a resounding "thud".)

Apparently it doesn't play well on forums either.

NSDukeFan
04-23-2015, 02:18 PM
Apparently it doesn't play well on forums either.

I liked it but I'm easily entertained, and has heard it before.

Kfanarmy
04-23-2015, 04:11 PM
if i have a model that says, when rolling dice, i have a 97.2% chance of rolling something other than a 12. then you roll dice and get a 12. is my model wrong? do i need an "excuse" for you getting a 12? No, but if I roll a 12 two out of the first three times, you might want to start analyzing the assumptions driving the model...like uniformity of the sides, uniform weight distribution, etc. Your model isn't necessarily wrong if an unlikely outcome occurs, but when the statistics of your results aren't in reasonable alignment with your model, something is probably off.

freshmanjs
04-23-2015, 04:14 PM
No, but if I roll a 12 two out of the first three times, you might want to start analyzing the assumptions driving the model...like uniformity of the sides, uniform weight distribution, etc. Your model isn't necessarily wrong if an unlikely outcome occurs, but when the statistics of your results aren't in reasonable alignment with your model, something is probably off.

totally disagree....in 3 dice trials, it is very likely (certain) that what happens it unlikely.

Duke95
04-23-2015, 04:26 PM
No, but if I roll a 12 two out of the first three times, you might want to start analyzing the assumptions driving the model...like uniformity of the sides, uniform weight distribution, etc. Your model isn't necessarily wrong if an unlikely outcome occurs, but when the statistics of your results aren't in reasonable alignment with your model, something is probably off.

In this case, I would say you don't have a sufficiently large sample size.

CDu
04-23-2015, 04:45 PM
totally disagree....in 3 dice trials, it is very likely (certain) that what happens it unlikely.

And even more unlikely when the die is a 68-sided die and not just two six-sided dice (so 68 possible outcomes instead of 36).

The 97.2% chance that you wouldn't roll a 12 is absolutely correct. The model wasn't wrong, and you don't need to go checking the dice. Rolling 3 straight 12s is just a really improbable occurrence. Technically, ANY combination of outcomes over three iterations is an unlikely event, as there were 36^3 possible outcomes. The chances of rolling a 12 3 straight times is 1/(36^3). Really unlikely, but possible, and doing so doesn't mean the model was wrong. Just that one of many of the unlikely outcomes happened.

Take the dice example. If you built a model predicting the chances for each number (team) possible, you'd get get the following:
Two: 1/36 (2.8%)
Three: 2/36 (5.6%)
Four: 3/36 (8.3%)
Five: 4/36 (11.1%)
Six: 5/36 (13.9%)
Seven: 6/36 (16.7%)
Eight: 5/36 (13.9%)
Nine: 4/36 (11.1%)
Ten: 3/36 (8.3%)
Eleven: 2/36 (5.6%)
Twelve: 1/36 (2.8%)

In fact, the dice example is a really good one, as it isn't wildly different than the distribution of chances of teams winning the tournament (which is, I'm sure, why freshmanjs picked it).

What some folks are essentially saying is that the model predicts that 7 will be the winning number (team) because that is the number (team) with the highest chances of winning according to the model. But guess what? The chances of seven winning are only 16.7% (1/6)!

And, specific to the earlier posters comment about three rolls of the dice, the chances of rolling something other than a 7 three straight times is 57.8% (5/6)^3. So, in fact, I'd not be at all surprised to see the model be "wrong" three straight times (not just 2 of 3). Comparatively, I'd be more surprised to see the model get it "right" at least once than "wrong" all three times; the probability of rolling a 7 at least once in 3 tries is only 42.1%.

Let's take it to the tourney results. Kentucky had a ~40% probability of winning. That meant they had a ~60% probability of losing. So the model got it right! ;)

But even if you want to take the (incorrect) case of saying the model should get the winner more than once in the past 3 years, let's play the game. Even if we assume the chances of the best team winning were the same as 2015 Kentucky's for each of the 3 years (which we know is not the case as Kentucky was a much larger favorite than normal this year, but I'm being ultra conservative), there is a roughly 43.2% chance that the model would get it "right" once and only once out of 3 years. The chance that the model would get it right twice or more out of three years is only 35.2%, and there is a 21.6% chance that the model would get it wrong all three times. So even when there is one team seemingly MUCH better than the rest of the field in each of three years, the probability of the model getting it wrong all three times is almost as high as the probability of getting it right twice. And the probability of getting it right once, while still not the most likely scenario, had the highest probability.

The closer the best team is to the rest of the field (i.e., the lower the "favorite's" chances of winning are), the more likely any model (even a 100% accurate one) is to get none or one of the three outcomes right. So, no, a model "getting it wrong" at least 2 out of 3 tournaments is not at all surprising. In fact, it should be expected. I'd be fairly surprised if a model "got it right" more than once in 3 tries.

Mtn.Devil.91.92.01.10.15
04-23-2015, 05:57 PM
And even more unlikely when the die is a 68-sided die and not just two six-sided dice (so 68 possible outcomes instead of 36).

The 97.2% chance that you wouldn't roll a 12 is absolutely correct. The model wasn't wrong, and you don't need to go checking the dice. Rolling 3 straight 12s is just a really improbable occurrence. Technically, ANY combination of outcomes over three iterations is an unlikely event, as there were 36^3 possible outcomes. The chances of rolling a 12 3 straight times is 1/(36^3). Really unlikely, but possible, and doing so doesn't mean the model was wrong. Just that one of many of the unlikely outcomes happened.

Take the dice example. If you built a model predicting the chances for each number (team) possible, you'd get get the following:
Two: 1/36 (2.8%)
Three: 2/36 (5.6%)
Four: 3/36 (8.3%)
Five: 4/36 (11.1%)
Six: 5/36 (13.9%)
Seven: 6/36 (16.7%)
Eight: 5/36 (13.9%)
Nine: 4/36 (11.1%)
Ten: 3/36 (8.3%)
Eleven: 2/36 (5.6%)
Twelve: 1/36 (2.8%)

In fact, the dice example is a really good one, as it isn't wildly different than the distribution of chances of teams winning the tournament (which is, I'm sure, why freshmanjs picked it).

What some folks are essentially saying is that the model predicts that 7 will be the winning number (team) because that is the number (team) with the highest chances of winning according to the model. But guess what? The chances of seven winning are only 16.7% (1/6)!

And, specific to the earlier posters comment about three rolls of the dice, the chances of rolling something other than a 7 three straight times is 57.8% (5/6)^3. So, in fact, I'd not be at all surprised to see the model be "wrong" three straight times (not just 2 of 3). Comparatively, I'd be more surprised to see the model get it "right" at least once than "wrong" all three times; the probability of rolling a 7 at least once in 3 tries is only 42.1%.

Let's take it to the tourney results. Kentucky had a ~40% probability of winning. That meant they had a ~60% probability of losing. So the model got it right! ;)

But even if you want to take the (incorrect) case of saying the model should get the winner more than once in the past 3 years, let's play the game. Even if we assume the chances of the best team winning were the same as 2015 Kentucky's for each of the 3 years (which we know is not the case as Kentucky was a much larger favorite than normal this year, but I'm being ultra conservative), there is a roughly 43.2% chance that the model would get it "right" once and only once out of 3 years. The chance that the model would get it right twice or more out of three years is only 35.2%, and there is a 21.6% chance that the model would get it wrong all three times. So even when there is one team seemingly MUCH better than the rest of the field in each of three years, the probability of the model getting it wrong all three times is almost as high as the probability of getting it right twice. And the probability of getting it right once, while still not the most likely scenario, had the highest probability.

The closer the best team is to the rest of the field (i.e., the lower the "favorite's" chances of winning are), the more likely any model (even a 100% accurate one) is to get none or one of the three outcomes right. So, no, a model "getting it wrong" at least 2 out of 3 tournaments is not at all surprising. In fact, it should be expected. I'd be fairly surprised if a model "got it right" more than once in 3 tries.

The ModGods won't let me spork you, but this is why I love this board.

wilson
04-23-2015, 05:59 PM
The ModGods won't let me spork you, but this is why I love this board.I'm likewise ineligible to spork CDu, but I also thought that was an excellent post.
I have learned a ton from this thread...thanks to everyone.

Indoor66
04-23-2015, 08:36 PM
Again, much of this thread reads, to me, like a debate about Angels and Pins from so long ago. Hair splitting to darn near infinity.

ice-9
04-23-2015, 11:48 PM
No, but if I roll a 12 two out of the first three times, you might want to start analyzing the assumptions driving the model...like uniformity of the sides, uniform weight distribution, etc. Your model isn't necessarily wrong if an unlikely outcome occurs, but when the statistics of your results aren't in reasonable alignment with your model, something is probably off.

Thank you, well said, exactly my point and which focused on Duke as the application.

The model's assumptions about Duke was not good.


Laying out a set of probabilities is more powerful than predicting the winner. All you need is a decision-operator to convert the probability into a prediction (or a bet or a set of bets). The distinctions here are more than semantics, but saying that specifying a set of tournament probabilities for 64 or 68 teams is not the same thing as predicting a tournament winner is true -- but a bit vacuous, since the former provides so much more detail.

It's not specifying, it's predicting. A subtle but crucial distinction that's lost on many and the cause of my aggravation.

Mtn.Devil.91.92.01.10.15
04-24-2015, 05:31 AM
Disclaimer:
I am not nearly as savvy with numbers as many folks on this thread.

I think perhaps it's less a question of semantics and numbers than it is about context. We understand empirically that a "fair" set of dice has an understandable and definable predictable model. We've each rolled the dice on enough occasions that when someone tells you that they rolled "snake-eyes" three times in a row that it's a rather remarkable feat. We don't throw a fit and say "What? Based on my predictive model, that's extremely unlikely!?"

Conversely, there's the lottery. If anyone really were able to conceptualize how miniscule one's chances of winning big money are, no one would play. If someone were to rattle off the other occurrences in life one a daily basis with similar odds - "Today there is a one in five million chance that you will make good friend with Kate Upton - great, that's even better than my chances at PowerBalll!" - you would quickly have a better understanding of the astronomical odds associated with the lottery.

If the dice analogy (which was very well explained above) doesn't resonate with some readers, consider perhaps the weather report and the predicted chance of rain. What does the percentages relayed in "chance of rain" actually mean to us? According to 538, Kentucky had a 41% chance of winning the tournament, which we all recognize as a remarkably high number. If you were planning a big picnic - the biggest and best of the year - and someone told you that there was a 59% chance of rain (thusly, an inverse of 41% chance that it would not rain) would you change your plans? More importantly, if it did rain, would you write an angry letter to the weatherman complaining that his predictive model that said there was a 41% chance of NOT rain was a broken model?

No, of course not. The percentages mean different things to us in different contexts. If the weatherman says there's 60% chance of rain and it doesn't rain, we realize we're fortunate if we had outdoor plans. The meteorologist doesn't go on the air the next day and apologize for his faulty metrics.

If he says there's a ten percent chance of rain and it downpours, we start getting frustrated and feel we have bad luck.

There's another discussion to be had that I would love to sit on the sidelines for regarding what exactly the forecasted rain percentage means from a scientific standpoint (if it sprinkles for three minutes at 7pm is that 100% rain? I guess yes) but I hope that this message just makes us realize that our acceptance of predictive models varies wildly based on context.

freshmanjs
04-24-2015, 07:40 AM
Thank you, well said, exactly my point and which focused on Duke as the application.

The model's assumptions about Duke was not good.


This is a remarkable statement. what do you think the correct % chance for Duke to win the tournament would have been before the tournament started? what would have been the number that you would say was "good?"

CDu
04-24-2015, 08:33 AM
Thank you, well said, exactly my point and which focused on Duke as the application.

Except that the post you are applauding is completely incorrect. You should expect the model to be "wrong" at least two out of three times when the probability of the best team not winning is 60% or greater (as is pretty much always the case). To say otherwise suggests a lack of understanding of probability. We would need LOTS more tournaments before we can say that the model is wrong.


The model's assumptions about Duke was not good.

That may or may not be correct. We don't have nearly enough information to say that. The results of the tournament certainly don't prove that.

Duke played substantially and consistently better defense in the tourney than they had at any point in the season. I would argue that it was improbable that Duke would do that for six straight games without hiccup, which is essentially what the model said. Maybe it really was more likely that Duke would become a defensive juggernaut in the tourney. But we would only be able to determine that with a much larger number of tourneys.

Kfanarmy
04-24-2015, 09:43 AM
totally disagree....in 3 dice trials, it is very likely (certain) that what happens it unlikely. You are changing the standard here. You proffered a model predicting something other than a "12" 97.2% of the time. All I'm saying is that if you roll a 12 two out of the first three times, your model is "probably" incorrect. I think you could show that the probability of your model being perfect is far outweighed by the probablity that you missed something which is why it doesn't agree with reality. It is convenient, accurate and misleading argument to say that everything is within the range of the possible therefore the model can't be faulty. Hypothetically I present a model that says the sun has a 99.99% chance of going supernova tomorrow. If it doesn't I can argue my model is still good simply because I left open the .1% chance that the sun wouldn't go supernova? I concede that is a hyperbolic example, but to not accept that a model "might" be faulty if something occurs that is statisically unlikely is making a religion out of the model rather than applying science to evaluate its performance.

Duke95
04-24-2015, 09:46 AM
You are changing the standard here. You proffered a model predicting something other than a "12" 97.2% of the time. All I'm saying is that if you roll a 12 two out of the first three times, your model is "probably" incorrect.

Nope. That's not right, and the reason why has been explained several times.

freshmanjs
04-24-2015, 09:59 AM
You are changing the standard here. You proffered a model predicting something other than a "12" 97.2% of the time. All I'm saying is that if you roll a 12 two out of the first three times, your model is "probably" incorrect. I think you could show that the probability of your model being perfect is far outweighed by the probablity that you missed something which is why it doesn't agree with reality. It is convenient, accurate and misleading argument to say that everything is within the range of the possible therefore the model can't be faulty. Hypothetically I present a model that says the sun has a 99.99% chance of going supernova tomorrow. If it doesn't I can argue my model is still good simply because I left open the .1% chance that the sun wouldn't go supernova? I concede that is a hyperbolic example, but to not accept that a model "might" be faulty if something occurs that is statisically unlikely is making a religion out of the model rather than applying science to evaluate its performance.

i've rolled 2 aces in a row. turned $5 into $4500 doing it. no one at the casino blinked an eye. no one thought there were loaded dices or that the laws or probability were wrong. an unusual event occurred.

the bigger point is that no matter what combination of things you roll in your first 3 rolls, it would be very unlikely. so according to your logic, ALL outcomes of 3 rolls would tell you the model is probably wrong.

CDu
04-24-2015, 10:35 AM
You are changing the standard here. You proffered a model predicting something other than a "12" 97.2% of the time. All I'm saying is that if you roll a 12 two out of the first three times, your model is "probably" incorrect. I think you could show that the probability of your model being perfect is far outweighed by the probablity that you missed something which is why it doesn't agree with reality. It is convenient, accurate and misleading argument to say that everything is within the range of the possible therefore the model can't be faulty. Hypothetically I present a model that says the sun has a 99.99% chance of going supernova tomorrow. If it doesn't I can argue my model is still good simply because I left open the .1% chance that the sun wouldn't go supernova? I concede that is a hyperbolic example, but to not accept that a model "might" be faulty if something occurs that is statisically unlikely is making a religion out of the model rather than applying science to evaluate its performance.

No, he/she isn't changing the standard. And no, rolling a 12 two out of the first 3 times doesn't show that the model is "probably" incorrect. Just is an extremely rare outcome occurred.

Is it possible (even probable) that these models are flawed? Absolutely. Do the results of one, two, three, or even ten tournaments suggest that the model is flawed? Absolutely not, as I showed in my extremely long post above. The most likely outcome of the last 3 tournaments was that, at least 2 of 3 times, the winning team would be some team other than the team with the best predicted chances of winning. That is the nature of the beast when no team is a heavy favorite against the field.

I think you are making the mistake of thinking that the team with the highest predicted probability of winning is the team that the model predicts will win. That is not (necessarily) the case. The model didn't predict that Kentucky would win this year. In fact, based on the probabilities, one should have predicted the field to win rather than Kentucky (it was close, but the odds were still against Kentucky winning). Heck, the model only predicted that Kentucky had a coin's flip chance of making the championship game (which, by the way, was pretty darn close to happening). Similarly, the model predicted even more strongly that Louisville wouldn't win in 2013. I don't remember who had the best chance of winning in 2014, but the model most certainly said that team was more likely to lose the tournament than it was to win it.

However, even if the models predicted one team had an 80% chance of winning, it wouldn't be unheard of for some other team to win it in 2 of 3 years. There would be roughly a 10% chance that the winning team would be someone other than that prohibitive favorite at least 2 of 3 times. And that is in the outlandishly extreme scenario in which one team was so vastly superior to the field that they had an 80% chance of winning the title before the tournament started! As I described above, even assuming the top team had a 40% chance of winning (which is higher than the actual chances in two of those three years), there was roughly a 65% chance that someone other than the "favorite" would win it at least 2 of the last 3 years. So if you were a betting man or women, the models say you should take the field over the best team every year. The models agree with the outcomes of the past three years: the "favorite" won it in only 1 of the last 3 years, so (as the models suggested) you should pick the field over the best team.

Nothing in the results of the last three years suggest that the models are wrong. Doesn't mean they are right, but the evidence certainly doesn't suggest they are wrong.

Mtn.Devil.91.92.01.10.15
04-24-2015, 10:41 AM
No, he/she isn't changing the standard. And no, rolling a 12 two out of the first 3 times doesn't show that the model is "probably" incorrect. Just is an extremely rare outcome.

...

Nothing in the results of the last three years suggest that the models are wrong. Doesn't mean they are right, but the evidence certainly doesn't suggest they are wrong.

ANY "particular outcome" is extremely unlikely. That's the crux of what makes this exercise so difficult. I mean, rolling a 7, then a 4, and then a 8 is unlikely. Not quite as unlikely as 12, 12, 12, but still very unlikely.

CDu
04-24-2015, 10:47 AM
ANY "particular outcome" is extremely unlikely. That's the crux of what makes this exercise so difficult. I mean, rolling a 7, then a 4, and then a 8 is unlikely. Not quite as unlikely as 12, 12, 12, but still very unlikely.

Exactly. I think the problem is that a lot of folks (a) don't have a good understanding of the unlikeliness of outcomes and (b) make the mistake of assuming that the outcome that occurred was the most likely outcome (simply because it happened).

These models clearly show that no individual team is likely to win the tournament. Almost by definition, any particular team winning it is unlikely. Even the best team has generally a 25-40% chance of winning. So there is a 60-75% chance that the best team isn't going to win it. So saying "the model appears to be wrong because the best team didn't win it in 2 of the last 3 years" seems to suggest a lack of understanding of the unlikeliness of a particular team (even the best team) winning. That's the expected outcome based on the model's predicted probabilities.

Mtn.Devil.91.92.01.10.15
04-24-2015, 10:55 AM
Exactly. I think the problem is that a lot of folks (a) don't have a good understanding of the unlikeliness of outcomes and (b) make the mistake of assuming that the outcome that occurred was the most likely outcome (simply because it happened).

These models clearly show that no individual team is likely to win the tournament. Almost by definition, any particular team winning it is unlikely. Even the best team has generally a 25-40% chance of winning. So there is a 60-75% chance that the best team isn't going to win it. So saying "the model appears to be wrong because the best team didn't win it in 2 of the last 3 years" seems to suggest a lack of understanding of the unlikeliness of a particular team (even the best team) winning. That's the expected outcome based on the model's predicted probabilities.

I guess the next question if we want to bring it back to basketball is... how often does the "prohibitive favorite" win the tournament? I mean, at the end of the day, it's an even dicier bet than "Tiger v. The Field" at his prime, right?

CDu
04-24-2015, 11:11 AM
I guess the next question if we want to bring it back to basketball is... how often does the "prohibitive favorite" win the tournament? I mean, at the end of the day, it's an even dicier bet than "Tiger v. The Field" at his prime, right?

Right. I mean, I'm sure we could go back and find the pre-tournament odds for each team each year somewhere and look to see how many times the #1 team won it. Or, I guess we could proxy by using the AP Final #1 and seeing how many times the #1 team won the tournament. My guess is that the percentage is pretty low. Going back the past 9 years, it appears that the #1 team per the AP poll has only won it once (UK in 2012) in the past 9 years, and the #1 or #2 team in the poll has only won it 3 times in the past 9 years (2009, 2012, 2013). I included 2009 because while Gonzaga was technically ahead of Louisville in 2013, I think everyone knew Louisville was the best team. Similarly, Louisville was #1 in 2009, but many felt UNC was really #1.

Note: I only went back 9 years because that was all the data ESPN had readily available. But I suspect the trend is similar if you go back further. UF certainly wasn't the overall #1 in 2006, and Illinois was the #1 in 2005, UConn wasn't even a 1 seed in 2004, Syracuse was a #3 seed in 2003, Duke was the overall #1 in 2002, so perhaps the last overall #1 to win it before UNC in 2009 was Duke in 2001. I think MSU was the overall #1 in 2000 (only because Cincy had the injury to Kenyon Martin), but Duke was the overall #1 in 1999. UNC was #1 in 1998, and Arizona certainly not #1 in 1997. So it appears that only 5 times in the past 19 years has the overall best team (heading into the tournament) actually won the tournament (26.3%). Which is, interestingly enough, roughly in line with the expected probability of winning for the typical "best" team each year according to these models.

Mtn.Devil.91.92.01.10.15
04-24-2015, 11:28 AM
Right. I mean, I'm sure we could go back and find the pre-tournament odds for each team each year somewhere and look to see how many times the #1 team won it. Or, I guess we could proxy by using the AP Final #1 and seeing how many times the #1 team won the tournament. My guess is that the percentage is pretty low. Going back the past 9 years, it appears that the #1 team per the AP poll has only won it once (UK in 2012) in the past 9 years, and the #1 or #2 team in the poll has only won it 3 times in the past 9 years (2009, 2012, 2013). I included 2009 because while Gonzaga was technically ahead of Louisville in 2013, I think everyone knew Louisville was the best team. Similarly, Louisville was #1 in 2009, but many felt UNC was really #1.

Note: I only went back 9 years because that was all the data ESPN had readily available. But I suspect the trend is similar if you go back further. UF certainly wasn't the overall #1 in 2006, and Illinois was the #1 in 2005, UConn wasn't even a 1 seed in 2004, Syracuse was a #3 seed in 2003, Duke was the overall #1 in 2002, so perhaps the last overall #1 to win it before UNC in 2009 was Duke in 2001. I think MSU was the overall #1 in 2000 (only because Cincy had the injury to Kenyon Martin), but Duke was the overall #1 in 1999. UNC was #1 in 1998, and Arizona certainly not #1 in 1997. So it appears that only 5 times in the past 19 years has the overall best team (heading into the tournament) actually won the tournament (26.3%). Which is, interestingly enough, roughly in line with the expected probability of winning for the typical "best" team each year according to these models.

So... about 33% of the time? And this year's favorite was ranked at 41%?

Sounds about right.

Kedsy
04-24-2015, 12:23 PM
Right. I mean, I'm sure we could go back and find the pre-tournament odds for each team each year somewhere and look to see how many times the #1 team won it. Or, I guess we could proxy by using the AP Final #1 and seeing how many times the #1 team won the tournament. My guess is that the percentage is pretty low.

I've gone back to 1979 (first year the tournament had real seeding) for pre-tournament AP, back 20 years for pre-tournament RPI, and back seven years for pre-tournament Pomeroy (all I have). Here are the results:



Year AP #1 AP #2 RPI #1 Pomeroy #1 Champ
2015 Kentucky Villanova Kentucky Kentucky Duke
2014 Florida Wichita State Florida Arizona UConn
2013 Gonzaga Louisville Duke Florida Louisville
2012 Kentucky Syracuse Syracuse Kentucky Kentucky
2011 Ohio State Kansas Kansas Ohio State UConn
2010 Kansas Kentucky Kansas Duke Duke
2009 Louisville UNC Duke Memphis UNC
2008 UNC Memphis Tennessee Kansas
2007 Ohio State Kansas Ohio State Florida
2006 Duke UConn Duke Florida
2005 Illinois UNC Illinois UNC
2004 Stanford Kentucky Duke UConn
2003 Kentucky Arizona Kentucky Syracuse
2002 Duke Kansas Kansas Maryland
2001 Duke Stanford Duke Duke
2000 Duke Michigan State Cincinnati Michigan State
1999 Duke Michigan State Duke UConn
1998 UNC Kansas UNC Kentucky
1997 Kansas Utah Kansas Arizona
1996 Massachusetts Kentucky Kansas Kentucky
1995 UCLA Kentucky UCLA
1994 UNC Arkansas Arkansas
1993 Indiana Kentucky UNC
1992 Duke Kansas Duke
1991 UNLV Arkansas Duke
1990 Oklahoma UNLV UNLV
1989 Arizona Georgetown Michigan
1988 Temple Arizona Kansas
1987 UNLV UNC Indiana
1986 Duke Kansas Louisville
1985 Georgetown Michigan Villanova
1984 UNC Georgetown Georgetown
1983 Houston Louisville NC State
1982 UNC DePaul UNC
1981 DePaul Oregon State Indiana
1980 DePaul Louisville Louisville
1979 Indiana State UCLA Michigan State


The totals come out:

AP#1: 5 of 37 (13.5%)
AP#1 or AP#2: 9 of 37 (24.3%)
RPI: 1 of 20 (5%)
Pomeroy: 2 of 7 (28.6%)

So, Pomeroy comes out best, but in a much smaller sample so I don't think that proves anything. RPI comes out worst (in a decent but perhaps not even big enough sample), and that makes sense to me.

None of these methods for predicting the champion even get particularly close to a 1 in 3 chance.

sagegrouse
04-24-2015, 12:54 PM
This is a remarkable statement. what do you think the correct % chance for Duke to win the tournament would have been before the tournament started? what would have been the number that you would say was "good?"

I'll say it: the performance model, as estimated by KenPom, was wrong for Duke and the other ACC teams because it totally relied on game stats -- the ACC was far better than the estimates generated by the inter-conference games in November and December. And, no matter how great the teams played in January and February, they were playing against each other, resulting in a zero-sum game where the overall average rating of the ACC remains unchanged.

The phenomenal record of the ACC in the NCAA's (plus Miami in the NIT) should be pretty good evidence of the above. Moreover, to me, the ACC teams just looked a lot quicker than the Big Ten, SEC, and Big 12 teams -- and, of course, Duke did a pretty good job of carving up both Wisconsin and Michigan State in the early season -- and at the end.

The 538 estimates presumably included more than game stats and results, but that's not exactly transparent, is it?

freshmanjs
04-24-2015, 12:57 PM
I'll say it: the performance model, as estimated by KenPom, was wrong for Duke and the other ACC teams because it totally relied on game stats -- the ACC was far better than the estimates generated by the inter-conference games in November and December. And, no matter how great the teams played in January and February, they were playing against each other, resulting in a zero-sum game where the overall average rating of the ACC remains unchanged.

The phenomenal record of the ACC in the NCAA's (plus Miami in the NIT) should be pretty good evidence of the above. Moreover, to me, the ACC teams just looked a lot quicker than the Big Ten, SEC, and Big 12 teams -- and, of course, Duke did a pretty good job of carving up both Wisconsin and Michigan State in the early season -- and at the end.

The 538 estimates presumably included more than game stats and results, but that's not exactly transparent, is it?

i have no issue with a view that the kenpom methodology is flawed (in fact, it is certainly flawed. we can debate about how much). i do have issue with the view that the tournament results somehow indicate that the model was "wrong" (as if there is any meaning to right and wrong based on a sample size of 1 tournament).

freshmanjs
04-24-2015, 01:53 PM
I've gone back to 1979 (first year the tournament had real seeding) for pre-tournament AP, back 20 years for pre-tournament RPI, and back seven years for pre-tournament Pomeroy (all I have). Here are the results:



Year AP #1 AP #2 RPI #1 Pomeroy #1 Champ
2015 Kentucky Villanova Kentucky Kentucky Duke
2014 Florida Wichita State Florida Arizona UConn
2013 Gonzaga Louisville Duke Florida Louisville
2012 Kentucky Syracuse Syracuse Kentucky Kentucky
2011 Ohio State Kansas Kansas Ohio State UConn
2010 Kansas Kentucky Kansas Duke Duke
2009 Louisville UNC Duke Memphis UNC
2008 UNC Memphis Tennessee Kansas
2007 Ohio State Kansas Ohio State Florida
2006 Duke UConn Duke Florida
2005 Illinois UNC Illinois UNC
2004 Stanford Kentucky Duke UConn
2003 Kentucky Arizona Kentucky Syracuse
2002 Duke Kansas Kansas Maryland
2001 Duke Stanford Duke Duke
2000 Duke Michigan State Cincinnati Michigan State
1999 Duke Michigan State Duke UConn
1998 UNC Kansas UNC Kentucky
1997 Kansas Utah Kansas Arizona
1996 Massachusetts Kentucky Kansas Kentucky
1995 UCLA Kentucky UCLA
1994 UNC Arkansas Arkansas
1993 Indiana Kentucky UNC
1992 Duke Kansas Duke
1991 UNLV Arkansas Duke
1990 Oklahoma UNLV UNLV
1989 Arizona Georgetown Michigan
1988 Temple Arizona Kansas
1987 UNLV UNC Indiana
1986 Duke Kansas Louisville
1985 Georgetown Michigan Villanova
1984 UNC Georgetown Georgetown
1983 Houston Louisville NC State
1982 UNC DePaul UNC
1981 DePaul Oregon State Indiana
1980 DePaul Louisville Louisville
1979 Indiana State UCLA Michigan State


The totals come out:

AP#1: 5 of 37 (13.5%)
AP#1 or AP#2: 9 of 37 (24.3%)
RPI: 1 of 20 (5%)
Pomeroy: 2 of 7 (28.6%)

So, Pomeroy comes out best, but in a much smaller sample so I don't think that proves anything. RPI comes out worst (in a decent but perhaps not even big enough sample), and that makes sense to me.

None of these methods for predicting the champion even get particularly close to a 1 in 3 chance.

but this doesn't tell you how good the Pomeroy model is (and I know you aren't saying it does). even if the Pomeroy model were perfect (it's not), it's most likely winner would only win about this often.

sagegrouse
04-24-2015, 02:06 PM
i have no issue with a view that the kenpom methodology is flawed (in fact, it is certainly flawed. we can debate about how much). i do have issue with the view that the tournament results somehow indicate that the model was "wrong" (as if there is any meaning to right and wrong based on a sample size of 1 tournament).

It is not that the KenPom methodology is "flawed," it is that the "experiment" the NCAA collectively runs was especially bad this year -- the results in November and December did not give good data on the relative strength of the conferences.

I don't have a particular view on how KenPom's methodology can be improved; but, as I understand it, it considers only statistics from games, rather than polls, heuristics, expert opinions, etc.

Uh,... freshmanjs, "sample size of 1 tournament" is a bit at variance, don't you think, with the 23 games that the ACC played in the tournament -- with a record of 18-5 and only four losses to non-ACC teams.

Other years, the November and December results might be better indications of the relative strength of the conferences. This year, they weren't.

Of course, I have posted frequently on other threads how it is basically impossible to select and seed teams for the tournament, when the only inter-conference results are in November and December (plus a the first few days of January).

NSDukeFan
04-24-2015, 02:11 PM
ANY "particular outcome" is extremely unlikely. That's the crux of what makes this exercise so difficult. I mean, rolling a 7, then a 4, and then a 8 is unlikely. Not quite as unlikely as 12, 12, 12, but still very unlikely.

I'm not a math major, but don't these two scenarios have the exact same likelihood?

freshmanjs
04-24-2015, 02:12 PM
Uh,... freshmanjs, "sample size of 1 tournament" is a bit at variance, don't you think, with the 23 games that the ACC played in the tournament -- with a record of 18-5 and only four losses to non-ACC teams.



not sure what the "Uh" is for.

not sure what you mean by "a bit of variance." if you mean sufficient sample size to tell if the model is wrong, i don't think so. but, i'm not a statistician.

Kedsy
04-24-2015, 02:26 PM
I'm not a math major, but don't these two scenarios have the exact same likelihood?

No.

I wasn't a math major, either, but the chance of rolling a 12 is 1 in 36. The chance of rolling three 12s in a row is (1/36)^3 which equals 1 in 46,656.

The chance of rolling a 7, then a 4, then an 8 is: (6/36)*(3/36)*(5/36), which equals 90 in 46,656, which comes out to 1 in 518.4.

uh_no
04-24-2015, 02:36 PM
No.

I wasn't a math major, either, but the chance of rolling a 12 is 1 in 36. The chance of rolling three 12s in a row is (1/36)^3 which equals 1 in 46,656.

The chance of rolling a 7, then a 4, then an 8 is: (6/36)*(3/36)*(5/36), which equals 90 in 46,656, which comes out to 1 in 518.4.

I initially thought the same thing, as it's a common fallacy that 3 of the same event occurring is more rare than 3 other unique events. In the case of drawing cards from a deck, he'd be correct, as the chance of drawing 3 aces (with replacement) is the same as drawing a 7, then a 4, then an 8 (still with replacement).

What he, and I for a moment, was forgetting was that the odds of rolling a number using 2 dice is not uniform.

I don't think it affects the other guy's point much at all though, as the likelihood of any particular outcome is still pretty rare.

Mtn.Devil.91.92.01.10.15
04-24-2015, 02:38 PM
I'm not a math major, but don't these two scenarios have the exact same likelihood?

The trick is that there's more ways to roll a 7 (2+6, 2+5, 3+4) than 12 (6+6).

Kedsy
04-24-2015, 02:41 PM
The trick is that there's more ways to roll a 7 (2+6, 2+5, 3+4) than 12 (6+6).

It really would be a trick if you could roll a 7 with a 2 and a 6. ;)

Mtn.Devil.91.92.01.10.15
04-24-2015, 02:48 PM
I initially thought the same thing, as it's a common fallacy that 3 of the same event occurring is more rare than 3 other unique events. In the case of drawing cards from a deck, he'd be correct, as the chance of drawing 3 aces (with replacement) is the same as drawing a 7, then a 4, then an 8 (still with replacement).

What he, and I for a moment, was forgetting was that the odds of rolling a number using 2 dice is not uniform.

I don't think it affects the other guy's point much at all though, as the likelihood of any particular outcome is still pretty rare.

I know we're getting way off track here... but "three of the same thing" is different than "three of this specific thing." Also, the chances of a 12 are lower than a 7 to begin with, as it requires two separate events (1 in 6, 1 in 6).

freshmanjs
04-24-2015, 02:50 PM
I know we're getting way off track here... but "three of the same thing" is different than "three of this specific thing." Also, the chances of a 12 are lower than a 7 to begin with, as it requires two separate events (1 in 6, 1 in 6).

in the dice example, if you take a more precise view of what is observed (separate the 2 dice and note the configuration), then all outcomes are equally likely and whatever happens in 3 rolls is just as unlikely as 3 12s.

Mtn.Devil.91.92.01.10.15
04-24-2015, 02:52 PM
It really would be a trick if you could roll a 7 with a 2 and a 6. ;)

Lordy. It's not even happy hour.

Mtn.Devil.91.92.01.10.15
04-24-2015, 02:54 PM
in the dice example, if you take a more precise view of what is observed (separate the 2 dice and note the configuration), then all outcomes are equally likely and whatever happens in 3 rolls is just as unlikely as 3 12s.

Well, sure but "rolling a 7" doesn't really suggest that you are looking for a roll of a 3 followed by a roll of a 4.

freshmanjs
04-24-2015, 02:56 PM
Well, sure but "rolling a 7" doesn't really suggest that you are looking for a roll of a 3 followed by a roll of a 4.

yup that's true. point is just that, no matter what happens in 3 dice trials, it will be unusual. you can't conclude anything at all about the validity of a dice probability model based on 3 trials. (much like you can't about the validity of a hoops model from 1 tournament).

uh_no
04-24-2015, 02:58 PM
in the dice example, if you take a more precise view of what is observed (separate the 2 dice and note the configuration), then all outcomes are equally likely and whatever happens in 3 rolls is just as unlikely as 3 12s.

I think his point saw that "3 of the same thing" means it can be the same 3 of any number, which is in general, <# of things>*<probability of thing>^3

So in the case of a uniform die roll, the probability of 3 in a row is 6 * 1/6^3, since you can get 3 in a row in 6 different ways (once for each number).

The probability of 3 SPECIFIC things happening in a row is only <probability of thing>^3, since there is only 1 way for it to happen. FOr a die roll, it's just 1/6^3

SO I was a tad loose in my verbiage and got called out :)

CDu
04-24-2015, 03:12 PM
I've gone back to 1979 (first year the tournament had real seeding) for pre-tournament AP, back 20 years for pre-tournament RPI, and back seven years for pre-tournament Pomeroy (all I have). Here are the results:



Year AP #1 AP #2 RPI #1 Pomeroy #1 Champ
2015 Kentucky Villanova Kentucky Kentucky Duke
2014 Florida Wichita State Florida Arizona UConn
2013 Gonzaga Louisville Duke Florida Louisville
2012 Kentucky Syracuse Syracuse Kentucky Kentucky
2011 Ohio State Kansas Kansas Ohio State UConn
2010 Kansas Kentucky Kansas Duke Duke
2009 Louisville UNC Duke Memphis UNC
2008 UNC Memphis Tennessee Kansas
2007 Ohio State Kansas Ohio State Florida
2006 Duke UConn Duke Florida
2005 Illinois UNC Illinois UNC
2004 Stanford Kentucky Duke UConn
2003 Kentucky Arizona Kentucky Syracuse
2002 Duke Kansas Kansas Maryland
2001 Duke Stanford Duke Duke
2000 Duke Michigan State Cincinnati Michigan State
1999 Duke Michigan State Duke UConn
1998 UNC Kansas UNC Kentucky
1997 Kansas Utah Kansas Arizona
1996 Massachusetts Kentucky Kansas Kentucky
1995 UCLA Kentucky UCLA
1994 UNC Arkansas Arkansas
1993 Indiana Kentucky UNC
1992 Duke Kansas Duke
1991 UNLV Arkansas Duke
1990 Oklahoma UNLV UNLV
1989 Arizona Georgetown Michigan
1988 Temple Arizona Kansas
1987 UNLV UNC Indiana
1986 Duke Kansas Louisville
1985 Georgetown Michigan Villanova
1984 UNC Georgetown Georgetown
1983 Houston Louisville NC State
1982 UNC DePaul UNC
1981 DePaul Oregon State Indiana
1980 DePaul Louisville Louisville
1979 Indiana State UCLA Michigan State


The totals come out:

AP#1: 5 of 37 (13.5%)
AP#1 or AP#2: 9 of 37 (24.3%)
RPI: 1 of 20 (5%)
Pomeroy: 2 of 7 (28.6%)

So, Pomeroy comes out best, but in a much smaller sample so I don't think that proves anything. RPI comes out worst (in a decent but perhaps not even big enough sample), and that makes sense to me.

None of these methods for predicting the champion even get particularly close to a 1 in 3 chance.

First of all Kedsy, thanks much for doing the legwork on this. I can't give you pitchfork points, but I'll give you "verbal" kudos. I'd also note that Pomeroy's models generally predict the "best" team to have a ~25-35% chance of winning. So observing the expected best team winning 28.6% of the 7 observations available (small sample size alert!) is pretty much exactly what Pomeroy would expect to happen: the "best" team won only 28.6% of the time, which is dead in the middle of the range of win probabilities for the best team over that span.

I'll also say this: it is very possible (I would say highly probable) that these models all suffer from substantial limitation in that they are only as good as the data available. As sagegrouse sagely notes, it's quite possible that there was a systematic bias against the ACC based on the limited number of inter-conference results not being as favorable as they should be for the ACC. Further, as someone (I apologize that I can't recall who) pointed out, it's especially hard to rate the West Coast teams vis-a-vis the rest of the country because the West Coast teams played fewer games against the Midwest, South, and East schools. So it's absolutely possible that there are systematic errors inherent based on the available data, and thus the predicted probabilities may indeed be flawed. I'd be surprised if that weren't true, actually.

However, the results of this tournament do not indicate that this is the case, nor do they indicate that it is not the case. Similarly, the results of the last three tournaments don't indicate presence/absence of flaw or error either.

Duvall
04-24-2015, 03:35 PM
i have no issue with a view that the kenpom methodology is flawed (in fact, it is certainly flawed. we can debate about how much). i do have issue with the view that the tournament results somehow indicate that the model was "wrong" (as if there is any meaning to right and wrong based on a sample size of 1 tournament).

Especially when the tournament results necessarily only reflect the performance of the teams that were invited to the tournament, and not the terrible league teams that stayed at home.

sagegrouse
04-24-2015, 03:37 PM
First of all Kedsy, thanks much for doing the legwork on this. I can't give you pitchfork points, but I'll give you "verbal" kudos. I'd also note that Pomeroy's models generally predict the "best" team to have a ~25-35% chance of winning. So observing the expected best team winning 28.6% of the 7 observations available (small sample size alert!) is pretty much exactly what Pomeroy would expect to happen: the "best" team won only 28.6% of the time, which is dead in the middle of the range of win probabilities for the best team over that span.

I'll also say this: it is very possible (I would say highly probable) that these models all suffer from substantial limitation in that they are only as good as the data available. As sagegrouse sagely notes, it's quite possible that there was a systematic bias against the ACC based on the limited number of inter-conference results not being as favorable as they should be for the ACC. Further, as someone (I apologize that I can't recall who) pointed out, it's especially hard to rate the West Coast teams vis-a-vis the rest of the country because the West Coast teams played fewer games against the Midwest, South, and East schools. So it's absolutely possible that there are systematic errors inherent based on the available data, and thus the predicted probabilities may indeed be flawed. I'd be surprised if that weren't true, actually.

However, the results of this tournament do not indicate that this is the case, nor do they indicate that it is not the case. Similarly, the results of the last three tournaments don't indicate presence/absence of flaw or error either.


Yeah, CDu, I agree. Muchas gracias a Kedsy for putting this together. Moreover, it is good forever -- all you have to do is add new data one year at a time, and you can become an "instant wizard." Kedsy, however, already being a lawyer and a novelist, doesn't need any more laurels.

I made the observations on the West Coast teams -- they played far fewer games against the major conferences east of the Continental Divide -- which is hardly surprising. Also, the Big 12 had fewer games (IIRC -- the sheets of paper are back in Colorado and I am now sitting in DC for a few weeks), primarily because it did not have a "challenge" against another conference.

I really admire what KenPom has done, and am even jealous 'cuz he found a good way to make a living and spend all his time studying basketball. It is really good to have evaluations just based on a play-by-play analysis of the games on the court -- no adjustments, no factoring in the final score -- just let the data speak. But there is the problem of the dichotomy of the schedules between conference and non-conference games, which makes it hard to do a national rack-up later in the season. The dichotomy is nearly absolute. The only game I found between major conference foes after very early January was the Duke-St. John's game. It's funny -- the ACC has an odd number of teams, so there are gonna be 15 times when a team doesn't have a conference opponent. Every other team just took a "bye" -- only Duke played an inter-conference game to fill the gap.

In statistics-speak, I would probably mumble something like, "the ACC results in the 2015 NCAA Tournament are consistent with the ACC out-performing its early season inter-conference results." But, we have had enough statistics-speak for one thread.

Have a great weekend --
Sage

Kfanarmy
04-24-2015, 04:43 PM
….
Let's take it to the tourney results. Kentucky had a ~40% probability of winning. That meant they had a ~60% probability of losing. So the model got it right! ;)…But even if you want to take the (incorrect) case of saying the model should get the winner more than once in the past 3 years….
You’ve moved the goal post…Didn’t this start out as the winner wasn’t even in the top 5 (which totaled to an 80% probability this year) in two of the three years. I agree with everything in that post. But you’ve changed the argument.

Is it possible (even probable) that these models are flawed? Absolutely. Do the results of one, two, three, or even ten tournaments suggest that the model is flawed? Absolutely not... This is just bad logic. If the models are probably flawed then predictions that are incongruent with reality can certainly "suggest" that the model is flawed.


I think you are making the mistake of thinking that the team with the highest predicted probability of winning is the team that the model predicts will win. That is not (necessarily) the case No..what I am saying is that a model probably has some bugs if what it predicts as "highly likely" normally doesn't happen. (I would say 80% probability is highly likely)


The totals come out:
AP#1: 5 of 37 (13.5%)
AP#1 or AP#2: 9 of 37 (24.3%)
RPI: 1 of 20 (5%)
Pomeroy: 2 of 7 (28.6%)
So, Pomeroy comes out best, but in a much smaller sample so I don't think that proves anything. RPI comes out worst (in a decent but perhaps not even big enough sample), and that makes sense to me.
None of these methods for predicting the champion even get particularly close to a 1 in 3 chance.
Only one of these is a predictive model, right?

i've rolled 2 aces in a row. turned $5 into $4500 doing it. . That’s awesome!

So the Dice and Cards comparisons break down significantly in evaluating the utility of an NCAA game model, because they have proven probabilities…and there are literally millions of validating outcomes. That’s bait I shouldn’t have taken.

Still I watched a dealer win 21 hands in a row in Vegas a few years ago, taking back the winnings of a gentlemen who had amassed a large pile of $100 and above chips. Every player walked out of the pit, emptying the tables. Most weren’t talking about how “lucky” the dealer was. It certainly is possible that an unlikely occurrence happened. It is much more probable the casino was playing with different odds at that moment.
If there is someone out there who believes their NCAA tourney model is as accurate as the mathematical probabilities in dice, pm me I’d like to place some bets next year.

Perhaps someone can explain how none of the predictive models have faults yet they yield differing predictions each and every year. Is the argument simply that the sample size is too small, therefore accuracy is undetermined? If that's the case, can you prove that uniform color isn't the dominant factor in determining NCAA tourney wins? Based on the past 25 tourney champions, I posit that next years winner will be wearing PANTONE 286 or 287 and argue that prediction is more accurate than all of the models cited.

Mtn.Devil.91.92.01.10.15
04-24-2015, 04:57 PM
If I may... the point isn't that the methods are perfect, but rather that calling it imperfect because highest probability didn't occur on one occasion is a much bigger imperfection. There is a ton of data in this thread, lots of real world examples, plenty of analysis, and it even all tied back to basketball at the end.

Is the 538 model perfect? Almost certainly not. Does the Kentucky team losing in the final four prove that? Certainly not.

CDu
04-24-2015, 05:15 PM
You’ve moved the goal post…Didn’t this start out as the winner wasn’t even in the top 5 (which totaled to an 80% probability this year) in two of the three years. I agree with everything in that post. But you’ve changed the argument.

No..what I am saying is that a model probably has some bugs if what it predicts as "highly likely" normally doesn't happen. (I would say 80% probability is highly likely)

There is a 10.4% chance that the winner would come from the 20% group (i.e., outside of the theoretical top-5) in at least 2 of the 3 years. So there is a fairly decent chance that the result we observed could happen even if the models were 100% accurate. We would need a lot more observations to be able to test your hypothesis with any degree of confidence.

If we expand beyond 2013, I'm quite certain that UK was Pomeroy's #1 in 2012. They won. So it's 2 of the last 4. In 2011, I'm quite certain that UConn was NOT in the top-5. So 2 out of 5. I'm 100% sure Duke was in the top 5 (pretty sure they were #1) in 2010. So 3 of 6. UNC was definitely top-5 in 2009 (they were top-2 in the country). Kansas was definitely top-5 in 2008. Florida was almost certainly top-5 in 2007 (they were a #1 seed and a defending champ). Florida was almost certainly NOT top-5 in 2006 (they were I think a 3 seed). UNC was top-5 in 2005 (they were #2 in the country). UConn was almost certainly top-5 in 2004. So we've over the last 12 years, 8 (67%) of the champions were in Pomeroy's top-5 pre-tournament. Again, not nearly sufficient sample size for confidence testing, but that's looking closer and closer to the 80% you would have predicted given an N of infinity.

The results in the last 3 years are heavily influenced by one ginormous outlier (UConn in 2014). When you expand it back further, you see that things look a lot better for Pomeroy. Again, that's not to say that Pomeroy's model is correct. Just that the evidence over a 3-year sample is most certainly insufficient to say it is incorrect.


So the Dice and Cards comparisons break down significantly in evaluating the utility of an NCAA game model, because they have proven probabilities…and there are literally millions of validating outcomes. That’s bait I shouldn’t have taken.

If there is someone out there who believes their NCAA tourney model is as accurate as the mathematical probabilities in dice, pm me I’d like to place some bets next year.

Nobody is saying that models are as accurate as the mathematical probabilities of dice. In fact, nobody is saying the models are right! We're just using a simple and obvious example to show that the models aren't necessarily wrong just based on the results of 1 (or 3) tournaments.

But I don't think the dice/cards example breaks down, as there are lots and lots of validating outcomes of a model predicting tournament results too. No, the probabilities aren't proven. But they don't have to be proven for the purposes of the example. We aren't trying to prove the models are correct. We are trying to illustrate that one (or three) tournaments isn't nearly sufficient evidence to say "the models are wrong!"


Perhaps someone can explain how none of the predictive models have faults yet they yield differing predictions each and every year. Is the argument simply that the sample size is too small, therefore accuracy is undetermined? If that's the case, can you prove that uniform color isn't the dominant factor in determining NCAA tourney wins? Based on the past 25 tourney champions, I posit that next years winner will be wearing PANTONE 286 or 287 and argue that prediction is more accurate than all of the models cited.

Again, nobody is saying that none of the models have faults. In fact, most of us (all of us?) believe that there probably are problems with these models. We are just saying, specifically, that 1, 2, 3, even 10 tournaments is an insufficient sample size to determine if the models are correct or not. And the results of 3 or fewer tournaments is absolutely not sufficient to make statements about the quality of the model.

Kedsy
04-24-2015, 05:41 PM
If we expand beyond 2013, I'm quite certain that UK was Pomeroy's #1 in 2012. They won. So it's 2 of the last 4. In 2011, I'm quite certain that UConn was NOT in the top-5. So 2 out of 5. I'm 100% sure Duke was in the top 5 (pretty sure they were #1) in 2010. So 3 of 6. UNC was definitely top-5 in 2009 (they were top-2 in the country). Kansas was definitely top-5 in 2008. Florida was almost certainly top-5 in 2007 (they were a #1 seed and a defending champ). Florida was almost certainly NOT top-5 in 2006 (they were I think a 3 seed). UNC was top-5 in 2005 (they were #2 in the country). UConn was almost certainly top-5 in 2004. So we've over the last 12 years, 8 (67%) of the champions were in Pomeroy's top-5 pre-tournament. Again, not nearly sufficient sample size for confidence testing, but that's looking closer and closer to the 80% you would have predicted given an N of infinity.

Actually, Florida was #3 in pre-tournament Pomeroy in 2006, so they were in his top five. (I just realized I had a few more years of pre-tournament Pomeroy ratings that I copied from the way-back machine.) I guess the committee didn't pay much attention to Pomeroy when they seeded the Gators that year. I don't have 2008, but if you assume Kansas was in KenPom's top five in 2008 and UConn was in KenPom's top five in 2004, then I can tell you that the national champion came from Pomeroy's top five in 9 of the past 12 seasons (75%).

I'm pretty sure Syracuse was not in KenPom's top five in 2003, but the champions in 2002, 2001, 2000, and 1999 all probably were. And I don't think he published a ratings system before that, so the total percentage of the champion coming from his top five is probably 13 of 17, for 76.5%, which is not too far from 80%, albeit with an N that is probably still too small to draw a conclusion with much confidence.

CDu
04-24-2015, 06:41 PM
Actually, Florida was #3 in pre-tournament Pomeroy in 2006, so they were in his top five. (I just realized I had a few more years of pre-tournament Pomeroy ratings that I copied from the way-back machine.) I guess the committee didn't pay much attention to Pomeroy when they seeded the Gators that year. I don't have 2008, but if you assume Kansas was in KenPom's top five in 2008 and UConn was in KenPom's top five in 2004, then I can tell you that the national champion came from Pomeroy's top five in 9 of the past 12 seasons (75%).

I'm pretty sure Syracuse was not in KenPom's top five in 2003, but the champions in 2002, 2001, 2000, and 1999 all probably were. And I don't think he published a ratings system before that, so the total percentage of the champion coming from his top five is probably 13 of 17, for 76.5%, which is not too far from 80%, albeit with an N that is probably still too small to draw a conclusion with much confidence.

Wow, thanks for the lift (again) Kedsy! So 9 of the last 12 and almost certainly 13 of the last 17. Kind of illustrates how makes statements like "2 of the 3 times the model was wrong means the model is poop" is faulty logic.

Again: not saying the models are wrong. Just that 3 years isn't nearly sufficient sample to say so.

ice-9
04-26-2015, 12:12 PM
Except that the post you are applauding is completely incorrect. You should expect the model to be "wrong" at least two out of three times when the probability of the best team not winning is 60% or greater (as is pretty much always the case). To say otherwise suggests a lack of understanding of probability. We would need LOTS more tournaments before we can say that the model is wrong.

That may or may not be correct. We don't have nearly enough information to say that. The results of the tournament certainly don't prove that.

Duke played substantially and consistently better defense in the tourney than they had at any point in the season. I would argue that it was improbable that Duke would do that for six straight games without hiccup, which is essentially what the model said. Maybe it really was more likely that Duke would become a defensive juggernaut in the tourney. But we would only be able to determine that with a much larger number of tourneys.

My understanding of probability is just fine.

I disagree with your statement about Duke. Duke's vastly improved defense was not a function of probability, but an element that KenPom's model missed / underestimated. And I already posted how I thought Duke was a better team than what KenPom predicted (on a probability basis) vs. San Diego St, Utah and Gonzaga. I could be wrong, but so could predictive statistics based on limited historical data.

I think the KenPom model is flawed and doesn't predict all that well (on a probability basis, a phrase I'm now compelled to include with every statement about prediction after all the outcry in this thread), though I do think it remains one of the best tools.

You think KenPom is awesome and that when something occurs that's unexpected, it's a matter of probability and that the model is just fine. That's OK. It's all good.

It's a free country and we can agree to disagree.

CDu
04-26-2015, 12:30 PM
My understanding of probability is just fine.

I disagree with your statement about Duke. Duke's vastly improved defense was not a function of probability, but an element that KenPom's model missed / underestimated. And I already posted how I thought Duke was a better team than what KenPom predicted (on a probability basis) vs. San Diego St, Utah and Gonzaga. I could be wrong, but so could predictive statistics based on limited historical data.

I think the KenPom model is flawed and doesn't predict all that well (on a probability basis, a phrase I'm now compelled to include with every statement about prediction after all the outcry in this thread), though I do think it remains one of the best tools.

You think KenPom is awesome and that when something occurs that's unexpected, it's a matter of probability and that the model is just fine. That's OK. It's all good.

It's a free country and we can agree to disagree.

See that's the thing: I don't necessarily think Pomeroy's model is great. Not nearly perfect, as pretty much nothing in the world is perfect. So if what you have assertained from this is that I think KenPom is great, then either I haven't been clear or you have misunderstood me (or both). I just disagree with saying that the model got the tournament wrong (it didn't, because it is virtually impossible for a model to get it wrong in a single iteration of a tourney this size), and I disagree that the tournament is proof that the model was wrong. And that a lot of posts in this thread don't seem to suggest a good understanding of why that is the case. That's all.

freshmanjs
04-26-2015, 12:43 PM
My understanding of probability is just fine.

I disagree with your statement about Duke. Duke's vastly improved defense was not a function of probability, but an element that KenPom's model missed / underestimated. And I already posted how I thought Duke was a better team than what KenPom predicted (on a probability basis) vs. San Diego St, Utah and Gonzaga. I could be wrong, but so could predictive statistics based on limited historical data.

I think the KenPom model is flawed and doesn't predict all that well (on a probability basis, a phrase I'm now compelled to include with every statement about prediction after all the outcry in this thread), though I do think it remains one of the best tools.

You think KenPom is awesome and that when something occurs that's unexpected, it's a matter of probability and that the model is just fine. That's OK. It's all good.

It's a free country and we can agree to disagree.

yeah, i certainly don't agree with the statement that comes after the "you think." but, I do think it's absurd that many posts in this thread are drawing conclusions about the model being wrong based on the outcome of one tournament.

House P
04-26-2015, 01:28 PM
I don't have 2008, but if you assume Kansas was in KenPom's top five in 2008 and UConn was in KenPom's top five in 2004, then I can tell you that the national champion came from Pomeroy's top five in 9 of the past 12 seasons (75%).

I'm pretty sure Syracuse was not in KenPom's top five in 2003, but the champions in 2002, 2001, 2000, and 1999 all probably were. And I don't think he published a ratings system before that, so the total percentage of the champion coming from his top five is probably 13 of 17, for 76.5%, which is not too far from 80%, albeit with an N that is probably still too small to draw a conclusion with much confidence.

Interesting take. Thanks for putting this together (with CDu).

Another thing we can do is see how good a particular model is at predicting the probability of a specific team winning a particular matchup.

KenPom.com gives us some information we can use to evaluate his model this way. Before each game, KenPom.com estimates the likelihood that his predicted "favorite" will win the matchup. For example, KenPom.com estimated that Duke had a 94% likelihood of beating Robert Morris, but only a 45% likelihood of beating Wisconsin.

If you look at the "predictions" for each of the 67 games in the 2015 tournament, KenPom.com was remarkably accurate in predicting the likelihood of a particular outcome.

For example, there were 20 games in the 2015 NCAA tournament where KenPom estimated that one team had a 50% to 59% chance of winning. His "favorite" actually won 10 of these 20 games. That’s about one game less than he “expected”. Similarly, his predicted "favorite" won 71% (10 of 14) of the games where they were predicted to have a 60-69% likelihood of winning. That’s about one game more than he expected. If you map this out for the entire 2015 tourney you get the following.




Predicted likelihood of winning
Games
Actual games won
“predicted” games won
Actual winning %
Predicted winning %


90-99%
8
8
7.6
100%
95%


80-89%
12
10
10.2
83%
85%


70-79%
13
10
9.7
77%
75%


60-69%
14
10
8.9
71%
64%


50-59%
20
10
10.9
50%
54%


Total
67
48
47.3
72%
71%



In other words, if KenPom predicted that his favorite would win about 71% of the 67 games in the tournament. In reality, his favorite won 72% of the 67 games.

Now 67 games is still a pretty small sample, so what about a larger sample? If you look at 583 post season games (NCAA, NIT, CBI, CIT) in the past 4 years, KenPom predicted that his “favorite” would win 69% of the games. His favorite actually won 68% of the time.

Here is the full table.




Predicted likelihood of winning
Games
Actual games won
“predicted” games won
Actual winning %
Predicted winning %


90-99%
39
36
36.7
92%
94%


80-89%
88
76
74.2
86%
84%


70-79%
137
97
101.8
71%
74%


60-69%
167
102
107.9
61%
65%


50-59%
152
84
83.4
55%
55%


Total
583
395
404.0
68%
69%



There are probably better ways to approach this data, but looking at things this way, the KemPom model appears to be pretty good at estimating individual games.



I'll say it: the performance model, as estimated by KenPom, was wrong for Duke and the other ACC teams because it totally relied on game stats -- the ACC was far better than the estimates generated by the inter-conference games in November and December. And, no matter how great the teams played in January and February, they were playing against each other, resulting in a zero-sum game where the overall average rating of the ACC remains unchanged.

The phenomenal record of the ACC in the NCAA's (plus Miami in the NIT) should be pretty good evidence of the above.

You may be on to something. Just because the KenPom model appears to be pretty good in general, doesn't mean that it doesn't under- or overestimate certain teams (or conferences).

The KenPom.com model wasn't very accurate with 2015 ACC postseason games. According to the approach described above, the KenPom model predicted that ACC teams should have won 15 of the 26 postseason games they played against teams from other conferences. In reality ACC teams won 20.

Now the sample size is pretty small, so this may be due to a systematic undervaluing of ACC teams or it could be that the ACC just overperformed "by chance". On the one hand, I have a hard time believing the Duke would lose to Utah 45% of the time (as KenPom predicted). On the other hand, much of the ACC's "over-performance" came from teams which barely survived their initial postseason game (UNC vs Harvard, Notre Dame vs Northeastern, Miami vs NC Central, NC State vs LSU).

Des Esseintes
04-26-2015, 04:00 PM
Kudos to CDu, Kedsy, Duke95, freshman, Mountain, and sage (among others) for fighting the good fight on behalf of basic math in this thread. That you have utterly failed to carry your point with your interlocutors cannot be laid at your doorstep. Some people just hate math when it disagrees with their gut. There're some fantastic data and some explanations of probability here. I would long since have chucked my keyboard through a window out of frustration, and I admire your collective patience and desire to educate.

sagegrouse
04-26-2015, 04:13 PM
Originally Posted by sagegrouse View Post
I'll say it: the performance model, as estimated by KenPom, was wrong for Duke and the other ACC teams because it totally relied on game stats -- the ACC was far better than the estimates generated by the inter-conference games in November and December. And, no matter how great the teams played in January and February, they were playing against each other, resulting in a zero-sum game where the overall average rating of the ACC remains unchanged.

The phenomenal record of the ACC in the NCAA's (plus Miami in the NIT) should be pretty good evidence of the above.

You may be on to something. Just because the KenPom model appears to be pretty good in general, doesn't mean that it doesn't under- or overestimate certain teams (or conferences).

The KenPom.com model wasn't very accurate with 2015 ACC postseason games. According to the approach described above, the KenPom model predicted that ACC teams should have won 15 of the 26 postseason games they played against teams from other conferences. In reality ACC teams won 20.

Now the sample size is pretty small, so this may be due to a systematic undervaluing of ACC teams or it could be that the ACC just overperformed "by chance". On the one hand, I have a hard time believing the Duke would lose to Utah 45% of the time (as KenPom predicted). On the other hand, much of the ACC's "over-performance" came from teams which barely survived their initial postseason game (UNC vs Harvard, Notre Dame vs Northeastern, Miami vs NC Central, NC State vs LSU).

This is not really a "sample size" issue or, necessarily, a modeling issue. The set of games available from which to estimate capability and performance constitutes a really lousy experimental design: (1) There are relatively few games between teams from different major conferences. (2) All of these occurred between November 14 and January 3 (or so), except for one game (Duke-St. John's). (3) The performance of teams in November and December may not be indicative of how they perform in March. (4) There is absolutely nothing that Pomeroy or Sagarin or any person developing and estimating a model based on game performance can do to correct the problem caused by the absence of good data.

In the case at hand, the ACC was probably a lot better than what the early-season results would lead one to believe. Moreover, there is no reason to expect November and early-December results to be particularly definitive, no matter how many games are played back then.