Dork Polls

[rtnorthrup]

I have long been a follower and admirer of KenPom's site for a while now. In fact, I probably found it through DBR. In the early years, KenPom was a welcome deviation from the flawed RPI polls. Recently, a bunch of new dork polls have come out, each with their own algorithm.

This year, there seems to be some wildly differing opinion in many of the dork polls, and it is going to be interesting to see how it pans out.

Tonight is a great example. KenPom has UNO #178 (-.52 AdjEM) and Mt. St. Mary's at #213 (-4.23 AdjEM). BPI currently has Mt. St. Mary's about a 1.5 point favorite, and FiveThirtyEight gives a slight edge to Mt. St. Mary's as well in their ELO ratings.

Should be interesting to see how accurate some of these ratings are.

[budwom]

I have long been a follower and admirer of KenPom's site for a while now. In fact, I probably found it through DBR. In the early years, KenPom was a welcome deviation from the flawed RPI polls. Recently, a bunch of new dork polls have come out, each with their own algorithm.

This year, there seems to be some wildly differing opinion in many of the dork polls, and it is going to be interesting to see how it pans out.

Tonight is a great example. KenPom has UNO #178 (-.52 AdjEM) and Mt. St. Mary's at #213 (-4.23 AdjEM). BPI currently has Mt. St. Mary's about a 1.5 point favorite, and FiveThirtyEight gives a slight edge to Mt. St. Mary's as well in their ELO ratings.

Should be interesting to see how accurate some of these ratings are.

I'm a big fan of KenPom but can't help but laugh at how his algorithm continues to rank Virginia at #7 despite having lost what, five of nine? Still a solid team, and excellent defensively, but
they most certainly are not a top ten team now...

[rtnorthrup]

I'm a big fan of KenPom but can't help but laugh at how his algorithm continues to rank Virginia at #7 despite having lost what, five of nine? Still a solid team, and excellent defensively, but
they most certainly are not a top ten team now...

Yes, agreed. His UVa ranking and Wichita State ranking are certainly two that caught my eye.

[Kedsy]

Yes, agreed. His UVa ranking and Wichita State ranking are certainly two that caught my eye.

I have more questions about his rankings of SMU (Pomeroy #11) and St. Mary's (Pomeroy #14). I mean, his rankings have SMU above Duke and St. Mary's above Arizona. These two teams, with their slow pace and poor schedule, are potentially poster-children for the kind of team for which Pomeroy's system just doesn't work. And if, e.g., St. Mary's isn't close to that good, what does that say about Gonzaga?

FWIW, Sagarin has SMU at #19 and St. Mary's at #25 (also Virginia at #9 and Wichita at #11) so not that far off from Pomeroy. RPI has them at #13 and #17. Maybe teams like these are problematic in any computer system. Or maybe they're legit. I'm not sure how to tell, either way.

[throatybeard]

Surely, a single elimination tournament will clear up these questions of legitimacy/fraudulence! 😂😂😂

[-jk]

Still like to look at Massey's composite - only it's not updated realtime...

-jk

[uh_no]

Surely, a single elimination tournament will clear up these questions of legitimacy/fraudulence! ������

so I think the biggest mistake I see on evaluating dork polls is whether they got a game right or wrong. That's the wrong way to look at it.

Suppose poll A picked 5/5 games right which it assigned 100% probability to, and 5/10 games right which it assigned 51% probability to. Another poll got 1/5 games right it assigned 100% probability to and 10/10 games right it assigned 51% probability to. Which poll did better?

The second poll got more games "right" (11-10) than the second poll...but the first poll nailed all of its 100% games, and got 50% on it's 51% games, which is effectively a flip of a coin. The second poll failed miserably at it's 100% games and happened to flip heads 10 times in a row on its 50% games...it's estimation of how likely a win was was WILDLY wrong. If the tournament were run again, it's far more likely the first system does better, since the second system is unlikely to get every single one of its 50% games right again.

In the short run, one system did well, but in the long run, the other system is far likely to have done better. A similar argument can be made about forecasting presidential elections...everybody was wrong last year, but 538 did "better" because they gave the outcome 30% instead of 5% chance. In short: percentages matter when evaluating a predictor...a lot...in terms of gauging future success.

One simple way to then evaluate is to bucket every game based on it's confidence interval (50%...60%...100%) and see of all the the games that were, say, predicted between 60 and 70%. How many times did the system predict that game correctly? A "perfect" statistical system would get 6/10 games it put 60% probability correct. With 67 games at a variety of imbalances, we should be able to get an idea of which system will be a good predictor in the long run.

But we have more data, the closeness of the games. A predictor which says a game was 50%, and the game was within a point or two almost assuredly did better than one that gave it 100% chance. We would expect a high correlation between win probability and point difference. The problem here is how to deal with differing tempi. Without adjusting for tempo, a game you put at 50% chance will naturally have far higher variance in point difference than one with fewer possessions. On the other hand, that should be reflected with a higher win percentage for the better team (law of large numbers). So because of that, there will be no adjustment for pace here. We'll simply take the point margin. We won't account for outliers since they should impact each system the same (in terms of big upsets), and if one system was way out of whack...then, well...tough luck...you should be penalized for being way out of whack with what actually happened.

Before the data is lost, here are the pregame probabilities for tonight. If anyone is interested in other ones, please grab the % from the start of the game. For SAG It doesn't seem he does win %, so I grabbed his point spreads. We can convert between the point spread and the winning percentages using the same method as KP (since SAG doesn't provide his own), which was determined via a regression seen here:
kp win percentages.jpg

but I'll do that conversion later.

	KP	538	bpi	sag
wake v ksu	ksu 51	wake 52	wake 56	wake 1.81 2.94 0.19
NO v MSM	NO 59	MSM 55 (anyone have exact #??)	MSM 55	NO 2.32 4.20 0.41

[ice-9]

I've gotten a lot of flak about this before on DBR, but in my view people put WAY too much credence on these dork polls.

I don't have an issue with the math or the projections of probabilities -- mostly agree with uh_no's post above -- but there simply isn't enough data to properly model 20, 30 games of a college basketball team to accurately forecast win probabilities.

"Team X is likely to beat Team Y." OK sure.

"Team X has a 84.245% chance of beating Team Y." Yeah...maybe. If Team X actually lost, the model isn't robust enough for us to say Team X lost because we just so happened to observe the 15% outcome.

Just look at how much these probabilities fluctuate throughout the season. The fact that it changes so much is an indication that the model hasn't really arrived at a "truth," that it's figuring things out with new information as we all are.

Make no mistake, these dork polls are a MASSIVE improvement over the AP/Coach's polls and the "eye" test of random people. Which is probably we've all gravitated towards these systems, but to me it's more guidance than religion.

I have no problem thinking KenPom might've simply incorrectly evaluated Team X, at least relative to Team Y.

[sagegrouse]

I've gotten a lot of flak about this before on DBR, but in my view people put WAY too much credence on these dork polls.

I don't have an issue with the math or the projections of probabilities -- mostly agree with uh_no's post above -- but there simply isn't enough data to properly model 20, 30 games of a college basketball team to accurately forecast win probabilities.

"Team X is likely to beat Team Y." OK sure.

"Team X has a 84.245% chance of beating Team Y." Yeah...maybe. If Team X actually lost, the model isn't robust enough for us to say Team X lost because we just so happened to observe the 15% outcome.

Just look at how much these probabilities fluctuate throughout the season. The fact that it changes so much is an indication that the model hasn't really arrived at a "truth," that it's figuring things out with new information as we all are.

Make no mistake, these dork polls are a MASSIVE improvement over the AP/Coach's polls and the "eye" test of random people. Which is probably we've all gravitated towards these systems, but to me it's more guidance than religion.

I have no problem thinking KenPom might've simply incorrectly evaluated Team X, at least relative to Team Y.

Agree, totally. Ken Pomeroy is a genius, and I am jealous of his success. His stunning breakthrough was to come up with a way of distilling a team's performance on both offense and defense, using points scored and allowed per 100 possessions, adjusted for the quality of the opposition measured in a similar way. These describe, in a common sense way, the performance of a team over the set of games included in the calculations.

Nevertheless, his stats (yes, these are statistics) have severe limitations:

1. These are two scalar values (times two) that, so his advocates maintain, purport to capture all the information needed to predict a game. Really? Going into a game there are only two values for each team that are important (plus a plug for home court advantage)? Basketball is a complex game with hundreds if not thousands of actions affecting the outcome of a game. How could any two (or four numbers) fully capture the essence of the competition?

2. The KenPom stats have other omitted niceties that can possibly be measured: injuries, match-ups, momentum, different rates of improvement of teams over the course of the season. Injuries and match-ups are the big daddies on this list, but the others are important as well.

3. Then there is my "fools errand" argument, which I gleefully apply to the task facing the Tournament Selection Committee every March. All play (with only a few exceptions) after January 1 is within conferences. Setting aside these exceptions (and a salute to the February Big 12-SEC challenge), no one has any way to know how teams compare between conferences after January 1, if teams' capabilities change significantly.

4. I can offer the special variant of "isolated populations," applying this year mostly to Gonzaga. Even beyond the pre-Jan. 1/post- Jan. 1 issue, the West Coast Conference has very little overlap in scheduling with the power conferences (five majors plus Big East), so their rankings relative to other conferences is suspect -- only because there is very little data. Kudos, nevertheless, for the Zags' December win over Arizona at the Staples Center and the Thanksgiving wins over Florida and Iowa State. Other top WCC teams went only 3-3 against, as it turns out, mid-level teams from the power conferences (only USC made the NCAA's, as an 11 seed). The "isolated population" problem is not a limitation only on the stat-based so-called Dork Polls; it also affects every other measure, including "expert eyeballs."



I first came

[COYS]

2. The KenPom stats have other omitted niceties that can possibly be measured: injuries, match-ups, momentum, different rates of improvement of teams over the course of the season. Injuries and match-ups are the big daddies on this list, but the others are important as well.

I wonder if there is a way to use KenPom's existing data to start adjusting probabilities for matchups. I'm speculating entirely here, but perhaps using the various stats KenPom has for individual players (height, rebound rates, minute percentages, usage rates, field goal percentage and shot type broken down between 2pt fga and 3 pts fga, etc) could be used to get a picture of the basic "type" for a given team. He already creates depth charts based on this data. For example, the depth chart for Duke clearly shows that Jayson Tatum plays the majority of his minutes (83%) at PF, meaning it's pretty easy to peg Duke as a 4-out-1-in offense with multiple high volume three point shooters. Meanwhile, his depth chart created off of UNC's individual stats clearly shows a 2-big lineup with Hicks, May, Meeks, and Bradley (all four of whom do not shoot threes) using up virtually all of the available minutes in the post. I'm not saying this is actually the case, but is it possible that a more detailed analysis would show that UNC is more likely to fail to meet their expected efficiency against a 4-out offense while Duke tends to do better against teams with 2 traditional big guys (Duke did well against Louisville, UVA, and UNC, three teams that do not really use a stretch 4)? It's possible there is already enough information in KenPom's database to begin accounting for matchups. I would be really curious to see Pomeroy explore this angle.

[throatybeard]

I wonder if there is a way to use KenPom's existing data to start adjusting probabilities for matchups. I'm speculating entirely here, but perhaps using the various stats KenPom has for individual players (height, rebound rates, minute percentages, usage rates, field goal percentage and shot type broken down between 2pt fga and 3 pts fga, etc) could be used to get a picture of the basic "type" for a given team. He already creates depth charts based on this data. For example, the depth chart for Duke clearly shows that Jayson Tatum plays the majority of his minutes (83%) at PF, meaning it's pretty easy to peg Duke as a 4-out-1-in offense with multiple high volume three point shooters. Meanwhile, his depth chart created off of UNC's individual stats clearly shows a 2-big lineup with Hicks, May, Meeks, and Bradley (all four of whom do not shoot threes) using up virtually all of the available minutes in the post. I'm not saying this is actually the case, but is it possible that a more detailed analysis would show that UNC is more likely to fail to meet their expected efficiency against a 4-out offense while Duke tends to do better against teams with 2 traditional big guys (Duke did well against Louisville, UVA, and UNC, three teams that do not really use a stretch 4)? It's possible there is already enough information in KenPom's database to begin accounting for matchups. I would be really curious to see Pomeroy explore this angle.

So, to use a baseball analogy, sort of like PECOTA scores, but for teams rather than individual players?

[COYS]

So, to use a baseball analogy, sort of like PECOTA scores, but for teams rather than individual players?

Actually, that's a great analogy. KenPom's current database might be limited in terms of the detail you can get from a team. For example, Jayson Tatum is a very different stretch 4 compared to say Ryan Kelly in 2013. Yet they both shot threes and got to the line at fairly similar rates. I'm not sure how you'd differentiate between the two even though they present radically different matchup scenarios.

I remember this article from a few years back that divided NBA players up into 13 different positions, with names like "Offensive ball-handler," "defensive ball-handler," "paint-protector" etc. I wonder if KenPom used something similar to create categories for players if he could hash out a general style of play for each team. Then we might find out that Duke struggles on defense against teams with "scoring rebounders" (a category I bet John Collins would fit into with his diverse offensive game AND excellence on the glass) and "offensive ball-handlers" (Perhaps Dennis Smith?) who can put our defense under pressure in PnR situations. That could allow for more accurate predictions based on matchups. For example, perhaps Duke would be favored in a game against UVA even though UVA has better adjusted efficiency numbers for a season.

That type of analysis would also allow Kenpom to better control for injuries, provided the injured player plays enough games to get an idea of their statistical profile. We know that Duke is better with Amile Jefferson on the court, obviously, but maybe we'd gain some insight into specifically why Duke matched up so much better against Louisville with Amile Jefferson than without.

[Indoor66]

Actually, that's a great analogy. KenPom's current database might be limited in terms of the detail you can get from a team. For example, Jayson Tatum is a very different stretch 4 compared to say Ryan Kelly in 2013. Yet they both shot threes and got to the line at fairly similar rates. I'm not sure how you'd differentiate between the two even though they present radically different matchup scenarios.

I remember this article from a few years back that divided NBA players up into 13 different positions, with names like "Offensive ball-handler," "defensive ball-handler," "paint-protector" etc. I wonder if KenPom used something similar to create categories for players if he could hash out a general style of play for each team. Then we might find out that Duke struggles on defense against teams with "scoring rebounders" (a category I bet John Collins would fit into with his diverse offensive game AND excellence on the glass) and "offensive ball-handlers" (Perhaps Dennis Smith?) who can put our defense under pressure in PnR situations. That could allow for more accurate predictions based on matchups. For example, perhaps Duke would be favored in a game against UVA even though UVA has better adjusted efficiency numbers for a season.

That type of analysis would also allow Kenpom to better control for injuries, provided the injured player plays enough games to get an idea of their statistical profile. We know that Duke is better with Amile Jefferson on the court, obviously, but maybe we'd gain some insight into specifically why Duke matched up so much better against Louisville with Amile Jefferson than without.

I do not believe that it is possible to account for all of the permutations that a basketball team provides. If you account for all of the on-court talents and tendencies, you then have to factor in situations and coaching decisions. No way to account for all of the factors.

[uh_no]

I do not believe that it is possible to account for all of the permutations that a basketball team provides. If you account for all of the on-court talents and tendencies, you then have to factor in situations and coaching decisions. No way to account for all of the factors.

It doesn't have to be perfect, though. It just has to be more insightful than not using any of that data.

It may turn out that the normal variance of a game outcome more than trumps any additional insight that a given system can provide. Of course, as a dork, I would LOVE to see such data anyway.

[COYS]

It doesn't have to be perfect, though. It just has to be more insightful than not using any of that data.

It may turn out that the normal variance of a game outcome more than trumps any additional insight that a given system can provide. Of course, as a dork, I would LOVE to see such data anyway.

Exactly. We're talking about simply trying to quantify some of the same facets of the game that we can plainly see with our eyes. I mean, Ryan Kelly and Jayson Tatum are clearly different players that put pressure on opposing defenses in different ways despite both being stretch 4's. It's possible a deeper analysis of the data would give us more insight into exactly how their styles affected the outcomes of games. It's possible that it wouldn't. I'm not worried about perfectly predicting every outcome. Just as uh_no says, it's entirely likely that during what is a very short college basketball season, there simply won't be enough data to overcome the built-in variance of a basketball game. But, just like uh_no, I also want to see someone try.

[-jk]

Does SportVu fit into this somewhere? Surely the NBA (and some elite NCAA teams) get more nuanced data than KenPom.

-jk

[Billy Dat]

SI's Luke Winn with some cool quant analysis of the contenders for the title
http://www.si.com/college-basketball...low_twitter_si

[Listen to Quants]

Agree, totally. Ken Pomeroy is a genius, and I am jealous of his success. His stunning breakthrough was to come up with a way of distilling a team's performance on both offense and defense, using points scored and allowed per 100 possessions, adjusted for the quality of the opposition measured in a similar way. These describe, in a common sense way, the performance of a team over the set of games included in the calculations.

Nevertheless, his stats (yes, these are statistics) have severe limitations:

1. These are two scalar values (times two) that, so his advocates maintain, purport to capture all the information needed to predict a game. Really? Going into a game there are only two values for each team that are important (plus a plug for home court advantage)? Basketball is a complex game with hundreds if not thousands of actions affecting the outcome of a game. How could any two (or four numbers) fully capture the essence of the competition?

2. The KenPom stats have other omitted niceties that can possibly be measured: injuries, match-ups, momentum, different rates of improvement of teams over the course of the season. Injuries and match-ups are the big daddies on this list, but the others are important as well.

3. Then there is my "fools errand" argument, which I gleefully apply to the task facing the Tournament Selection Committee every March. All play (with only a few exceptions) after January 1 is within conferences. Setting aside these exceptions (and a salute to the February Big 12-SEC challenge), no one has any way to know how teams compare between conferences after January 1, if teams' capabilities change significantly.

4. I can offer the special variant of "isolated populations," applying this year mostly to Gonzaga. Even beyond the pre-Jan. 1/post- Jan. 1 issue, the West Coast Conference has very little overlap in scheduling with the power conferences (five majors plus Big East), so their rankings relative to other conferences is suspect -- only because there is very little data. Kudos, nevertheless, for the Zags' December win over Arizona at the Staples Center and the Thanksgiving wins over Florida and Iowa State. Other top WCC teams went only 3-3 against, as it turns out, mid-level teams from the power conferences (only USC made the NCAA's, as an 11 seed). The "isolated population" problem is not a limitation only on the stat-based so-called Dork Polls; it also affects every other measure, including "expert eyeballs."



I first came

All good points. BPI used to try to discount the performance of teams in games with an injury if the player was a high use player and did return eventually. I don't know if they still do that.

More interestingly, matchups. You could, with work, actually do a big analysis of the teams that play each other twice to find the average separation expected without accounting for the variable (same matchup as before). If the average score difference was tighter than expected you'd have a measure of matchup import. It's crude by the direction should be right.

KenP, want to do the work?

[mr. synellinden]

I raised this in another thread - teams change so much during the course of the season - Duke is a great example, Arizona got Trier back. Florida lost their big man. And injuries are a major factor - for example what is the impact to Oregon of losing Boucher. I would love to see KenPom data that showed performance during the last 10 and last five games.

I would bet that if you looked only at the last five games, Duke would be in the top 3 if not #1 in KenPom's rankings.

[duke4ever19]

Does SportVu fit into this somewhere? Surely the NBA (and some elite NCAA teams) get more nuanced data than KenPom.

-jk

Speaking of KenPom, did we just move down another step on his rankings? I swear we were at #12 right behind SMU, but I looked just now and we are at #13.

If true, it's hard to believe this team went down in his rankings since the ACC Tournament, rather than up.