BPI > RPI, Kenpom and Sagarin? (Duke current #1)

[JNort]

http://espn.go.com/mens-college-bask...ndex-explained

New system that ranks teams and accounts for missing players. Goes more in depth than Kenpom. What do yall think?

[Bob Green]

I'm sorry but all these computer based rankings don't get me excited. I'm all for the old eye test. This new BPI says Duke is #1 and Miami is #9 but we all saw Miami beat Duke by 27 points in a head-to-head match-up.

[sporthenry]

I'm sorry but all these computer based rankings don't get me excited. I'm all for the old eye test. This new BPI says Duke is #1 and Miami is #9 but we all saw Miami beat Duke by 27 points in a head-to-head match-up.

In games with their full starting 5, Miami is actually #1 (a little bit ahead of Kelly).

[Listen to Quants]

http://espn.go.com/mens-college-bask...ndex-explained

New system that ranks teams and accounts for missing players. Goes more in depth than Kenpom. What do yall think?

Interesting. I like the adjustment for absent player(s). Not that it needs emphasis, but Duke's recent experience with Kelly demonstrates the importance of such an absence. I also like the 'reward' the power index gives for winning. Often a team will sacrifice points in a lead in order to run the clock out and, arguably (and argued much on this board), preserve a win. The win then contains information the final score gap does not.

Perhaps most interesting was that ESPN seems to have run their BPI off against Sagarin and RPI in the last 5 years of tourny contests. I was surprised as how well RPI did, I thought it was more flawed than that. Perhaps all these are flawed though since in a seeded tournament an awful lot of the games are 'no brainers' and the performances of the indexes, just below 75% correct is not impressive (random is of course, 50%). Sadly, nothing was said about the Vegas line. I'd bet strongly that it did much better. Also unfortunate is that they did not include a link (or I missed the link) to exact formula or explanation.

[vick]

http://espn.go.com/mens-college-bask...ndex-explained

New system that ranks teams and accounts for missing players. Goes more in depth than Kenpom. What do yall think?

It's tough to evaluate without knowing how they account for injuries or lower the impact of blowouts, which I think are clearly the key weaknesses of Kenpom. I would note though that Dean Oliver, the author of that article, is really, really smart--his Basketball on Paper is like the Bible of the statistical analysis of basketball (most of what Pomeroy does is derived from it, as he freely acknowledges), so if he says it's worth looking at, I sure believe him.

[JNort]

I'm sorry but all these computer based rankings don't get me excited. I'm all for the old eye test. This new BPI says Duke is #1 and Miami is #9 but we all saw Miami beat Duke by 27 points in a head-to-head match-up.

Yeah I know what ya mean but many people love the stat side of it and all the rankings. We also saw Miami Gulf Coast beat Miami. I think Duke wins comfortably in the match up in Durham.

[Kedsy]

Perhaps most interesting was that ESPN seems to have run their BPI off against Sagarin and RPI in the last 5 years of tourny contests.

Since RPI was used to make the seedings, there's at least a little bit of self-fulfilling prophecy going on there. And the main question I'd ask about testing the BPI on past tournaments is did they use ratings that included the tournament games? That's what a lot of people seem to do when they backtest Pomeroy and it makes for flawed analysis.

[Listen to Quants]

Since RPI was used to make the seedings, there's at least a little bit of self-fulfilling prophecy going on there. And the main question I'd ask about testing the BPI on past tournaments is did they use ratings that included the tournament games? That's what a lot of people seem to do when they backtest Pomeroy and it makes for flawed analysis.

Absolutely right. testing on the data bed that created the model is (or should be) forbidden. However, the average team plays about 2 tourny games and 30 regular season games so if they are unweighted the touney results won't affect things too much. The basic results from my perspective though, was all these systems are pretty bad.

[Wander]

So, they basically stole kenpom's method and added a few minor adjustments?

I've always felt that injuries/suspensions shouldn't be taken into account at all for selection and seeding purposes, though I'm probably in the minority in that one. I guess it's OK for a computer rating system, but tough to say without knowing how they do it.

Anyway, the coolest part about kenpom isn't the straight-up rankings or using it to try and predict games (as Kedsy has argued before, it's probably not that much better than the RPI or other methods, if at all), but the detailed and objective breakdown of how good a team is at various parts of the game. And that doesn't seem to be present here.

[sporthenry]

Absolutely right. testing on the data bed that created the model is (or should be) forbidden. However, the average team plays about 2 tourny games and 30 regular season games so if they are unweighted the touney results won't affect things too much. The basic results from my perspective though, was all these systems are pretty bad.

I think Kenpom has said as much about using pre-tourney data but the problem with the data after tourney is how much value does he put on it? For the team that eventually wins it, that is 6 more wins, including 3-4 against ranked teams and 2-3 games against top 10 teams. And it is unclear how much weight he puts on these top games.

So, they basically stole kenpom's method and added a few minor adjustments?

I've always felt that injuries/suspensions shouldn't be taken into account at all for selection and seeding purposes, though I'm probably in the minority in that one. I guess it's OK for a computer rating system, but tough to say without knowing how they do it.

I have no problem with taking Kenpoms system and adjusting with it. Getting beyond that this is how most things are created/made better, Kenpom has acknowledged flaws in his system but done little to address them.

As far as injuries, I have no problems accounting for it. Not only is it the committee's job to put the best teams in the tourney but a corollary to that would be to put the better team in a better seeding position. If two bubble teams have identical resumes but one missed their leading scorer, it is clearer who the better team is. Not to mention, this doesn't even address the equity or inequity that would happen without taking into account injury. Imagine Withey gets injured in the Big 12 tournament and is done for the year. But their resume commanded a 2 seed. Whatever #1 seed in their bracket would gain an unfair advantage over other #1 seeds b/c of the weakness of their 2 seed.

[Kedsy]

Absolutely right. testing on the data bed that created the model is (or should be) forbidden. However, the average team plays about 2 tourny games and 30 regular season games so if they are unweighted the touney results won't affect things too much.

You'd be surprised. For example, Butler in Pomeroy's 2010 ratings went from 26th going into the tournament to 12th in his final ratings. UConn in 2011 went from 17th to 10th (their defense went from 31st to 14th). Butler in 2011 went from 54th to 41st; VCU in 2011 went from 84th to 52nd. In 2009, coming into the tournament Duke was Pomeroy's #7 and Villanova was #19. In his final rankings, Duke was #11 and Villanova #14, making it look much closer than he presumably predicted pre-tournament.

And I realize the ordinal rank isn't as important as the rating, but obviously the ratings changed too. My point is if ESPN ran a system not that different from Pomeroy's and backtested it, they can't really say their model is more predictive than anybody's unless they used pre-tournament ratings.

[throatybeard]

You'd be surprised. For example, Butler in Pomeroy's 2010 ratings went from 26th going into the tournament to 12th in his final ratings. UConn in 2011 went from 17th to 10th (their defense went from 31st to 14th). Butler in 2011 went from 54th to 41st; VCU in 2011 went from 84th to 52nd. In 2009, coming into the tournament Duke was Pomeroy's #7 and Villanova was #19. In his final rankings, Duke was #11 and Villanova #14, making it look much closer than he presumably predicted pre-tournament.

And I realize the ordinal rank isn't as important as the rating, but obviously the ratings changed too. My point is if ESPN ran a system not that different from Pomeroy's and backtested it, they can't really say their model is more predictive than anybody's unless they used pre-tournament ratings.

Well, I dunno. Six games is a lot of games out of thirty-someodd. An NCAAT run could change some things.

[Kedsy]

Well, I dunno. Six games is a lot of games out of thirty-someodd. An NCAAT run could change some things.

I agree, but then one can't go back using the final ratings (including those last six games) and say the system correctly predicted anything, is all I'm saying.

[Wander]

Not all games are weighted equal in kenpom - more recent games are given more value. I don't know to what degree that weighting is done, but that's part of what's going on with the NCAA tournament games affecting the ratings thing.

[moonpie23]

I think Duke wins comfortably in the match up in Durham.

these kind of predictions seem to be falling flat.

[Listen to Quants]

I agree, but then one can't go back using the final ratings (including those last six games) and say the system correctly predicted anything, is all I'm saying.

Technically correct. I may be beating this to death, but the two examples you give (Butler, UConn) are both 6 game runs, three times as much weight as the average team, and thus fairly extreme examples (ancedote.plural.data.not). Again, we don't know if this is an unweighted system. I don't even know if the system includes the tourney at all (?). But even if they made that mistake, the 2/35ths (or so) 'cheat' isn't massive (on average which is how they rated their systems). Then again, the detail I'm discussing isn't that massive either, I think.

[Kedsy]

Technically correct. I may be beating this to death, but the two examples you give (Butler, UConn) are both 6 game runs, three times as much weight as the average team, and thus fairly extreme examples (ancedote.plural.data.not). Again, we don't know if this is an unweighted system. I don't even know if the system includes the tourney at all (?). But even if they made that mistake, the 2/35ths (or so) 'cheat' isn't massive (on average which is how they rated their systems). Then again, the detail I'm discussing isn't that massive either, I think.

I don't know, either. But I do know that if you apply the published Pomeroy rankings to past NCAA tournament brackets, they perform significantly better than if you use pre-tournament rankings, and that's what we're talking about for the BPI.

The "average" two game run is less relevant than it sounds, because the low-seeded first round losers would have been picked to lose anyway, but the prediction on the relatively close games has a good chance of changing if you take "future" tournament results into account. I know you know all this, but I think you're discounting the impact of using the tournament games when you backtest, at least for Pomeroy and I'd guess for BPI too.

[hurleyfor3]

Dork polls have gotten waaaaaaaaaaaaaay too mainstream.

[Indoor66]

Dork polls have gotten waaaaaaaaaaaaaay too mainstream.

And they are used to support each position in every discussion - as if they are Truth.

[NSDukeFan]

And they are used to support each position in every discussion - as if they are Truth.

I am pretty sure they are only relevant when they support my position.