PDA

View Full Version : Some New Year's NET Insights



scottdude8
01-01-2019, 05:58 PM
I've found myself fascinated with the new NET rankings, love them or hate them... specifically, the details that the NCAA is providing in what they call the "NET Nitty Gritty (https://extra.ncaa.org/solutions/rpi/Stats%20Library/NET%20Nitty%20Gritty%20-%20Games%20through%20December%2031,%202018.pdf)". Some highlights:


Duke is No. 1 in the current NET, followed by Virginia. N.C. State is (somehow?) at No. 8. Va. Tech, Florida State and Louisville are also in the top 26. Go ACC!
However, the difference between the strength of Duke and Virginia's non-conference schedule is pretty staggering... Duke's is 12th, while Virginia's is 223. The other major conference undefeated team, my Wolverines, also have a disappointing non-conference SOS (202), although we've also already played two conference games, including a road tilt.
When it comes to the all important "Quadrant 1" wins, Duke is still tied for the most with 4, and our loss to Gonzaga also counts as a Q1 loss. Michigan now has four quad 1 wins as well, along with Wisconsin... but again, the early B1G conference games probably played a key role there. The only teams with three Q1 wins are Kansas and Marquette.
The two undefeated mid-majors, Houston and Nevada, are ranked oddly... Houston is No. 4, while Nevada (the team near-universally though to be the better of the two) is down at No. 11. This is even more unusual considering Nevada's NC SOS is 30th, while Houston's is 161. If you're looking for an example where the NET has gone wonky, this may be the best one!
Another odd ranking: Nebraska at No. 10 despite another weak NC SOS. The fact that their two losses are Q1 losses probably factored into that a bit.


I'll keep posting these NET rundowns every couple weeks since, for some unknowable reason, I find writing these summaries and sifting through those NET spreadsheets a relaxing break from my research, haha. Feel free to add in any interesting tidbits you've found in the rankings, or your pro or anti NET arguments based on what I've found!

HereBeforeCoachK
01-01-2019, 07:05 PM
I've found myself fascinated with the new NET rankings, love them or hate them... specifically, the details that the NCAA is providing in what they call the "NET Nitty Gritty (https://extra.ncaa.org/solutions/rpi/Stats%20Library/NET%20Nitty%20Gritty%20-%20Games%20through%20December%2031,%202018.pdf)". Some highlights:


Duke is No. 1 in the current NET, followed by Virginia. N.C. State is (somehow?) at No. 8. Va. Tech, Florida State and Louisville are also in the top 26. Go ACC!
However, the difference between the strength of Duke and Virginia's non-conference schedule is pretty staggering... Duke's is 12th, while Virginia's is 223. The other major conference undefeated team, my Wolverines, also have a disappointing non-conference SOS (202), although we've also already played two conference games, including a road tilt.
When it comes to the all important "Quadrant 1" wins, Duke is still tied for the most with 4, and our loss to Gonzaga also counts as a Q1 loss. Michigan now has four quad 1 wins as well, along with Wisconsin... but again, the early B1G conference games probably played a key role there. The only teams with three Q1 wins are Kansas and Marquette.
The two undefeated mid-majors, Houston and Nevada, are ranked oddly... Houston is No. 4, while Nevada (the team near-universally though to be the better of the two) is down at No. 11. This is even more unusual considering Nevada's NC SOS is 30th, while Houston's is 161. If you're looking for an example where the NET has gone wonky, this may be the best one!
Another odd ranking: Nebraska at No. 10 despite another weak NC SOS. The fact that their two losses are Q1 losses probably factored into that a bit.


I'll keep posting these NET rundowns every couple weeks since, for some unknowable reason, I find writing these summaries and sifting through those NET spreadsheets a relaxing break from my research, haha. Feel free to add in any interesting tidbits you've found in the rankings, or your pro or anti NET arguments based on what I've found!


So let's get this straight. Q1 is what, about 80 teams? So you think it's Nebraska's two LOSSES to TOP 80 teams is what has them at Number 10? Oh that makes perfect sense.

SCMatt33
01-01-2019, 07:22 PM
So let's get this straight. Q1 is what, about 80 teams? So you think it's Nebraska's two LOSSES to TOP 80 teams is what has them at Number 10? Oh that makes perfect sense.

Q1 is meant to represent top 50, but adjusted for location. IIRC, it’s top-35 at home, top 50 neutral, and top 75 road.

Also, remember that the new NET system accounts for more than who did you play and who did you beat. Things like margin of victory (capped at 10) and raw offensive and defensive efficiency (not capped) are factored in as well.

Bluedog
01-01-2019, 08:15 PM
NET may not be perfect or always make the most sense, but I think it's clear that it's a massive improvement over RPI so I at least give the NCAA credit for that. RPI lasted far too long though.

scottdude8
01-01-2019, 09:10 PM
So let's get this straight. Q1 is what, about 80 teams? So you think it's Nebraska's two LOSSES to TOP 80 teams is what has them at Number 10? Oh that makes perfect sense.

I think you misunderstood... I’m speculating that their two losses didn’t hurt them as much as other two loss teams because they were Q1 losses. Obviously something else is at play in some of the other metrics that truly explains that outlier.

scottdude8
01-01-2019, 09:11 PM
NET may not be perfect or always make the most sense, but I think it's clear that it's a massive improvement over RPI so I at least give the NCAA credit for that. RPI lasted far too long though.

My thoughts exactly. NET has some clear flaws, and the fact that the NCAA hasn’t been transparent with how the different “factors” are weighted and calculted is problematic. That said, I think it’s a clear step forward from the archaic RPI, flaws and all.

HereBeforeCoachK
01-01-2019, 10:14 PM
My thoughts exactly. NET has some clear flaws, and the fact that the NCAA hasn’t been transparent with how the different “factors” are weighted and calculted is problematic. That said, I think it’s a clear step forward from the archaic RPI, flaws and all.

An improvement over RPI is a low bar. I still think there are other systems already out there better than NET, but of course, the proof will be in the entire season. ...so it's too early to judge, but count me as still very skeptical.

UrinalCake
01-02-2019, 08:25 AM
Our SOS should be #1 based on the number of top teams we’ve faced... but we have offset it by playing such atrocious cupcake games. If we would schedule top-150 teams instead of top-300 teams then our SOS would better reflect what we’ve faced.

Regardless, we’ll have a ton of opportunities to collect quad 1 wins in conference, so as long as we don’t take too many unexpected losses we should be on track for a #1 seed. Indiana is playing better and Kentucky has turned it around, so those wins plus TT and Auburn make us look really good.

HereBeforeCoachK
01-02-2019, 08:55 AM
Our SOS should be #1 based on the number of top teams we’ve faced... but we have offset it by playing such atrocious cupcake games. If we would schedule top-150 teams instead of top-300 teams then our SOS would better reflect what we’ve faced.

Regardless, we’ll have a ton of opportunities to collect quad 1 wins in conference, so as long as we don’t take too many unexpected losses we should be on track for a #1 seed. Indiana is playing better and Kentucky has turned it around, so those wins plus TT and Auburn make us look really good.

Cupcake Princeton beat Arizona State. Cup cake Hartford beat BC. (low bar, admittedly...) I think for a team of four frosh, the scheduling has been brilliantly balanced. If you want to guarantee that your team faces some 20-22 year old men as a way of growing your team up, you have to go to certain schools to do that.

SCMatt33
01-02-2019, 09:26 AM
but of course, the proof will be in the entire season. ...so it's too early to judge

I think this point doesn’t get enough emphasis. I can guarantee when they designed this rating, they applied it to previous seasons’ data as a test, but I’d also bet they only applied it to year end data, not data from just November and December. With a small sample size of games, this likely leads to some undesired consequences (such as some teams being over-rewarded for crushing terrible competition in the efficiency metric). But since no one gets seeded based on today’s numbers, it doesn’t matter much. I’m actually surprised they released these rankings as early as they did. Even systems that people like better such as KenPom have this issue. Even today, the effect of KenPoms preseason rankings is not fully gone as a way to address this particular issue, and we’re nearly two months into the season.

Indoor66
01-02-2019, 09:33 AM
But I WANT to watch the sausage being made so that I can get angry and complain.

duke2x
01-02-2019, 11:15 AM
Our SOS should be #1 based on the number of top teams we’ve faced... but we have offset it by playing such atrocious cupcake games. If we would schedule top-150 teams instead of top-300 teams then our SOS would better reflect what we’ve faced.

Only Army and Stetson should be dragging our schedule down. Army is a special case, and Stetson might have been the only team we could get on football conference championship Saturday. The rest are in KenPom's top 200. Everything will take care of itself by the end of the year. Of the other top teams in the ACC, the only break we catch is playing NCSU at home only. The rest are either double (UVA, UNC) or away only (FSU, VT).

Troublemaker
01-02-2019, 11:25 AM
So let's get this straight. Q1 is what, about 80 teams? So you think it's Nebraska's two LOSSES to TOP 80 teams is what has them at Number 10? Oh that makes perfect sense.

Let's make sure we don't conflate two different things.

The NET rankings are independent of the quadrant records.

The NET is just a replacement for RPI. Just like how the top 4 RPI teams didn't automatically get the four #1 seeds, neither will the top 4 NET teams. The committee will instead compare resumes. What's your Q1 record, what's your Q2 record, did you take any Q4 losses, etc?

Incidentally, kenpom has Nebraska at #15, not far off from #10. They're a good team.

uh_no
01-02-2019, 12:26 PM
I think this point doesn’t get enough emphasis. I can guarantee when they designed this rating, they applied it to previous seasons’ data as a test, but I’d also bet they only applied it to year end data, not data from just November and December. With a small sample size of games, this likely leads to some undesired consequences (such as some teams being over-rewarded for crushing terrible competition in the efficiency metric). But since no one gets seeded based on today’s numbers, it doesn’t matter much. I’m actually surprised they released these rankings as early as they did. Even systems that people like better such as KenPom have this issue. Even today, the effect of KenPoms preseason rankings is not fully gone as a way to address this particular issue, and we’re nearly two months into the season.

This needs to be driven home. People are accostomed to kenpom, and human polls, and all the other rating systems that take a significant amount of pre-season data into account at this time of the year. NET does not.

So on the first day of the year, hartford beats grambling by 30. Duke beats UK by 30 (never gets old) (assuming similar tempi). Congratulations, hartford and Duke look the same to the system despite anyone under the sun telling you that duke is a lot better than hartford. Then duke plays hartford, beats them by 70 and wow, okay now we have more data to rank those teams relatively. We'll get that for a lot of teams once they start conference play.

I really just ask that people stop complaining about it until closer to march. And stop complaining that the rankings differ from KP or whomever else. It's not supposed to be KP. It's not supposed to be predictive.


On NCSU, which people seem to be hemming about, it's in the right ballpark. KP has them at 21, and NET has them at 8. yeah it's a 2 seed vs a 5 seed, but that'll shake out with conference play, and it's certainly not the issues we saw with RPI when you'd have teams in the 30's and 40's in the top 4 seeds (cough cough st. louis).

WRT nevada, theres a bit of a misunderstanding of SOS. People seem to use it as a be-all-end-all....but it's not...it's more nuanced. Houson has 3 wins against teams in the top 50. Nevada has no games at all against teams in the top 50. Even if nevada's overall SOS is higher, houston has better performances against better teams. the SOS in these rankings is an output of the algorithm, not an input. I imagine nevada will not do well in NET at all because nobody on their schedule is challenging. This makes sense, though. The NCAA has a vested interest in having people play good teams. Nevada didn't, and it seems that the ranking is appropriately penalizing them.

scottdude8
01-02-2019, 12:48 PM
WRT nevada, theres a bit of a misunderstanding of SOS. People seem to use it as a be-all-end-all...but it's not...it's more nuanced. Houson has 3 wins against teams in the top 50. Nevada has no games at all against teams in the top 50. Even if nevada's overall SOS is higher, houston has better performances against better teams. the SOS in these rankings is an output of the algorithm, not an input. I imagine nevada will not do well in NET at all because nobody on their schedule is challenging. This makes sense, though. The NCAA has a vested interest in having people play good teams. Nevada didn't, and it seems that the ranking is appropriately penalizing them.

That's a good point that I didn't recognize... thanks!

Mtn.Devil.91.92.01.10.15
01-02-2019, 12:57 PM
This needs to be driven home. People are accostomed to kenpom, and human polls, and all the other rating systems that take a significant amount of pre-season data into account at this time of the year. NET does not.

So on the first day of the year, hartford beats grambling by 30. Duke beats UK by 30 (never gets old) (assuming similar tempi). Congratulations, hartford and Duke look the same to the system despite anyone under the sun telling you that duke is a lot better than hartford. Then duke plays hartford, beats them by 70 and wow, okay now we have more data to rank those teams relatively. We'll get that for a lot of teams once they start conference play.

I really just ask that people stop complaining about it until closer to march. And stop complaining that the rankings differ from KP or whomever else. It's not supposed to be KP. It's not supposed to be predictive.


On NCSU, which people seem to be hemming about, it's in the right ballpark. KP has them at 21, and NET has them at 8. yeah it's a 2 seed vs a 5 seed, but that'll shake out with conference play, and it's certainly not the issues we saw with RPI when you'd have teams in the 30's and 40's in the top 4 seeds (cough cough st. louis).

WRT nevada, theres a bit of a misunderstanding of SOS. People seem to use it as a be-all-end-all...but it's not...it's more nuanced. Houson has 3 wins against teams in the top 50. Nevada has no games at all against teams in the top 50. Even if nevada's overall SOS is higher, houston has better performances against better teams. the SOS in these rankings is an output of the algorithm, not an input. I imagine nevada will not do well in NET at all because nobody on their schedule is challenging. This makes sense, though. The NCAA has a vested interest in having people play good teams. Nevada didn't, and it seems that the ranking is appropriately penalizing them.

Thanks for spelling this out. People want hot takes and want to criticize the new system before it has had a chance to bear out. Well, that's fine I guess, but not much sport in it.

English
01-02-2019, 01:03 PM
Most of the gripes in this thread and the earlier NET thread have been explicitly discussed in the two episodes of KenPom's pod--the first with the NCAA guys (Gavitt and Worlock), and the subsequent episode where KP discussed some additional thoughts about it:

1. The NCAA guys actually say, in no uncertain terms, that they recognize some issues exist now, but they're very pleased because the NET is an objectively better tool than the RPI, and that's progress. This isn't the end of shaping the NET ratings, but it's an improvement and it's a good start.

2. They considered using other existing ratings tools, but there were a couple of considerations--first, they wanted something they could own and have control; and...

3. KenPom himself speaks to lending his rankings system to the NCAA--he was uncomfortable with the idea that he would be, for all intents and purposes, bound to maintaining it at status quo. He cited something he's considered for some time, as an example of something he wouldn't have the freedom to do if the NCAA used his system, which is to potentially add a Vegas-odds component to his metrics. He suspected the other proprietors of the rankings (e.g., Sagarin) would feel similarly. Anyway, the other rankings systems will still be included on the team sheets for Committee assessing teams relative to the competition.

4. They discussed why they didn't go with a composite of all the accepted rankings systems--it's because when the projection-based rankings and the outcome-based rankings systems are conflated, they lose the nuanced meaning that makes them considered valuable in their own right. KP was actually an advocate of avoiding that, and he said so during the brainstorming meeting the NCAA had with a lot of the stakeholders before really diving into the developing the guts of the NET. There is a possibility of a sort of post-NET meeting with a similar group of stakeholders, to discuss lessons learned and what's working/not working, etc.

5. Gavitt and Worlock do have the NET rankings for historical seasons, back to 2002 (I think?) and have thought about releasing them publicly. They haven't yet because everything is so fresh, and it would almost certainly open another can of worms, but it's not lost on them.

6. They released the NET when they did because, for better or worse, that's when they've historically released the first RPI rankings and they wanted to keep with tradition. That, and because they felt more transparency, rather than less, was the way to go. They knew full well that people were bound to go nuts when the first rankings were a mess. However, even after a few weeks, things have mostly corrected and will continue to do so. PLUS, they figured with the NFL and CFB, any pub was a net-positive, even just to get some attention on basketball over football (which worked, even if it was met with overwhelming condescension).

7. They did mention that some factors are weighed more heavily than others, but that was as specific as they got, sadly. I suspect we'll find out more when data pours in and some enterprising stats wonks can reverse engineer the rankings later in the season. Say, even someone around these parts with some time on his/her hands.

8. KP is pretty overtly disapproving of Google being involved (even if he couches it in humor), although it's ambiguous why. During the process of developing the NET, the NCAA guys said they drove all of the decision-making for what to measure and what they wanted to get out of it (i.e., the Google team didn't make recommendations; rather, they asked questions and the NCAA guys essentially designed the system and the Google team built it to their specifications).

Anywho, I recommend the KenPom pod for real analytics-focused hoops fans. Ken isn't the consummate showman, but it is entertaining if a bit dry. He's still getting used to the format and identifying guests to bring onto the show.

Troublemaker
01-02-2019, 01:17 PM
Plus, as I've discussed somewhere else, you can see all the ranking systems here on Massey's site: https://www.masseyratings.com/cb/compare.htm

You can eyeball it and see that any individual ranking system will have outliers compared to the Massey consensus (average of all the ranking systems). I don't think NET stands out as being particularly outlier-ish.

I'm sure all these individual dorks with their own ranking systems are smart. I'm sure the Google engineers that helped the NCAA are smart.

Life is good. Everything will be okay.

PackMan97
01-02-2019, 01:23 PM
N.C. State is (somehow?) at No. 8.

This is called the "Kevin Keatts is a winner" effect.

It could also be a factor of having a top 10 offensive and defensive efficiency rating. I would say that for the most part all of State's numbers compare very favorably with Dukes. We are neck and neck in scoring, scoring margin, offensive efficiency, defensive efficiency, etc. The only "small" difference is strength of schedule ;) It may be that the NET isn't weighting strength of schedule enough?

In addition, State actually has two decentish wins (Vandy and Penn State) and one really good win over Auburn. We also lost a close one to Wisconsin at their place. State is also ranked #18/19 in the polls so other people think we are pretty good.

Kedsy
01-02-2019, 01:50 PM
This is called the "Kevin Keatts is a winner" effect.

It could also be a factor of having a top 10 offensive and defensive efficiency rating.

I assume you're looking at unadjusted efficiencies, where State is ranked #4 offense and #12 defense (Duke is #7 and #3)?

But in KenPom's adjusted efficiencies, Duke is ranked #2 offense and #3 defense, while NCSU is ranked #16 offense and #49 defense. Of course the difference is mostly attributable (as you point out) to strength of schedule, but I'm not sure how much the unadjusted efficiency numbers are worth in evaluating team strength.

English
01-02-2019, 01:59 PM
I assume you're looking at unadjusted efficiencies, where State is ranked #4 offense and #12 defense (Duke is #7 and #3)?

But in KenPom's adjusted efficiencies, Duke is ranked #2 offense and #3 defense, while NCSU is ranked #16 offense and #49 defense. Of course the difference is mostly attributable (as you point out) to strength of schedule, but I'm not sure how much the unadjusted efficiency numbers are worth in evaluating team strength.

They're unadjusted in the NET rankings, so they may be worth something in this thread*.

* Net efficiency is one of the formula components

Ian
01-02-2019, 06:33 PM
While a huge improvement over the ridiculous RPI, NET does seem to have a major flaw emerging which is the unadjusted efficiency numbers. Which is basically the same as a scoring margin since scoring margin is just unadjusted efficiency margin times number of possessions. By having a scoring margin (even capped at 10), and another criteria which is basically scoring margin again, is just double counting scoring margin, making it the most important factor.

Which unsurprisingly why NET seems to overvalues teams that have a lot of blow out victories over weak schedules. Houston, NCSU are the most prominent examples, Texas Tech is a lesser example.
Conversely, teams who've play tough schedules and hence aren't able to have large scoring margins are over-penalized, examples are Kansas and UNCheats.

They need to count either efficiency margin, or scoring margin, but not both.

camion
01-02-2019, 06:38 PM
While a huge improvement over the ridiculous RPI, NET does seem to have a major flaw emerging which is the unadjusted efficiency numbers. Which is basically the same as a scoring margin since scoring margin is just unadjusted efficiency margin times number of possessions. By having a scoring margin (even capped at 10), and another criteria which is basically scoring margin again, is just double counting scoring margin, making it the most important factor.

Which unsurprisingly why NET seems to overvalues teams that have a lot of blow out victories over weak schedules. Houston, NCSU are the most prominent examples, Texas Tech is a lesser example.
Conversely, teams who've play tough schedules and hence aren't able to have large scoring margins are over-penalized, examples are Kansas and UNCheats.

They need to count either efficiency margin, or scoring margin, but not both.

Flagrant 2. You used "over-penalized" and "UNCheats in the same sentence. :( :eek:

The grammar police are never around when you need them.