Proving your point -- in 2022, per the SRS:
Duke was the 32nd best team, finished 9-4, playing the 77th toughest schedule.
Oklahoma was the 26th best team, finished 6-7, playing the 10th toughest schedule.
From the period 1970-present, using the SOS at sports-reference, it looks like 1984 Duke (2-9) and 1996 Duke (0-11) faced the toughest relative slates -- both at 18th-toughest schedule in country those years.