I'd say stay the whole weekend! Sun morning traffic on 95 is more civilized than Sat evening.
Of course, I'm not the best one to ask: I typically drive down and back for mid-week games, leaving DC by about 2:15 (to catch HOV), have dinner in Durham if it's a 9:00 game, and drive home after the game. The adrenaline usually gets me beyond 85.
-jk