The seventh-seeded Wolf Pack (28-7) advances in Nashville into a Sunday matchup with second-seeded Cincinnati, which beat Georgia State 68-53 earlier in the day.
A seventh loss in their last 11 games ended the season for the Longhorns (19-15), who led by as many as four in the extra session.
Caleb Martin's biggest hoop came at the 2:29 mark of overtime, giving Nevada its first lead since 5-2 in the early going. The 3-pointer put Nevada up 78-77.
Martin then buried another 3-pointer on the Wolf Pack's following possession, extending the lead to four and putting Nevada in a position to ice the win from the free-throw line.
Martin had 18 points, 10 rebounds and five assists for Nevada, which had been eliminated by Iowa State in its NCAA Tournament opener last season.
Cody Martin added 15 points and six assists, while senior guard Kendall Stephens led the team with 22 points and five 3-pointers.
Sophomore guard Josh Hall (15 points, six rebounds) and junior guard Jordan Caroline (14) also scored in double figures for Nevada, which had lost its previous game to San Diego State in the Mountain West Conference Tournament semifinals.
Junior guard Kerwin Roach II had 26 points and freshman guard Matt Coleman added 25 for Texas, which led 68-65 in the final minute of regulation before being forced to overtime.
Roach (six) and Coleman (four) combined for 10 of Texas' 11 3-pointers.
Six-foot-11 freshman Mohamed Bamba recorded a 13-point, 14-rebound double-double before fouling out for the Longhorns, while junior forward Dylan Osetkowski had 11 rebounds to complement seven points.
After rallying from a 14-point deficit in the second half, Nevada had two chances to win in the final three seconds of regulation.
With his team down 68-67, Caroline made just the second of two free throws with 3.0 seconds remaining, tying the score.
Then, after the Longhorns lost the ball out of bounds with 1.6 seconds remaining, Caleb Martin airballed a contested 3-pointer just before the horn sounded, sending the game to overtime.
Facing a ranked team for the 13th time this season, Texas used 10-, nine- and seven-point runs to build a 35-26 halftime advantage.
The lead reached 14 early in the second half, and a layup by Osetkowski in the fourth minute of the period had the Longhorns up 45-32 before Nevada rallied.
The loss was the Longhorns' school-record eighth overtime game of the season, with Texas going 3-5.
--Field Level Media
Texas | Nevada | |
Kerwin Roach II 26 | Scoring | Kendall Stephens 22 |
Matt Coleman 4 | Assists | Cody Martin 6 |
Mohamed Bamba 14 | Rebounds | Caleb Martin 10 |
Matt Coleman 3 | Free Throws Made | Jordan Caroline 3 |
Dylan Osetkowski 2 | Steals | Josh Hall 1 |
Mohamed Bamba 3 | Blocks | Cody Martin 4 |