It could be worse. You should see the University of Texas's out of conference schedule since 2000:
2000: Louisiana Lafayette, Stanford, Houston
2001: N Mexico State, UNC, Houston
2002: North Texas, UNC, Houston, Tulane
2003: N Mexico State, Arkansas, Rice, Tulane
2004: North Texas, Arkansas, Rice
2005: Louisiana Lafayette, Ohio State, Rice
2006: North Texas, Ohio State, Rice, Sam Houston State
2007: Arkansas State, TCU, UCF, Rice
2008: FAU, Texas El Paso, Rice, Arkansas
2009: Louisiana-Monroe, Wyoming, Texas El Paso, UCF
2010: Rice, Wyoming, UCLA, FAU
2011: Rice, BYU, UCLA
2012: Wyoming, New Mexico, Mississippi
2013: New Mexico State, BYU, Mississippi
2014: North Texas, BYU, UCLA
2015: Notre Dame, Rice, California
2016: Notre Dame, Texas El-Paso, California
2017: Maryland, San Jose State, USC
2018: Marlyand, Tulsa, USC
2019: Louisiana Tech, LSU, Rice
Only twice between 2000 and 2014 did Texas schedule a top-tier out of conference opponent.