I honestly think the southern Florida sun is and always has been one of the major factors affecting attendance, possibly the main factor. Unless you have some nice shade, it simply isn't nice to be outside here from about 9AM-4PM for a significant part of the year. **** there are even days in December and January with temps in the mid 80s that aren't that comfortable under the noon sun.
I think back to the Orange Bowl days. Those noon games would almost never come close to being sellouts - only if we happened to be playing FSU, UF or ND. Even a bit further back when the Dolphins were Super Bowl contenders and played in the OB, few of those 1PM games were ever sellouts, while the rare 4PM games were better attended and Monday night games were obviously really big deals. It's the sun. No one wants to sit out in it other than 20 year old college girls working on their tans. It's mostly a southern Florida thing, too. Noon in Ann Arbor, MI in September could be pretty pleasant, yet even for our games in November, there is zero guarantee that a daytime game still won't be hot.
I was very little at the time, but I think my hottest game was in 1984, the game when we got blown out at home by FSU. I seem to remember it raining pretty hard for a while, then the sun came out right after and the steam room effect was just off the chart. One of the worst games I've ever attended, probably top 5 worst.