You'd be surprised.
On my list last time around were a Clio 182, DC5, E92 M3, GT-R and the 911. Okay, so I ended up buying the most expensive one anyway, but honestly they were all considered.
This time round when the 911 goes I'm torn between an RX8, a 350Z, and an M6. It's looking likely that the RX8 will win, despite being the cheapest and technically most inferior car. But yeah, I do tend to agree with you that most people go up the chain when buying cars, they rarely come back down. I'd love a Caterham as well, but like you say they're just too impractical to do the other weekend things.