It was probably you when doing the test drive as the natural thing is to make sure you are feeling comfy and so you are more aware of the 'fit'?
I suspect that when you sat in the passenger seat you were either concentrating on the ride etc and/or looking around at the bits and bobs rather than worrying if the seat was OK, as that would not be your normal place in the car