Ignoring for the minute that Clarkson is a very good journo who knows which buttons to push for maximum exposure, he's probably correct on this one. Thinking it through:
Stiffer springs and shocks affect the ride quality for the worse
Lowering the car means more scrapes on the front bumper
Larger wheels add weight at the worst place and give a crashy ride
Better brakes are more likely to squeal and are not so good from cold
Exhaust and air filter are going to make the car noisier
Better stereo MAKES YOUR EARS BLEED WITH BASS
...and so on.
Technically from a generic POV 99% of mods make a car 'worse' as a daily driver, and it's only because we're enthusiasts who do it with a specific end goal in mind that we can put up with the compromises: To us, a bone stock car is the compromise, hence the need to modify.