Haha, if you don't use it much, it gives bacteria etc a chance to grow and spread in the system resulting in the aforementioned smell. If you run it all the time, there is much less chance of this happening.
If it's really stuffy in the car I'll run with the window down for a bit, but once I know the A/C is up to speed I put the window up again.
Each to their own I guess, but I always run with it on so I don't get dodgy smells