I never use it, ever.
Then, when I do, its often by mistake and makes a horrid noise being dragged across the dusty, dry window.
Also, I think it ruins the side profile of the rear of the car, or rather, like you say, I prefer it without.
Takes about half an hour to remove and is really simple to do.