I never use it, ever.
Then, when I do, its often by mistake and makes a horrid noise being dragged across the dusty, dry window.
Also, I think it ruins the side profile of the rear of the car, or rather, like you say, I prefer it without.
Takes about half an hour to remove and is really simple to do.
I share the same view as GM on this one suits.. It really doesn't bother me enough to remove it, plus putting a rubber bung there won't look too great!!!
I'll consider your proposal mr suits!!! Hahaha