I can see what you mean in this picture below of a 350Z Milltek exhaust.
If you can get the whole inserts out then that may be your best option, ...not sure how it will look afterwards but at least it won't be buzzing anymore. If you have problems though a custom exhaust place still might be your best bet to sort it.