They're correct. Someone was in charge of the vehicle at the time, so the accident is someone's 'fault'. Whoever is driving is responsible for the load, regardless of if they loaded it or not (that's a legal thing, same as making sure a car is roadworthy before use etc).
However, depending on how it's worded when your insurer asks if you have had any accidents, claims or convictions, depends on whether you need to tell them. In your case there was no accident whilst driving, you didn't personally claim and certainly no points involved, then I wouldn't feel the need to disclose it. Had you binned it whilst driving, that would clearly be different.
If I wasn't driving when it happens, I'm not coughing up to it. Whoever was needs to take responsibility, however galling that may seem. If they don't, then they're likely to get the boot for gross misconduct.
A grey area, definitely.