Have you pre-allocated the virtual disk? If you haven't, it will keep asking for extensions to the underlying OS and slow things down. When you create the virtual disk for the VM, make sure its all preallocated.
Good point. I will have to change the disk so its all pre-allocated. Still does not explain why there was such a difference between nativly on slower CPUs than on the larger server