is it just with the window down? if so, try with the window at different heights to identify approx where the noise is coming from.
then take the door panel off and have a gander, might be worth getting some sound deadening stuff to pad the inside of the door out if you can't see the immediate problem.