Could be moisture in the AC system that condenses during use and sits there once it's switched off.
Like you say, switching it on clears the moisture but as soon as you turn car off and restart but with AC off you can smell that damp musty smell.
Option 1
Keep AC on permanently.
Option 2
If you use your AC, 5 mins before you end your journey turn AC off and heater on high. This dries off any moisture that has condensed in the system so that next day when you start car with AC off NO SMELL
You should run the AC on at least once every week or so even in winter or the pipes dry out, crack and leak resulting in no AC come summer time.
In winter turning heater on with AC acts like a dehumidifier. Don't have to run the AC on cold during winter!