This is a tough 1, as he might just see you as a good friend and not just as a neighbour. The sensible thing to do would keep your answers short and play on the fact you don't have time to chat for long. try to be nice to them just not engage them so much, i am sure they will understand. I have to be honest I am glad some of my neighbours don't talk to me or should I say i won't talk to them! I wouldn't even put them out if they were on fire, not even with a shovel!!