That is definitely a problem. YOU should be in control of the situation and decide who is allowed to approach you, not him.