If he agrees to come then you should consider that he also feels that this is a problem, but if he does not agree then it is not a problem for him. In that case you need to find out if you wish to continue this relationship in the same given scenario. As the chances of him changing are very low as the friendship that you consider as problem that needs a solution, is not vital or problem for him.