It seems that to him his friends are more important than you and he is taking you for granted. In this case either discuss with him clearly your feeling about this situation and seek a solution or else you need to deiced whether you want to continue this relation or not.