maybe hes jealous. that you get to spend time with his friends without him, and vice versa.
explain to him that you do understand he might want to see his mates on his own, but that it would be great if you could all hang out together as you get on so well.
maybe hes happy hanging out with your mates as maybe theres more girls? but he doesnt feel happy with you hanging out with a bunch of guys, even if hes there?
just make sure that you're not interfering too much, but it doesnt sound like you are.
if he makes such a big fuss over this, hes not worth it in the long run, trust me.
you cant go on if he doesnt change, as this is a big issue and will make you miserable, and feel like he has something to hide.