Hmm, maybe try to feed him. Also after you can feed him, try to feed him from your hands. Also you can work on training with him. It will make your bond with him grow stronger. That is about all the advice I can give. I hope others can help you out a little more.
Good luck, and please keep us updated. Also welcome to this forum, it is a wonderful place for advice and I have made some great friends here as well!
Welcome to the forum
Gracie's mom gave you great advice. If you were to participate - if not take over - his primary care, like feeding, walking and training it will help establish your bond with him. Where did you get him from? If from a breeder then you should definitely contact them with this problem and seek advice. Is it just you or men in general that he growls with? Do you know what kind of environment he was in before you brought him home?
Nip the problem in the bud is my advice. Let him know he is the lowest in the pack and make sure you establish yourself as the alpha. The advice given by this guy is pretty spot on, with regards to my own experience with dogs. Simple stuff to establish yourself, feed him AFTER he has seen you guys have dinner, don't let him onto your couch/bed (especially while you're there) and dictating when and where he should be, such as being crated while you're having dinner.
Deciding when he eats is also important. Feed him after you're done eating. Teach him to sit, then make him sit before he eats. I teach my dog to only approach his food when given the command "Eat", my trainer also does this to his 5 dogs. If he does not finish and walks off, the food is forfeit and he goes hungry until the next meal. Don't confuse this with taking the food away intermittently to teach the dog who is boss, that is a wrong method IMHO.