Or maybe it's a generalized androphobia (a fear of men).    From Wordnik.com.  [Who's Afraid of the big Bad Elf?] Reference

If you fear men, you are afflicted with androphobia, if you fear women, gynephobia.    From Wordnik.com.  [VERBATIM: The Language Quarterly Vol X No 1] Reference

What about “androphobia”?.    From Wordnik.com.  [The Volokh Conspiracy » Publishing a Misogynistic Student Newspaper Column = Sexual Harassment?] Reference