I've found that most racists, sexist, whatever-phobes don't actually hate people for being people, they hate shitty people who do shitty things that are completely understandable to hate them for.
It's just that a lot of shitty people aren't white men.