I do not understand why people think this way. What I do know is that it makes easy for abusers to get away with it, because the social pressure is totally on to keep quiet if it's happening. Are people willing to accept that their "speaking up marks the guilty" attitude perpetuates abuse? I bet not. Or maybe this is only the attitude when women speak up. Wifey's supposed to keep her mouth shut; the women are always to blame for men's problems, after all.
I am really not okay with this. But I don't know what it will take to change people's attitudes. Anyway, for me, I think the best response is to just avoid people who make this a part of their approach to life, and to remember to always listen to both people in these situations. This may mean a life in which there are fewer parties to go to but I think on the whole less of these kinds of judgmental people will lead to an overall improvement in the quality of my life.