Because no matter what kind of counterattack you make, people who already like you will think that he can't be a man, and people who already like him will disapprove of your counterattack.
Do more things to help others at ordinary times, and people will be more inclined to you in your struggle with him.