I think you should leave it alone and let others have their say.

Many people in South Korea have had plastic surgery, just like everyone knows that South Korea is a country of plastic surgery.

Maybe your face looks as good-looking as if you had plastic surgery. So, plus you have been to South Korea, others will think that you have had plastic surgery in Korea.

Even if others say this, I don’t think you need to mind. Just tell them that they haven’t persecuted people and that’s it. Why should you care about what others say?