Is it racist to tell someone you can't understand their foreign accent?

Apparently it could be, but does that hide a bigger issue?