You are being unrealistically optimistic. Maybe you don't live among enough immigrants. You can always tell who is an immigrant that came at the age 15-20+
Learning a foreign language at the age of 40+ becomes a challenge. I see it with many people who emigrated as adults. Different culture, different music, different jokes, different customs. Blending in is not easy, especially if the number of foreigners isn't too high. And then local people know you by "the American", "the black", "the Korean", "the Arab", "the Jew" etc etc.
No, it's not something I imagine, it's something I have seen.