I think there are several reasons people have these type of negative responses to immigrants and different cultures.

The first is that there just seems to be some naturally occurring diversity in thought/personality. Most of these people who are anti immigration are fearful, they cover their fear with anger but underneath they are desparate for a world that is static and unchanging, where the life-raft of monotony can be grasped. They are threatened by change of all kinds and hark back to 'the good old days' even though they may not be that old.

Secondly I see that immigration is very good for a nation, and in fact most economists and governments (quietly) agree. I lived and worked in London for a long time, we have a lot of highly skilled, talented, innovative and entrepreneurial immigrants who add massive value to the economy. These people could basically live and work anywhere, attracting them is critical to national success. Compare that experience to someone in a small town, where work is already hard to come by, seeing immigrants come and get houses and benefits. The issue is that the benefits of immigration are not shared by everyone in society, some people get the positive outcomes, some do not. This is the fault of government, they need to help people in need, and they need to tell people the truth.