Brains are weird and very easily influenced/scared.
I talked to someone who voted for the PVV (in the Netherlands) because he feared the country would otherwise get taken over by muslims. When I told him that muslims have had the most miniscule of cultural influence in this country for as far I can remember - no protesting Christian holidays, no demanding laws be more inline with their religious teachings, no banning or buring the Bible - he responded with 'yeah I guess, but still'. And when I told him how they're just being used as a scapegoat and pointed to other moments in history where minorities were used as scapegoats, he said 'I know that, but I still feel this is going to happen'. And this is a guy who is constantly talking about books he read about the West's treatment of people from different cultures.