The environment* people lived in changed and the use of constant derogatory language and actions in the media/Farage etc scapegoating foreigners set up that process up, enabling the xenophobes free reign. And it wasn't a handful as you had the Daily Mail, Telegraph, Daily Express, Radio media etc and some of the leading advocates of the Leave Campaign regularly using derogatory language about EU workers over a period of many years. Also, I'm wasn't even talking about racists being emboldened, I'm talking xenophobia but it did also give those racists a similar free reign.
And Farage hasn't been misquoted, for an example the Poster about Turks swamping Britain or did somebody kidnap him and drill his feet to the ground in front of the poster?! This wasn't a one off as you well know.
*It matched up with Theresa May's idea of a "hostile environment".