The issue with GM plants is not that you're increasing the speed at which you can select naturally occurring genes, but that you're moving genes from one species into another, and the potential environmental damage that could occur should the vector for transplanting that gene re-activate and move on to other species. Strains of weeds immune to weed-killers and pests, for example, would not only be annoying to deal with but could also seriously adversely affect the ecosystem.
The controls and safeguards for containing GM crops and monitoring the spread of pollen and seeds have been proven to be insufficient, and GM crops have been grown adjacent to non-gm crops in the past - basically, the industry has consistently been blasé about the need to assuage fears of shit going wrong and the potentially irreparable damage that that would cause, and once the governing body fails to take responsibility activists step up and take action. This sort of thing could be moderated by thorough testing and honest, open reportage, but there's no money in prolonged tests and at the end of the day that's all these companies care about - fuck the potential consequences if there's a buck buck in it!