Note that these remarks refer to Ultron as he appears in the comics, since we don't know what he will be like in the film yet.
Ultron has protection from most basic attack forms, and it is conceivable that his adamantium outer coating gives him added protection against things like EMP as well as physical attacks. That being said, a sufficiently powerful one could probably take him down.
BUT-- the problem with Ultron has never been taking him down, the problem with Ultron is that he plans ahead and keeps downloads of his memory and programming in remote safe locations, attached to "Dead Man's Handle" type devices that rebuild his body and install his last backed-up programs and memory in to the new body if he doesn't check in with them periodically. Additionally he keeps improving his own design so each rebuilt body is tougher than the previous one. Each new Ultron then investigates what caused the destruction of the previous one, and incorporates improved defenses against whatever took the older model down in to the schematics for the next one.
Taking him out isn't the problem, keeping him down is the problem.