I honestly think he can pull it off, and the hate stems from his being typecast as roles that are more like him. His work in The Town was stellar and very off-character for his normal roles. He can do well when not typecast nor given a horrible script (I still contend he was a good Daredevil, the movie was just crap).
Also people need to stop judging people because they don't "fit the look". Thats all subjective bullshit. Heath Ledger was a pretty-boy, which didn't fit with the "look" of the Joker, yet he definitely looked the part anyway and defied expectations so much that it is damn hard to top what he did with the role.
Affleck can be a good Batman, and having a bit of objectivity can't hurt. But I'm a minority when it comes to these things, and I'm actually more worried he would be panned because the movie itself doesn't pass muster. Blamed again for ruining the movie when it was more than likely the movie itself that was bad.
BTW, Kevin Conroy has been the default voice of Batman and everyone expects anyone in the role to sound like him, which is so unfair its borderline insane. Batman is a comic book first, and it doesn't have voice acting, so essentially no one knows what Bats actually sounds like...
Ugh... negativity, the internet nerd's default setting. It just proves my point that so many gamers and nerds of any walk will never be satisfied with anything because of preconceived notions and an overblown sense of holier-than-thou.
Give the dude a chance, and if he doesn't work out, feel free to glad-hand and slap yourselves on the back of being right. But honestly the negativity that pervades this community is sad in that it seems most are incapable of enjoying anything or being any form of objective.