The enderman should be as follows:
* Current "silent and peaceful till you look at him" behavior.
* When you do, he lets out a blood-curdling strangled screech of some sort.
* At this point, you have a random amount of time, up to about two seconds, to look away - so that he's not in your field of vision any more - before he attacks. If you do, he will not attack - but he will remain active for a random amount of time after. If you look at him while he's active, he will attack.
* He should deal about half your health bar in damage per hit.
And voila! Instant genuinely scary enemy!