Firstly don't allow yourself to be nervous, easier said than done, but being jittery can really screw you over. If you have some method of relaxation or something you do to calm pre-test nerves, i suggest employing it.
Go in fast and hard. Don't hold back or give him the opportunity to build momentum. If you have a longer reach, stay out of his. If you can, try and predict his attacks, a little bruise on your elbow hurts a lot less than a punch to the face. Put as much power into blocks as you would an attack, a lot of people who's only ever seen blocks get their guard broken cause they underestimate the power behind a punch.
Always follow through your attack, punch like you're trying to hit something behind your target. A quick disorienting punch followed by a heavier attack is a lethal combo.
Kicks will only serve to destabilise you. Unless you have the opportunity for a knee, don't bother.
Don't be afraid to hurt, or be hurt. Humans are surprisingly resilient. I don't advocate emerald-morons advice to put him in hospital however, most bullies and smarmy pricks will back down once they realise it will not be an easy fight, unless he's been gearing up to fight as well, i don't really know your circumstances.
In any case, fight only as long as you need to to defend yourself, or if you're the instigator, to get your point across. You'll want the moral victory here, for more reasons than one. The self defence argument falls a little shallow if you're seen kicking a downed opponent, and if you let him leave with the majority of his limbs and dignity intact, he's less likely to retaliate. The best black belt in the world would have trouble with four pissed off opponents twice his size.
In all honesty though, avoid fighting when you can. The bigger man always avoids conflict, so unless you REALLY have no choice, try to solve it with words. Violence is your absolute last resort, though it's the most effective one
