In this context, something can only be "gay" if it's being done with gay intent. So, if you're not gay, nothing you can do, up to and including having sex with a man*, is actually gay.
*It would raise the question of why, exactly, you were having sex with him, but the definition stands.
*It would raise the question of why, exactly, you were having sex with him, but the definition stands.