You are always the only one because you are the only one who ever has the mixture of feelings about the subject in the exact way that you do.
Anyways, the point behind the expression, "No you are never the only one," is to get people away from the useless whining that accompanies the, "Am I...