Not really a fan either, but for some reason lots of the anime series I watch seem to become popular with yaoi fans, so I come across it often. And the thing that I catch most often that people seem to like isn't overt sexual interaction -- it looks to me like a big part of yaoi's draw is to see the undertones and the subtlety of the male relationships.
I do know some people who like yaoi and they don't like sex scenes -- they just want to see well-drawn, beloved male characters as if they were getting a glimpse into that man's most private life. Men are rarely emotionally intimate with each other in our society in front of girls -- we girls don't get a chance to see men expressing close emotional feelings to other men very often. So maybe by being able to see favorite media characters doing this, it makes girls feel closer to them?
I do know some people who like yaoi and they don't like sex scenes -- they just want to see well-drawn, beloved male characters as if they were getting a glimpse into that man's most private life. Men are rarely emotionally intimate with each other in our society in front of girls -- we girls don't get a chance to see men expressing close emotional feelings to other men very often. So maybe by being able to see favorite media characters doing this, it makes girls feel closer to them?