I don't like infants, babies, and toddlers. Once kids hit about 4 years old (varies by development, all kids are different on the maturation scales), when they can communicate and understand that other people exist, they get loads more tolerable.
I think a large part of it for me, is my current life. I'm just starting my career, and basically exist just a little bit above the ramen level for food and lodging. I think if I were more comfortable with my life, I'd be more tolerant of babies. Because I wouldn't be smoking hand-rolled super-stress cigars every two hours that I'm awake.
But enough about me, let's talk about you, OP. Do NOT be ashamed of who you are. Everyone has their own opinions and preferences, and yours are no less valid than anyone else's. Decide how you feel and stand your ground. The people who say stupid shit to you like 'You'll come around,' or whatever, they're trying to belittle and other you. Don't get sucked into their trap. Let their hate roll off you like water off a duck's back. Do not respond. Don't give them control of your self-esteem.
And I know it's corny, but you're not alone. There are loads of us not interested in reproducing. My recommendation is this: be awesome, whether you have reproduced or not. If you are awesome, other people's kids will want to be like you, and success is the best revenge.