Yes, height matters. Physical stature is one of the most obvious indicators of strength and health. Our instincts drive us to select strong, healthy mates. This is the same reason a jellyroll like myself will always lose out to a chiseled Bowflex-commercial bod. Muscle-guy isn't really going to be more useful -- I can lift a sofa / install a new kitchen sink / mow the lawn as well as he can -- but that means less than nothing in the face of instinct. A tall man literally stands above a short one, and we're programmed to accord status to that fact.
My preference in women? There are many things more important than height, and all of them are intangibles. All else being equal, I'd prefer a tall, robust Valkyrie type. Ideally she should match my height of 6'2", which I know isn't a realistic expectation.
Do men care about their own height? Ask a short guy. I don't especially care about being taller than average. I suspect most tall guys don't. Would a man date a woman taller than he is? Again, comes down to the individual. I would, but it would be extremely unlikely I'd ever meet one in the first place.
Do women care about their own height? Ask them. I suspect it's much as with men: the tall ones don't care as much as the short ones. What is a woman's preference? Tall, of course. It's no secret that the large, chiseled alpha type beats out every other on the scale of attractiveness.