Let's take an anthropological perspective here, the quotes I use are published in the 1996 edition of Conformity and Conflict.
Firstly:
"The social relationship between men and women has emerged as one of the principal disputes occupying the attention of scholars, and the public in recent years" (Friedl 244), basically, odds are we're not going to settle this here.
"Most anthropologists today believe that, early civilizations ability to recognize a correlation between abuse and (what is known today as) still birth. This gave women a valued role in society, resulting in the modern social stigma regarding the delicacy of women" (Friedl 256)
"Still, other anthropologists point to the presence of abuse in both past societies, and modern Asian societies, and believe that the social stigma, regarding the abuse of woman is aminly a Western belief, in an effort to distance one's self from societies modern-Western societies, believe to be 'barbaric'"(Friedl 264)
The point is, there's many different theories as to why modern society views it as "wrong" to hit a woman, even when the hit you first.