Is striking a small built man the same thing as striking a woman?

Males are constantly told growing up that you don't ever lay a hand on a woman. What about a small petite man? A lot of these midgets seem to have a napoleon complex and often times I find my self in situations where I have to curb stomp these leprechauns. It would be no problem if they could get their egos in check. But I have been wondering if defending myself against these frail males is the same thing as attacking a helpless woman. Since they both lack the stature and upper body strength to do any real damage.