That seems to me to be a great simplification. There are people who don't succeed in gaining attraction - and I wouldn't necessarily deny them inner or outer beauty. What I mean is that someone is definitely being left behind, socially. We have evolved from an originally instinct-driven and primal living being into a calculating and judgemental (devaluing) species. And I'm not saying that there aren't similar phenomena in the animal kingdom.
You're right. If you view it on a social basis, then it will get extremely complex. :)