I came across an interesting take on this; it's a social manoeuvre. As in, animals use physical attributions to show emotions; just like dogs wag their tails to show happiness, or a cat hisses and turns "snake like" in times of danger, humans use yawns as a flight or fight signal.
Since yawns are infectious, humans are wired to realise that an adjacent human who does not get "infected" with your yawn is probably a threat. This goes back to the days where homo sapiens were much simpler.
Interesting isn't it?