I have asked this question of lots of people, I get kind of esoteric answers but in the end they just say, look I can't explain it but it works. And frankly I am too old to believe in that kind of justification.

I agree!

nobody knows how far away they are how can they be a reference point?

It is not necessary to know their exact distance, it is enough to know that they are very, very far away - so far away, that they seem to be fixed and therefore usable in the triangulation. As if they would be in an infinite distance.