@cryptogee, OK, I think I better understand what you were saying about the trolley dilemma now. A truly conscious machine would use logic and other human qualities to make decisions, but it could be a lot better than we are at using logic when it is appropriate, as in the case with the 2% and 89% chances of success. Tha's what seems "cold" and is disconcerting to us.

You should do a separate post on anthropomorphism! Maybe it will benefit us when building A.I. In the movie "Ex Machina," I thought this tendency sometimes caused the characters to assume machines had empathy when they didn't, which was scary.

Whether something has to be like us to empathize with us depends, I think, on your philosophy and definition of life. If all life is somehow part of a "universal mind," and therefore connected or literally one, then maybe it's possible for advanced life forms to empathize with each other simply because they share the quality of being alive. Then they wouldn't have to have other things in common.