Then the question is can you be at peace with this?

I think it is a misconception that karma is about balancing out the world's circumstances and making it "fair" in accordance with their actions.

Definition

(in Hinduism and Buddhism) the sum of a person's actions in this and previous states of existence, viewed as deciding their fate in future existences.
informal
good or bad luck, viewed as resulting from one's actions.

Notice it doesn't say anything about who "decides" or whether the results are measured the same way.

This is a somewhat supernatural definition that I don't completely believe, however I believe it comes from a phenomenon that is real. I don't know what to call that phenomenon other than karma.