Evil people have a hard time taking advantage of other evil persons. It logically follows that the ones they would take advantage of are the good people. IF you consider yourself a kind person, then you will be taken advantage of. Prepare for it, accept it. Gruber needs for his family to abandon him completely becaus people like him never change. They don't change because just like this time, they believe they will always get away with their inconsiderate behavior. And if you guys around him never say no, he really will keep getting away with it and therefore will never have a reason to change.