you don't have to care much if the account reputation is lower than yours and their Steem Power not that much to affect your earning on Steem. But if it otherwise you may start to concern about it and ask the downvoter if they could explain why such action coming from them to you.