I think I've down voted maybe 3 posts in 4 years. I tend to down vote blatant spam or people seriously abusing the system. Just let it go. It's their karma not yours. Just keep doing you, most times those people are just looking for a reaction. If you feel something is worthy of a down vote, then by all means it's your right. Don't feel obligated though if it's not in your personality.

Happy New Year!