Yes, I agree.

I just got a reply to one of my postings from @fiveboringgames threatening to flag me for encouraging people to 'comment spam' on my post, as he sees it. He then says downvoting is no big deal. Does he realize that this can affect reputation score?

I think downvoting someone in this manner (on their own post) is extremely rude and antagonistic, and I can see things going badly wrong, as others have already suggested tit-for-tat flaggings.

I find this attitude totally unfair.