i have not been upvoting comments on my post and any other post maybe because It doesn't really make a difference

I do believe that it makes a difference. First, it makes a difference for the commenter that probably would comment again the next time (since he got rewarded for it) 😉
Second (as a consequence from point 1), it also makes a difference for you (the voter) since the same user might come back and read and comment your stuff again.
So there it is: the beautiful communicative circle 😊