I don't find it rude, it's entirely situational. It would stand to reason that it would then be polite to reply to a reply, and so forth until an infinite loop is formed and no one wants to rudely end the comments. It's like refusing to end a phone conversation graciously and people just stay on the line...
Some comments engage more, pose a question, give some humor deserving of reply, etc and warrant a response. But at some point the conversation comes to a natural end, and that can be as soon as the first comment!
I firmly believe that authors more active in their replies build a better follower base though.