I have to admit i thought it was a bit strange that it got categorized a sport but in the end it doesn't really matter because the world is all about supply and demand.
I remember years ago where Pepsi had an advert that featured the "world's greatest athletes" and a Nascar driver was one of the athletes and I thought that was strange. I think it was Jeff Gordon. Is he talented? Absolutely... is he a world-class athlete. I think not... and the same applies to these e-sports fellas but again.... it doesn't matter.
For me personally, i would much rather watch E-sports than regular sports any day.
There are not many sports I enjoy watching especially without friends.
But when I was really into Starcraft I could watch that all day.
If you look at dart, billard or Sports like this the people dont really look like professional athlets.
I think this discussion will never end and especially for older people it is hard to grasp that something like this even gets the tag sport by some.
At the end of the day it doesnt really matter like you said.