Some people say it's only a game if it has a failure state but that would exclude most point & click adventures, my favorite genre of games, so that's silly. I think kinetic novels really can't be called games, otherwise you'd have to call reading books on a Kindle "gaming", but visual novels absolutely, they are a part of interactive fiction like those old school text adventure games you mention.

Those are games, just a special kind of games, just like some type of games today are called "arcade games", even when they never were available in an arcade. It's basically another level of categorization put on top of game genres.

In the end though it doesn't really matter. If they aren't all games they are still all entertainment media so they are all related, just like books and movies. Something being a game doesn't make it good and something not being a game doesn't make it bad. In the end, labels like "game" are just meant to make it easier to talk about it, nothing more.