I encourage you to try it out. Yes, it might lose some of the magic if you are reading fiction. But sometimes you don't read just for pleasure, you read to learn about some topic that really interest you, you read to seek advises from successful or brilliant people.
Take for example "4-hour Workweek" by Tim Ferriss or "Rich Dad, Poor Dad" by Robert Kyiosaki and see if it benefits you as you listen.