Theres a chinese robinhood story that explains why you should be worried, even if there are that many BTC addresses:

A wise chinese man does a favor for the king. The king offers to pay him lavishly for his favor, but instead, the chinese man tells the king to simply pay him 1 grain of rice on day 1, 2 grains of rice on day 2, and then 4, 8, 16, etc. The dumb king laughs and thinks the old man foolish, until 30 days later the king must pay the wise man over 1 trillion grains of rice.
Imagine that bitcoin addresses are the grains of rice. Quantum computers can exhaust through the space of all bitcoin addresses at the same rate the wise man was being paid in rice.
The cryptocurrency community needs to be aware of this vulnerability because many currencies are secured only through problems that would take thousands of years to solve on a traditional computer, but are easily solved by quantum computing. Blockchains need to start implementing quantum-resistant cryptography algorithms in order to truly be secure.