In short, SegWit can flip the incentives for some (bandwidth restricted) miners such that they benefit from not downloading or verifying the transaction signatures. For a user, this causes a SegWit transaction to be less secure than a non-segwit one. A non-SegWit transaction is secure with the assumption that no attack is able to obtain more than 51% of the network hashrate. With SegWit, this becomes 51% - (percent of miners with flipped incentives).

If 20% of miners end up in a situation where they are not downloading signatures because it is more profitable for them not to, a 31% miner can attack the network effectively. Security becomes 51% - X.

The bigger problem is that it is very hard to estimate the proportion of miners with this flip in incentives before their portion grows too large and an attack occurs. This basically means that SegWit transactions will always be less secure than normal Bitcoin P2PKH transactions. Even if just by the smallest bit, this is what fundamentally breaks the fungibility of Bitcoin.

https://steemit.com/bitcoin/@kyle.anderson/fork-christ-s-sake-bitcoin-is-going-to-fork-is-segwit-bitcoin-whose-side-should-i-be-on