Correct, also you need to confirm any activity with the device also. Unless they have your device you're funds will remain in your hardware wallet.
Side note There are more decentralized exchanges which don't have the issue of you needing to trust a centralized domain. Check out Blocknet and Bisq as well :)