Yep... this is something to think about when using Electrum.
When you pair a HW wallet with passphrase too electrum, the last page in the "new wallet wizard" is "Encrypt this wallet with the HW key [y/n]". If you pick yes, then the Electrum wallet file is fingerprinted with your passphrase and can serve as an oracle for any future brute force attack. Or to state differently, your Electrum will fail to load if you change your passphrase.
So if your guarding against a wrench attack, take care as to what configuration you use on any SW wallet you pair to your HW wallet.