> I thought this required me to unlock my phone and say 'I trust this computer'
It does, but only once, and then it's almost impossible to "un-trust" the computer without wiping your phone. Besides, you probably have already "trusted" your home computer -- which could be exploited between then and now and then used as a vehicle for attack.
And, as some commenters have mentioned, when you plug into a new device, use an airplane USB charging port for example, the phone may repeatedly ask you to accept a pairing with the other side (the USB charger/device), and an accidental tap on the wrong button can leave the door open.
A Mac can be booted in single user mode if one has physical access That seems like a more serious issue.