You'll then get more warnings if you want to give the sideloaded app additional permissions. And if they want to make the sideloading warnings more dire, that wouldn't be nearly as unreasonable.
Pins can still be phished. Just make the phishing a live proxy resembling the real site.
A fundamental difference with e.g. FIDO2 (especially hardware-backed) is that the private credentials are keyed to the relying party ID, so it's not possible for a phising site to intercept the challenge-response.
The sideloading warning is much much milder, something like "are you sure you want to install this?".