Are we just not using the same screen protectors? I don't ever notice my screen protector and I've been using them for years. The only annoyance I can remember with those is applying the plastic flexible ones, but the rigid glass ones are very easy to apply.
I'm curious as to what you dislike so much about them.
I think there's an extremely wide sensitivity in range of eyes, I honestly cannot imagine how in the world a screen protector couldn't significantly affect color and brightness that comes from the display naturally (your comment is the first time I learned this isn't the common case, I swear)
For me it's feeling the ridge at the edges of the protector, as a do a swipes from the edge of the phone. For things like deleting mails, going back, or especially for usng the alt-tab like feature that my iphone 8 plus has.
I'm curious as to what you dislike so much about them.