That's not how vision works. You see an extremely post processed image that's extremely far away from the original light that hit your retinas. There's nothing at all privileged about shifting something into the visible spectrum directly and seeing junk. You're just making an image that your visual system isn't good at understanding. It's not pure, it's garbage. You would hallucinate things that aren't there, you would miss obvious things that are there, etc. For you to really comprehend something the transformation needs to be designed.