Though shouldn't be that hard to know if you are being spoofed. You probably have a decent idea of how much the IRS drifts and any large deviations from that or unexpected jumps in GPS should be noted and possibly, maybe manually, rolled-back so that the IRS only considers data before that point.
I understand that current civilian aircraft wasn't designed with that in mind though.
Avionics were designed with the opposite assumption. When a high accuracy location source provides a location (GNSS or DME/DME) the position is updated and it's assumed that could mean a significant jump for the INS.
Practically that was what happened all the time before GPS. You would fly for a few hours over the ocean with no ground based reference, having a reasonable but not perfect INS location. Then get close to the coast where a DME/DME fix was done which updated the INS position as a big jump of up to a few miles.
Filtering GPS updates that are too far "off" the INS state would be an almost opposite design to the original assumptions that DME and GNSS are highly accurate.
I understand that current civilian aircraft wasn't designed with that in mind though.