but the manager is obviously (unless they are very, very bad) using a metaphor (that you don't like). By responding literally, what you're really saying is "I don't really give a f what happens at this stupid company after I LEAVE MY CURRENT RESPONSIBILITIES" regardless of why. It sounds like you have extreme trust issues with your manager if they can't make a (pretty benign) verbal mis-step with you, without this sort of response, and your follow up suggests you're in a really bad space. I can't believe it's not visible and leaking into other aspects of your work and interactions.
I did have a previous workplace, though, where they couldn't stop yapping about the bus factor and I disliked that phrase because it kept reminding me that one day I die and am wasting one more hour of my living days in a pointless retrospective that will have no positive effect on anything.