This is dangerously close to trying to repurpose the term “black box” for irrational fearmongering.
The higher the stakes of the decision, the more incentive to approach it like a rational Bayesian agent who would use whatever tool has the best risk / reward tradeoff, totally regardless of explainability — or if “explainability” (which is not some universal concept, but instead differs hugely from situation to situation) is directly part of the objective, then its importance will be factored into the risk / reward tradeoff without any of this dressed up FUD language, and you might even have to pursue complex auxiliary models to get explainability in the primary models.
For a good example, consider large bureaucratic systems, like a military chain of command connected to a country’s political apparatus — the series of decisions routed through that is way too complex for a human to understand and it’s almost impossible to actually get access to the intermediate data about decision states flowing from A to B in, say, a decision to use a drone to assassinate someone and accidentally killing a civillian.
You could consider various legal frameworks or tax codes the same way.
What does “explainable” mean to these systems? A human can give an account of every decision junction, yet the total system is entirely inscrutable and not understandable, and has been for decades.
Turning this around on ML systems is just disingenuous, because there is no single notion of “explainable” — it’s some arbitrary political standard that applies selectively based on who can argue to be in control of what.
> The higher the stakes of the decision, the more incentive to approach it like a rational Bayesian agent who would use whatever tool has the best risk / reward tradeoff, totally regardless of explainability
These sorts of problems are mostly difficult precisely because we don't know the risk/reward tradeoff.
For example, in the prison system you need to evaluate the risk of recidivism. How exactly do you evaluate the tradeoff between releasing someone who is potentially violent vs. keeping someone potentially reformed behind bars? You'd need to weigh the dollar value of keeping them imprisoned and the personal harm to them and the rest of the prison population when they're imprisoned against the potential harm to anyone else if they're released... plus many other factors that I'm sure I've overlooked.
Anyone who tries to bake those tradeoffs into a "rational Bayesian agent model" will fail.
You're trying to describe the world in terms of game theory, but we live in a world where we usually don't know the risks, or the payoffs, or the rules of the game.
> “These sorts of problems are mostly difficult precisely because we don't know the risk/reward tradeoff.”
If you don’t know this trade-off, then you don’t have a decision problem.
Basically what you’re saying here doesn’t add up at all. Issues where an ML model is used but people want more “explainability” are places where we absolutely do know the risk / reward characteristics, and we’re specifically trying to measure more of the relationship between the internals of the decision process and those components of the risk / reward characteristics (for purposes of political arguments over who controls the process, dressed up as if they are scientific inquiry instead of politics).
If you did not even know what objective you are pursuing at all, you would not be using an ML model, and likely could not even identify what type of decision process you even need yet.
> If you don’t know this trade-off, then you don’t have a decision problem
Let me clarify what I mean.
In the cases I've worked with professionally (finance, autonomous driving), no-one gives you a reward function. You have a rough idea of the outcome you want, and you fiddle around with reward functions until your algirithm kinda-sorta delivers the outcomes you knew you wanted in the first place.
In this sense, the OP gets it precisely backwards. OP wants some sort of hyper-rational Bayesian model that optimises outcomes based on the one true loss function.
There is no correct reward function, and objective functions in general tend to be far too simple to capture the nuance one encounters in the real world.
> If you did not even know what objective you are pursuing at all, you would not be using an ML model...
It's not that you don't know at all, but you should be very clear that your objective is a fudge that probably doesn't capture what you really want. Maths is clean: the real world is messy.
I think the prevailing political system is intentionally set up so that it is “nobody” or a low-level scapegoat. That’s the whole point of the system. Similar with corporate legal structure, corporate oversight and the way executive actors can avoid personal liability.
My overall point is that “explainability” is inherently subjective & situation-specific and whether a decision process “is explainable” has virtually nothing to do with the concept substrate it is made out of (e.g. “machine learning models” or “military chain of command” or “company policy” or “legal precedent” and so on...).
It’s about who successfully argues for control, nothing more.
The one who ordered the drone strike is responsible (but hardly held responsible). Easy to draw the parallel with ordering the deployment of anti-personel landmines: the one who ordered deployment is responsible (but may not have signed the treaty, and thus, is hardly held responsible).
Autonomy or explainability is often a red herring. Look at who gives the orders. It is unlikely to ever be the programmer, even if they made a grave mistake. We have a history for that with smart rocket systems.
The higher the stakes of the decision, the more incentive to approach it like a rational Bayesian agent who would use whatever tool has the best risk / reward tradeoff, totally regardless of explainability — or if “explainability” (which is not some universal concept, but instead differs hugely from situation to situation) is directly part of the objective, then its importance will be factored into the risk / reward tradeoff without any of this dressed up FUD language, and you might even have to pursue complex auxiliary models to get explainability in the primary models.
For a good example, consider large bureaucratic systems, like a military chain of command connected to a country’s political apparatus — the series of decisions routed through that is way too complex for a human to understand and it’s almost impossible to actually get access to the intermediate data about decision states flowing from A to B in, say, a decision to use a drone to assassinate someone and accidentally killing a civillian.
You could consider various legal frameworks or tax codes the same way.
What does “explainable” mean to these systems? A human can give an account of every decision junction, yet the total system is entirely inscrutable and not understandable, and has been for decades.
Turning this around on ML systems is just disingenuous, because there is no single notion of “explainable” — it’s some arbitrary political standard that applies selectively based on who can argue to be in control of what.