I think there is a point where assisted driving no longer becomes practical, and either the evolution of autonomous cars has to stop to allow the driver to have a continued say in how the car reacts to its environment or move on to where there is just no place for time-sensitive human input any more. As drivers trust their cars more they will pay attention less (not to mention practice less, an even deeper problem for post-autonomous-car generations than those before), and their ability to meaningfully react to stimulus will be roughly nil.

If we chose the former we are probably arbitrarily limiting our safety in order to save the ego of a human driver. If we chose the latter we will reach a point where we won't have a steering wheel for a driver to potentially become a hazard rather than a help.