I imagine, once automated cars are safe and popular enough, they will mandate that all new cars must have it and set a date after which only automated cars are allowed on the road. It will become an offence to take over and drive manually unless the circumstance demands it. You'll be fine driving your Aston Martin, but your kids, or their kids might not be.
I expect that as the automated cars get sophisticated enough that they no longer act individually, but rather each car is in constant communication with the cars in the area so that they can coordinate maneuvers and engage in global planning, a semi-manual option might be added to the system so people can drive for fun.
They way that would work is that when you car is in semi-manual mode, it tells the others cars that a human is controlling steering and speed. The other cars will then give the human extra room (e.g., no drafting or close formation driving on the freeway, no crossing at right angles in an intersection at full speed just inches apart, or that kind of stuff that the cars will do when computers are driving all of the cars in an area).
The automated system will still be monitoring the human and can take over if the human does something bad, so the other drivers don't have to worry that a human driven car will suddenly cross the line into their lane and kill them.
This could even be turned into a game, where the automated system can score the human driver based on how well the human drove and how often the system had to intervene to keep the human from crashing.
Note that racing could also be allowed in such a system. Want to take that Aston Martin out for a race against your neighbor's Ferrari? Let the system know, and it can give you a course through the city, clear a bubble around you two, and let you go at it.
Most of these things will not be possible merely because they give you an unfair advantage. What you call "racing" I call "being at my destination much sooner", and, yes please, I would like every other car to step aside so I can get home faster. Which, of course, only means that everybody will try to abuse this.
Make "race mode" cost money then. Cheap enough that an occasional race is not ridiculous, but expensive enough that it is not practical for a commute.
It probably wouldn't be an incredible advantage anyway. Your car could drive much faster if it knew exactly what every car around it was planning on doing.
At least in urban areas, you're eventually going to end up having vehicle performance arbitrated via radio by a computer network outside the vehicles. (I'm pretty sure it's much easier than trying to do everything "peer to peer," if you will) This gets you intersections where nobody has to fully stop and vehicles merely coordinate and modulate their speed, which is a pretty powerful efficiency advantage in itself.
With a system like that you could easily design a market where vehicles can choose to pay a toll to get a favorable path through traffic. You might see a few options and incentives given for choosing a more fuel-efficient path, etc.
Honestly, I'd expect semi-fun mode to be implemented to prevent human drivers from passing, thus killing the "fun" and reinforcing the "driving is tedious" meme in order to serve a greater safety goal.
I don't buy it. In that scenario, how does anybody learn to drive in the first place? Someone who hasn't spent years of their life driving manually won't have developed the skills to intervene properly in an emergency. At best, driving would, become a specialty career/hobby, like piloting aircraft.
By the time all personal transportation is automatic, it won't just be the status quo with "robot chauffeurs". The whole transportation ecosystem will have to change substantially. Self-driving cars are just a first step to get people used to (or even enamored with) the idea of automatic transportation on a large scale.
I love driving. I find it incredibly enjoyable. I don't think automatically driven cars will change that because I'd happily give up 80 - 90% of my real world driving to a computer. Driving on a congested motorway (highway) or in a busy town just isn't fun, no matter how nice the car you are driving.
I think horse riding might be the best example of the way things will go if and when automatically driven cars become a reality. Once a life skill required by everyone, now a hobby in all but the most extreme of communities.
We're already heading down that route [1]. Although I can't see it ever being an offense to manually drive your car, the car will just automatically take over the driving duties if it doesn't like the way you are driving.
You're not factoring in the Media, which will raise a big fuss about taking away their rights and robots not being safe... you get the idea. I can't imagine how people will react once there's an actual possibility a car could be driving on the road with no human controlling it.
