The problem is not when a company is suddenly deciding to do something outright evil.
The problem is when the company starts slowly pushing the boundaries in ways that can be justified if you squint right.
E.g. consider this extra tracking. The feature in itself is fine, as long as the data is protected well enough and not abused.
The problem is that staff not used to think about nefarious uses are likely to defend this kind of collection without thinking about all the tiny little privacy implications.
So when a less moral person suggests a little tweak here or a little tweak there and justify it with a seemingly ok reason, it is likely to fly straight past a lot of people.
Another little change here and there, and suddenly things start to fall out of it that wasn't intended.
How many IBM engineers in the 20's and 30's had any inkling that their work on tabulating machines would aid in the Holocaust? Most would presumably have been horrified. Even the IBM execs that negotiated IBMs deals with Hitler would likely have been utterly horrified. Each little step was just one tiny step further from the last, and most had plausible positive spin on them. And the worst was hidden from them.
This is the difficult part of ensuring the morality of corporations, or organizations, or society in general: It is incredibly hard to get a sufficient overview to be able to predict whether any given action will contribute to an immoral outcome. Even when you're right there, "flicking the switch" that finally ties it all together. It's just one more tiny step, that from ground level will often not seem any different than the last step, which turned out ok.
EDIT:
The only way of ensuring the a corporation acts morally, is not to act morally, but to act specifically to take steps that binds the corporation so it is unable to do bad things:
Make sure you don't have data that can be abused, if at all possible, or at least destroy it as soon as possible.
Make sure you have systems that actively prevent staff from accessing data they have no need for.
Consider if you can transform data in ways that makes it less intrusive (e.g. if you don't need the full precision of certain location data, reduce the precision; if you don't need to be able to tie it to a certain individual, anonymise; if you only need aggregate/summary data, aggregate/summarise as early as possible and aggressively purge the raw data).
Ever barrier you remove from access to data, creates opportunities for innocent-sounding requests that turns out to have immoral reasons.
E.g. I've had managers request data from ex-customers that wuld have been in backups except for aggressive policies on purging it, where it turned out that rather than legitimately e.g. wanting to help them recover data, said manager wanted to hand the data to another manager, without realising that said other manager was planning on mining said ex-customers data for sales leads for a new product that would directly compete with them.
Flagrant violation of our contracts, and of UK data protection laws. But the initial request sounded innocent enough, and I only found the reason because I'm extremely paranoid about these things. The manager that made the request didn't suspect a thing, for example.
Actually not having the data made refusing the request a lot easier, and also prevented the manager that wanted the data from trying to find ways around me.
I generally agree with you that not having the data would make it easier to refuse. But not having the data does come at a cost: in this case, poorer user experience.
As a side note, I was mainly talking abt Uber. Your post seems generic and your concerns apply to every large tech company. Most of these companies already have policies in place such as audit trail for data access, access control etc.
I'm not saying that in this specific case it's wrong to collect the data.
I'm saying that to "avoid evil" in the future it is insufficient for current developers to be ethical now.
You need to build systems that actively make it difficult, so that it takes persistence and intent to overcome if you want to do something bad whenever possible. Force people to face that they are putting in effort to bypass restrictions and breaking rules to do what they want to do, instead of "just" accessing data that are readily available to them, and far more will question the justification for requests they are given.
The problem is when the company starts slowly pushing the boundaries in ways that can be justified if you squint right.
E.g. consider this extra tracking. The feature in itself is fine, as long as the data is protected well enough and not abused.
The problem is that staff not used to think about nefarious uses are likely to defend this kind of collection without thinking about all the tiny little privacy implications.
So when a less moral person suggests a little tweak here or a little tweak there and justify it with a seemingly ok reason, it is likely to fly straight past a lot of people.
Another little change here and there, and suddenly things start to fall out of it that wasn't intended.
How many IBM engineers in the 20's and 30's had any inkling that their work on tabulating machines would aid in the Holocaust? Most would presumably have been horrified. Even the IBM execs that negotiated IBMs deals with Hitler would likely have been utterly horrified. Each little step was just one tiny step further from the last, and most had plausible positive spin on them. And the worst was hidden from them.
This is the difficult part of ensuring the morality of corporations, or organizations, or society in general: It is incredibly hard to get a sufficient overview to be able to predict whether any given action will contribute to an immoral outcome. Even when you're right there, "flicking the switch" that finally ties it all together. It's just one more tiny step, that from ground level will often not seem any different than the last step, which turned out ok.
EDIT:
The only way of ensuring the a corporation acts morally, is not to act morally, but to act specifically to take steps that binds the corporation so it is unable to do bad things:
Make sure you don't have data that can be abused, if at all possible, or at least destroy it as soon as possible.
Make sure you have systems that actively prevent staff from accessing data they have no need for.
Consider if you can transform data in ways that makes it less intrusive (e.g. if you don't need the full precision of certain location data, reduce the precision; if you don't need to be able to tie it to a certain individual, anonymise; if you only need aggregate/summary data, aggregate/summarise as early as possible and aggressively purge the raw data).
Ever barrier you remove from access to data, creates opportunities for innocent-sounding requests that turns out to have immoral reasons.
E.g. I've had managers request data from ex-customers that wuld have been in backups except for aggressive policies on purging it, where it turned out that rather than legitimately e.g. wanting to help them recover data, said manager wanted to hand the data to another manager, without realising that said other manager was planning on mining said ex-customers data for sales leads for a new product that would directly compete with them.
Flagrant violation of our contracts, and of UK data protection laws. But the initial request sounded innocent enough, and I only found the reason because I'm extremely paranoid about these things. The manager that made the request didn't suspect a thing, for example.
Actually not having the data made refusing the request a lot easier, and also prevented the manager that wanted the data from trying to find ways around me.