I feel the root cause here is the culture of setting hard deadlines by management.
Getting a deadline moved even for logical reasons involves pushing against massive internal red tape. Missing a deadline invites greater punishment from management than compromising on quality. Naturally, quality loses out.
Happens all the time in our own industry, and sadly, this case shows that it happens even in safety critical industries.
All engineers, regardless of industry, should be trained in how to negotiate with management. All managers, regardless of industry, should be trained to respect opinions of subordinates.
> […] managers […] should be trained to respect opinions of subordinates.
That's your problem right there: the subordinacy of engineers.
We are biased to listen to high-status people more than low-status people. The strength of this bias varies from people to people, and from culture to culture. But if you are asking someone to listen to the opinion of a direct subordinate, you are asking for trouble.
This is one reason why Michael O. Church is so big on guilds, or "professions". If instead of being a subordinate, the engineer was a (possibly certified) master craftsman, things would be different. Just picture the manager calling the the engineer "master", then not listen to her opinion.
> […] engineers […] should be trained in how to negotiate with management
Conversely, it's harder to negotiate when you have lower status than the manager. You may feel it is not your place. But if you're a master programmer, offering your services to said manager, then you could feel more confident about talking to him as a peer. You could feel more confident about saying things like "your idea won't work —trust me, it's my speciality."
I agree with you. I'd like to live and work in such a world myself.
But my rather pessimistic opinion is that status, hierarchy, etc are deeply embedded in us as a species - just like animal species who have pack leaders or dominant males or whatever - and this won't be easy to get rid of. The ones who hold financial strings, the ones who act more arrogant, more confident, more powerful, more knowledgeable - they are unconsciously seen as having higher status.
The hierarchy could be inverted with a very small change to the laws governing engineers, just as it has been done for lawyers. The details vary by jurisdiction, but generally lawyers must practice in some form of partnership where non-lawyer members are restricted. Work similar to project management is frequently performed by secretaries that are subordinate to the lawyers.
Whether or not strengthening the profession of engineering would be good for society is another question. It is inherently anti-competitive. It could easily slip into a system with very few and highly paid "qualified" engineers whose small number would hinder development. This is a problem with medicine and the supply of doctors in some places.
I would note that in the legal field, effective DOJ threats have prevented any attempt to limit the supply of JD grads, while the other rules such as management of lawyers by non-lawyers has remained in place.
The problem is not hierarchies as such, but in failing to recognize the hierarchy as an expert organization where the experts are the masters of their expertise. Managers are project managers who have the responsibility to direct their experts to solve the issues to make the project succeed. It's therefore the managers job to have a best overall understanding of the project, and incidentally it's the job of the experts to inform the manager of any serious issues they cannot solve alone, and possible solutions for the problem if possible.
Expert organisations are not hypothetical, and actually have been present in all hitech companies I've worked for. Then again it could be a cultural difference, as I'm working in Finland where social equality is a big thing, and calling your manager, director or CEO with honorifics like "mister" would be considered awkward or almost inappropriate.
This would change almost nothing in the current scenario. Companies still have in-house legal teams, who are trained lawyers and I'd argue they are still 'subordinate' to whoever their managers are. Given the number of startups that regularly skirt the edges of the law it's obvious that many founders don't pay heed to legal advice either (or even seek it out) [1]. Why should it be any different if engineering was more 'professionalised'?
[1] I'm not making any judgements about this, simply stating that it happens.
> This would change almost nothing in the current scenario. Companies still have in-house legal teams, who are trained lawyers and I'd argue they are still 'subordinate' to whoever their managers are.
I have no idea what your scare quotes mean, so I'll pretend I didn't see them.
What you describe may be yet another problem of subordinacy, but there is a difference: when you get legal advice, it's not binding. You may chose not to follow it. But only engineers can build a bridge. That gives them a limited veto power.
In the current scenario, there are several steps. First, the lawyers says how much the different kind of litigations cost. Second, the boss (or some other manager) works out which costs more: letting people die, or fixing the lethal stuff. Finally, there's the engineer, which either fixes the damn thing, or does not.
This suggests at least two angles of attack. First we could increase the cost of death for companies. Second, we could held specific people personally accountable. Either the manager for trading lives for money, or the engineer for implementing that trade off.
Now, as another commenter warned about, we should be careful about not freezing innovation in the process. At this point, the problem is so hairy I have no idea what's best.
The quote marks around 'subordinate', to me, are intended to indicate that the status that the relationship entails on the surface does not hold where it matters. The lawyers are not really subordinate to the managers and are able to extract more value from management than the managers are able to extract from the lawyers. If this dynamic persists long enough, then, like the Janissaries of Ottoman fame, the subordinates eventually find themselves the masters.
That almost sounds like you're absolving the engineers (as though they can do no wrong). Changing a part, without also changing it's part-number, doesn't sound like a normal thing to do (and then forget about).
This is a complex issue and I do not agree with the (over-simplistic) reduction to 'management' and 'deadlines'.
Not absolving the engineers. The engineer in this case - and other cases - may well be guilty - both legally and morally - of mistakes.
What I'm trying to understand is, what would make an experienced senior engineer prefer such shortcuts over doing the right thing.
The article says, "Faced with a deadline, DeGiorgio replied: If increasing the torque will destroy the switch then do nothing. Maintain present course. Under no circumstances do we want to compromise the electrical performance of the switch.".
I interpret it as the deadline being a root cause. At the heart of any complex issue lies one or more root cause(s). What were they here? What made him behave that way? Fudging the records is a consequence, not a root cause.
>What I'm trying to understand is, what would make an experienced senior engineer prefer such shortcuts over doing the right thing.
Organizational failure. An onerous documentation change/approval management scheme. Covering up a mistake. A culture that causes people to fear losing their jobs, especially for cost reasons, or the feeling that company management are looking for reasons to cut staff. Having your group be blamed for an expensive product recall makes you an easy target. One of the last things a middle-aged engineer in the US auto industry wants to be right now is laid-off.
We naively expect a senior engineer to do the ethically correct thing without respect to whether it has negative personal consequences. The problem with this idea is that the consequences for being "that guy" are too great; and both engineering societies and state licensing boards have been derelict in their roles of helping to protect engineers who refuse to do the ethically gray/wrong.
>I interpret it as the deadline being a root cause. At the heart of any complex issue lies one or more root cause(s).
There are so many things wrong with product engineering management, but yeah laying a deadline that no engineer has the courage to break would definitely be one of them.
It doesn't sound like such a horrible position to be in; working for a multi-billion dollar company that was trying to force you to ship a part that could kill people. Whistleblowers get 1/3rd of damages, it's like wining the lottery for ethical people.
How long does it normally take to design one of these switches, when you've been doing it most of your life? Knowing absolutely nothing about the complexity of these switches, but knowing what small hardware teams have been able to accomplish at companies I've worked at, 3 years sounds like a more than reasonable time frame.
>It doesn't sound like such a horrible position to be in; working for a multi-billion dollar company that was trying to force you to ship a part that could kill people.
It's probably not a case of some moustache twisting monocle wearing fiend who was the manager and the engineer was absolutely convinced that it was a faulty part that would ultimately be responsible for death and destruction. It was more likely one of a number of similar sub-assemblies, all having some questionable attributes and people trying to ship things under the guise of "perfect is the enemy of the good". At some points along this road, various people probably began realizing that there was a real problem, one that they might have some culpability in. Why the people involved continued to make poor decisions, I don't know.
>Whistleblowers get 1/3rd of damages, it's like wining the lottery for ethical people.
It's a nice theory. I've yet to see it happen consistently in practice (or pretty much ever). Engineering ethics case studies are littered with cases where engineers tried to alert management of problems and were punished for their efforts.
>How long does it normally take to design one of these switches, when you've been doing it most of your life?
Not very long given ample resources and nothing else to do. I doubt they were working under those conditions.
Engineering is the art of technical and economic tradeoffs. Engineers use their judgement, experienced engineers simply have a larger pool of good and bad judgements to draw on. They still get things wrong for the same reasons Uncle Bob has to debug the code he writes, they're human and make mistakes.
The domain of automobile use encompasses more variables than a website - there's no sleet over TCP/IP and broken internationalization doesn't result in crushed vertebrae. The vast majority of software engineers and programmers live in a world where the calculus for balancing technical/economic tradeoffs don't involve life safety and billions of dollars of capital investment in the pans.
You're taking four words from an article (not original sources) and deducing a root cause. In the absence of those four words -- which is the only time 'deadline' is mentioned -- would you draw the same conclusion? I do not.
I could argue that fudging the records is a 'root cause' of why the investigation took longer. If you want to go reductio ad absurdum then really the root cause is ultimately human fallibility.
> then really the root cause is ultimately human fallibility.
I'm not sure why you see this as 'absurdity', because it is a problem and it is something we try to design around with everything from good UIs to fatigue management.
I see it as absurd because when you get to that level, there's no information about what you can practically do to fix it. You've gone way past the threshold of usefulness — if you get my meaning (in other words, this reductionist view could end with "because, physics", but that's not very helpful).
You don't attempt to fix human fallibility itself but instead create systems and process to reduce both it's likelihood and impact (as you describe) for specific scenarios. Those systems and processes can then be improved and new scenarios added.
In this particular case, I find the reduction to 'management and hard deadlines' to be a useless summary (as well as incorrect) as it goes beyond the threshold.
> I see it as absurd because when you get to that level, there's no information about what you can practically do to fix it.
I disagree. I see it as important to recognize our shortcomings and find successful ways to deal with them. It may be that we haven't yet figured this out in general, but figuring out specific cases may allow us to solve individual pieces of the problem one at a time and improve things even if we cannot fully overcome our flaws.
> Changing a part, without also changing it's part-number, doesn't sound like a normal thing to do (and then forget about).
This really depends. If the original part was not being made to spec and the "new" part was to spec, the specs never changed and therefore the part number shouldn't change. They would just identify that parts with serial numbers or stamp dates up to X were not made to spec or defective.
Did you read the email from the chief switch engineer about the 'switch from hell'? Seems like this guy seriously fucked up and then tried to cover it up and then continued to lie about it at depositions?
Management is absolutely also to blame but it all started with the engineer. His negligence and 'fuck this switch' attitude killed people. He failed technically and ethically to do his job and people died.
As reported in [1], it looks like he wasn't negligent - "DeGiorgio discussed the problem with Delphi...in February 2002, he had a choice: do nothing to fix it or change the switch and delay production."
He certainly knew about the problem, but then decided to not fix it, to keep the production schedule on track. This is what makes me talk about management and deadlines.
It's easy to blame it all on that single engineer - and I'm sure GM would love to do so - but isn't it possible the environment in which he worked made it difficult - even risky - to implement the right solution? Perhaps biased by some of my own experiences, I tend to favor the organisational failure explanation that other comments here describe quite well.
Of course, everything he did later - the cover up, the lies - were attempts to cover up this one bad decision, and show that his ethics were less than stellar.
I guess I see it differently. Management definitely failed to react properly once it became obvious the switch was faulty. I mean, apparently it came up in car reviews of all places.
However, if you're the lead engineer on the switch, you have 23 years of experience on the job, it's your responsibility not to say "fuck it, just ship it." It's his signature on the paperwork. This wasn't a one-time one-day bad decision. This is a switch he worked on for three years.
The right course of action is refuse to sign off on a faulty part, and then not try to conceal that the part was faulty once people start dying. Every project has schedule pressure, so if schedule pressure is somehow a defense then engineers are never responsible. I simply don't buy that. Management can decide they want to override the engineer and ship it anyway. GM can fire the guy for not agreeing to ship a faulty part, but then DeGiorgio could have been a whistle blower collecting 1/3rd of the damages against GM right now.
I don't blame it singularly on the engineer. TFA says that GM’s Director of Product Investigations shut off the car with her knee, so this goes way beyond just the engineer. But I don't think that lets the engineer off the hook either. The engineer failed spectacularly, possibly criminally, and I hope he has to face a jury of his peers for this. Management also failed the same way, and if the investigators were in place to really push this, I'm sure the paper trail is there to at least indite some of these greedy complacent fools.
"However, if you're the lead engineer on the switch, you have 23 years of experience on the job, it's your responsibility not to say "fuck it, just ship it." It's his signature on the paperwork. This wasn't a one-time one-day bad decision. This is a switch he worked on for three years."
I think you're still not getting the point. The way modern management works, and this seems to be nearly universal now, is that it leaves the professional still formally in charge and still liable but creates a situation where there is massive pressure to follow the schedule.
GM can fire the guy for not agreeing to ship a faulty part, but then DeGiorgio could have been a whistle blower collecting 1/3rd of the damages against GM right now.
Oh, you can't collect damages for fatal decision you prevented and then were fired for. Even more, the point bad engineering decisions is not that they are not guaranteed fatal but simply that they might be. If you dig your heals to prevent them, you get fired - the bad decision gets made but turns out not to be fatal but sequence of events is still fatal your career. Even if you warn about each one, you'll get fired 'cause management will see and won't want you covering your ass at every turn. They want you, silently making the decisions on the impossible schedule they set, to be the device they use to cover their asses instead. That is the point of the whole structure.
Of course there's pressure to follow the schedule. There's also pressure not to kill people with sloppy work. Did you miss the part about fudging the part number, the lying, the cover-up?
I don't understand your second point. Keep in mind that he could have signed off on the faulty part initially, and only after the first reports of a safety issue started trickling in he could have raised holy hell with management and if they did nothing then he could have become a whistleblower. 20% of the problem was initially failing to design a proper switch, but 80% of the problem was how DeGiorgio and GM overall behaved afterwards.
The point is there is a strong regulatory framework which lets you fully and properly fuck your managers and live out a very happy life if they actually try to force you to cover up a mistake like this, but only if you have the ethics to do the right thing. Instead DeGiorgio allegedly spearheaded the cover-up personally.
I've worked at a hardware startup verifying specifications and watched the level of process around BOM changes (it's off the charts compared to software change tracking). I'm really surprised it was even physically possible for DeGiorgio to make the hardware change without changing the part number.
There was one time in my career an engineer fabricated test results because the schedule didn't allow enough time to properly run them. I discovered the falsified results and discovered it, and the engineer was ultimately fired. The next guy in the position worked within the system to properly communicate how long the tests took to run, and the result was that some less critical testing was skipped, and the rest got the appropriate resources to complete them. But you do not get to just lie and say everything was done just because the schedule was unrealistic. Schedules are always unrealistic.
Doesn't matter. As an engineer, you don't get to pass the blame like that. When you sign off on something, you are taking responsibility for it. Being able to tell your employer 'no' is fundamental requirement of being a professional engineer. It's the most important difference between a professional engineer and 'some guy who designs things'.
If management was acting irresponsibly, they deserve blame as well, but that does not excuse his actions. My engineering association probably has somewhat different rules than his, but they're all going to be similar. From the handbook sitting beside my desk:
-------------------------------------------------
Rules of Conduct:
1. Professional engineers[...] shall, in areas of practice, hold paramount the health, safety and welfare of the public, and have regard for the environment.
....
A client's or employer's interests should be held in high regard. However, the following duties take precedence over the interests of the professionals' client or employer:
- the duty to protect the safety of the Public;
- the duty to the professions under the Code of Ethics; and
- the duty to act fairly and justly to all parties when administering a contract on behalf of a client or employer.
....
Having a Recommendation Overruled:
- When a client or employer makes a decision that adversely affects the public interest and is contrary to the recommendation of the professional, the latter should inform the client or employer of the consequences of the decision. If the client or employer is unavailable or unresponsive, the professional should notify the appropriate regulatory authorities that have the ability to evaluate the concerns and the power to suspend activities until the technical issue is resolved.
> the professional should notify the appropriate regulatory authorities
Sounds like a great way to lose your job. Don't get me wrong, but it could very well be a very costly thing to do, and not all people in management likes that.
It's interesting - and encouraging - to see such things codified as a handbook. Out of curiosity, what industry are you in? (and you need not reply if you wish to maintain privacy)
I should mention that it's not just something written down in a handbook, either. Accredited engineering programs are effectively required to contain an engineering ethics course, the purpose of which is to explain what the rules are and why they exist. Mostly by looking at incidents like this one.
EDIT: Thanks, krallja. It seems that US national ethics guidelines are indeed very similar.
> Management is absolutely also to blame but it all started with the engineer.
No. Whatever the engineer's responsibility, the engineer's management decided to keep him, making it their responsibility.
"A loss of X dollars is always the responsibility of an executive whose financial responsibility exceeds X dollars."
- Gerald Weinberg's 'First Principle of Financial Management' and 'Second Rule of Failure Prevention' [1]
[1] 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg, Dorset House Publishing, 1993
No. Actually, professional engineers have a legal responsibility to do their work correctly and can be held personally accountable for failures to do so. This problem was not a failure of a management.
From what I understand, this is exactly what happened. A few people were just trying to cover their own asses and it resulted in tragedy.
It's ridiculously unfair to all the people who died and families who lost someone. It's also unfair to the engineers and management at GM who are ethical and good at their jobs.
It's an understandable byproduct of a stack ranked organization. Any time you have the threat of being ranked and rewarded based on delivery well anything that threatens the delivery reflects badly on the rankee and is actively avoided.
The threat of being ranked? We are ranked constantly starting at 1st grade. Competitive ranking should be a very familiar feeling for every American at least.
When humans face the prospect of being ranked, the only option then becomes to lie, cheat, take shortcuts, or put lives at risk?! It may be an understandable byproduct for the weak, the proud, or the immature. Cheating is certainly endemic in our society, but I don't believe that absolves the cheaters; it's still criminal/unethical and it should still get you fired, throw off the team, or sent to jail.
There is something important to study and learn from this disaster, but my personal opinion is it's not so much a problem with performance evaluations, but rather a spectacular engineering failure combined with a company rotten to the core that completely lost its sense of purpose (hint: serving customers, not short term profit).
I do expect that adult humans can be subjected to significant workplace stress and pressure and still be expected to behave ethically and responsibly. I think that the law and justice expect the same.
Getting a deadline moved even for logical reasons involves pushing against massive internal red tape. Missing a deadline invites greater punishment from management than compromising on quality. Naturally, quality loses out.
Happens all the time in our own industry, and sadly, this case shows that it happens even in safety critical industries.
All engineers, regardless of industry, should be trained in how to negotiate with management. All managers, regardless of industry, should be trained to respect opinions of subordinates.