Oh, please folks, temper your enthusiasm to hang these guys. For all we know it might very well be grotesquely misplaced.
As I stated in another thread, I'd prefer to read commentary from someone who has actually had first-level access to the relevant design and test documentation. Everything else is nothing more than conjecture.
What are some of the failure modes of battery systems, including Li-based systems?
OK, here's an incomplete list that has nothing whatsoever to do with the 787 case because, well, no design and test data has been released. Again, this has nothing to do with the 787. Think of it as the start of a list of things to watch out for if you were going to set out to design a battery pack and related systems.
Sourcing
- Latent defects due to manufacturing QA issues
- With Li-based cells such things as humidity content can
be critical as they might lead to the generation of methane
gas during the service life of the cell.
Assembly
- Handling and storage problems
- Mechanical damage, from dropping a cell to any number of
imaginable issues.
- Assembly defects that can lead to (I**2)*R heating issues
Testing (both at by supplier and OEM)
- Failure to detect defects (sometimes impossible if internal)
- A faulty testing process that actually causes or accelerates damage
to cells, perhaps only cells that have specific problems
- Inadequate issue escalation workflow
Electrical
- (I**2)*R heating of faulty interconnects (inter-cell and elsewhere)
- Component defects that can lead to (I**2)*R heating issues
- Oxide formation that can lead to (I**2)*R heating issues
- Interconnect crimp quality problems that can lead to (I**2)*R
heating issues
- Interconnect crimp quality problems that can lead to other issues
- Main pack electronics problems
This can include chargers and monitoring electronics
- Charge termination failure or inadequacy
- Thermal sensing failure or inadequacy
- Pressure sensing failure or inadequacy
- Fluid sensing failure or inadequacy
- Inadequate semiconductor rating (for example: operating temperature)
- Inadequate power semiconductor heat dissipation
This doesn't necessarily mean bath thermal design.
It could be caused by inadequate or faulty mounting of TIL
(thermal interface materials).
It could also be caused by mechanical and environmental issues
affecting the TIL or the mating of the semiconductor to the heat
dissipation mechanism. For example, vibration.
- Inadequate or faulty failover modes
- Static damage to semiconductors during electronics assembly, testing
or general handing
- PCB via or trace fractures, contamination or over-etching
- BGA ball soldering problems
(mostly only visible via x-ray inspection)
- Bad, cold, fractured or chemically compromised solder joints
- Dendrite buildup in lead-free solder chemistry leading to shorts
in small pitch components
- Unaccounted race conditions
Example: pull-pull technologies without adequate dead-band
- Thermally-induced failures due to changes in operating
characteristics of components as temperature fluctuates.
Example: The aforementioned push-pull topology where the deadband
might have been adequate within a certain temperature range but
causing shot-through shorts outside of this range.
- Unaccounted for effects of such environmental issues as humidity.
Example: Many VCO chips do not deal with humidity very well,
particularly if it can condensate.
- Faulty or non-existing conformal coating
- Inadequate static damage counter measures
(inputs, anything expose to handling)
- Sub-pack electronics problems
This is applicable if a large pack is made from a multitude of
smaller packs, each with it's own control and monitoring electronics.
Sub-pack electronics can suffer from the same issues as the main pack
electronics above.
Software
- Myriad of code-based issues
(lock-ups, endless loops, pointer problems, uncaught exceptions, etc.)
- Software problems caused by hardware issues
Example: Static damage to a chip causing a bit to unexpectedly flip.
If not caught in software this can lead to catastrophic failure
Mechanical
- Faulty inspection and certification process
- Inadequate design materials
- Adequate materials but supplier quality issues resulted in unexpected
materials issues
- Inadequate design tolerances
- Adequate design tolerances but faulty assembly, subcontractor or testing
process caused them to be violated
- Faulty fasteners
- Faulty fastener application
- Unexpected interference between components
- Mechanical pinching or chafing of wires
- Mechanical puncture damage to cells during installation or operation
- Mechanical abrasion to protective cell wall or packaging
- Insufficient mechanical rigidity
- Insufficient venting mechanism
- Insufficient containment or venting of gasses (methane),
flames or heat
- Mechanical issues once pack is installed
- Thermal
- Mechanical (pinching, squeezing, twisting, cutting)
- Resonance
Thermal
- Inadequate thermal management
- Lack of consideration of thermal runaway problem
- Lack of thermal isolation between individual cells or sub-packs
- Insufficient thermal sensing
- Insufficient thermal overload coping mechanism
(both mechanical and electronic)
- A design that might cause severe differential heating of cells deep
inside the pack
- Inadequate electronic and mechanical fail-over measures in the case
of thermal runaway
- Inadequate thermal runaway detection
- Cell supplier issues (quality control, testing, etc.)
- Thermal issues once installed
- Problematic heat exchange
- Heat input from adjoining assemblies
- Accidental plugging of filters, inlets, etc.
Final release
- Electrical testing
- Mechanical testing
- Environmental testing
- Random destructive testing at the cell level
Operational
- User fails to follow published procedures
- Alarms are ignored
- Mechanical issues are ignored
- Inadequate maintenance
- Handling issues
Anyhow, this is not, by a long-shot, an exhaustive list. There's a lot I am ignorant about and, of course, that stuff is missing. Feel free to expand.
The point is that a proper investigation would use --at a bare minimum-- something like this and, more than likely, a book full of specs and procedures before reaching a verifiable and reproducible conclusion.
Maybe it's just me. I don't know. I just don't like it when people do some of what is being done to Boeing. Particularly when they don't have access to the reams of FAA, Boeing and airline data that is required to really understand the problem.
From your comment history it sure seems like you have a vested interest in maintaining favorable PR for Boeing in this forum.
Why shouldn't the industry discuss this in open terms? The fact is we don't have access "to the relevant design and test documentation". Boeing and their partners won't share it. So, how can we discuss this without the data, then?
Conjecture and informed opinion is all we have, and it seems as if you have rather a bit of the latter .. so why not share a technical analysis of the circumstances based on your understanding rather than attempt an appeal at what could be construed, frankly, as a PR move?
What if, Boeing are simply mis-managing the project? It surely is not the first - nor the last - time that a major aerospace company was bit by that bug - in fact, for anyone that pays attention to the big, fat, military-industrial corn-fed cows of the last few decades, its rather the norm than the exception!!
Nothing says "management really, really screwed up" than a smoking pile of brand new tech, being doused with fire-retardant foam, while the world asks questions like "why is our technology failing us?" from afar .. lucky, we're not there yet with the 787 - so this is really the best time for us to be discussing these issues. Lucky no lives are lost .. yet.
Nope. Never worked for Boeing. Never worked directly in aerospace. I did manufacture motor and battery controls that were used in drones and robots maybe twenty years ago mostly in academia. I have no way to know if the motor controls we sold to institutions such as MIT ended-up in Boeing product. I doubt it.
It's easy. Go to my profile on HN. Send me an email. I'll reply back with my LinkedIn profile. Then you can go figure out if I have anything to gain from my comments.
I'm just an engineer with a lot of experience making mistakes who hates it when people run their mouths without data. That's all. Things at this scale are never as simple as some journalist might make it seem. Most people don't have a clue as to how such massive machines as a 787 are designed, built and tested. Not a clue.
Many years ago I had a chance to meet and enjoy long talks with the engineer who had been in charge of the F18 project. It was a fascinating learning experience for me. If I remember correctly, he said he had some 3,000 engineers working for him. Now, how many people would imagine that 3,000 engineers had their hand on the F18 design simply by looking at it? Better yet, how many journalist? And, how many of these people could diagnose an F18 problem --even if qualified-- without access to real engineering and test data?
> Conjecture and informed opinion is all we have, and it seems as if you have rather a bit of the latter .. so why not share a technical analysis of the circumstances based on your understanding rather than attempt an appeal at what could be construed, frankly, as a PR move?
Crap, I didn't catch that one.
Really? That's how you see what I wrote? Amazing.
Here you have an engineer (me) saying: "We don't know enough. We need facts before we can reach a conclusion. Please stop."
And your interpretation is that I am using conjecture for PR purposes?
Are you an engineer? If you are not, maybe you just don't understand my vantage point. And that's OK. Life goes on.
Here's an analogy, imperfect, but, what the hell:
A guy gets arrested and accused of rape.
You say: "He probably did it. Hang him!".
I say: "Let's wait and look at the evidence, which should ideally include DNA. Then we can look at the facts as they may be available to us and make a decision."
These are vastly different approaches to the problem.
Who would you want in the jury if YOU were the accused? Someone like me or someone like these clowns that are flapping their jaws without having access to the evidence?
Isn't that the truth! It has happened to me a few times. I actually wanted to serve too. Nope. The minute I revealed what I did: business owner, engineer, software, etc. Done. Excused.
If we're going to report anecdotes, two (aerospace) engineer friends of mine (one of whom professed this purported fact) at the same company got picked for a jury on the same drunk-driving case.
I think it depends on which attorney gets a turn when they question you. I don't remember exactly, but don't they take turns? I would think that, depending on the nature of the case and the evidence, one or the other attorney might prefer to inject jurors into the jury who might tend to take a more analytical approach to things.
Not much to add, except that you're not giving yourself enough credit. You didn't just say, "Let's wait for the evidence, like X." You said, "There are a lot of factors that could account for the phenomenon we've observed, some of which would implicate the accused. Here's an extremely thorough list ..."
Look, since none of us actually have the data, what you're asking is that we just simply stop discussing the issues that Boeing is having. What can we do but discuss it, at this point - no proper analysis can be made, only one based on conjecture and experience.
Isn't it clear by this point that Boeing have a problem? Its either a) an engineering problem, b) a management problem, or c) a combination of both.
Its not unreasonable to expect that a company with as much money, and - frankly - unwarranted guaranteed income from the people of the United States of America - to get its act together - and certainly, the means by which it gets it act together is of high interest to the average denizen of hacker news ..
So, why call for people to 'not discuss it unless they are experts', then? There are many lessons to be learned here, and we can certainly use our intelligence - the intelligence of the whole - to make a social comment on the issues revolving around Boeing, and in fact the entire military-industrial machine. It has its frailties, obviously - and we are seeing them - so isn't this of High Interest to the hacker news community?
Calling for an 'authority' to comment on this issue is hardly a way to gain more insight into the problems of technology in this industry. The 'authority' has already been consulted; and clearly, somewhere along the line, screwed things up completely. The 787 grounding is costing Boeing, its partners, and the American people - a lot of money.
Discussing why this happens, how it happens, and what can be done about it, is a very topical use of hacker news, in the first place: so why call for silence-unless-authorized unless there is some desire to reduce the noise from the rabble? The rabble aren't doing any harm; whereas the likelihood that the management personnel of Boeing are doing harm. Only by discussing this can we gain any real insight into just how that can occur.
Clearly, the engineering is at fault: there is a fault. Is this management, is it engineering, or is it just plain hard science involved - unless we discuss this situation, openly and with a free will to apply to the investigation, there is nothing much to be gained from the incident in a public setting.
I certainly hope they get the problem sorted. The means by which they do that, is of high interest to me. I hope to fly on a 787 one day, and know that my safety is a result of real solutions and not a faulty process being forced on engineering by management under constraint of financial market performance duress..
>>Here you have an engineer (me) saying: "We don't know enough. We need facts before we can reach a conclusion. Please stop."
Here you have an entrepreneur saying: "We don't know enough. We need to know more. No conclusions are being made, but the process of discussion sure is interesting - it reveals a lot about human nature when billion-dollar budgets - and human lives - are at risk. Please continue to discuss this issue as a community."
I can't say much more than I already have because I don't have any real data to work with. None. You and others are, of course, completely free to visualize all manner of scenarios based on no direct evidence whatsoever. And, again, that's OK. Perhaps a disclaimer might be nice?
I also look at this from another angle. As an American (US Citizen, whatever sounds better to you) I am proud of the accomplishments of companies like Boeing and, yes, SpaceX and others. These are the companies I point out to my kids as examples of what smart people can achieve. Put bluntly, I don't like it when I see a feeding frenzy that could destroy their reputation, cost jobs and do other damage and said frenzy is based on not much more than pure unfiltered bullshit.
Do they have an engineering problem?
Maybe.
How about a manufacturing problem?
Maybe.
How about a sourcing problem?
Maybe.
Testing?
Maybe.
Sabotage? Terrorism?
Maybe.
Negligent operators (airlines)?
Maybe.
I could go on.
All answers which originate from a lack of evidence can, at best, be "maybe's" and at worst complete fabrications.
Now, you could take the time to go through the list I presented on this thread and expand it with research into each of these areas and even expand my list. That could be an interesting discussion of potential failure mode of such systems. This CAN BE DONE without making categorical statements as to what "actually" happened to the 787 battery pack. And, yes, that would be an interesting and useful discussion. So, there you go. Take one item and fly with it. Let's see where it leads.
It seems to me that it is very clear that there are multiple problems, that the information necessary to identify the underlying causes is not available to the general public, and that Boeing, partners, and USG are working on getting this resolved. It seems interesting but not particularly useful to speculate.
An interesting but useful line of questioning would be to look into industries we know and our past companies for what happened and what lessons can be learned in places where we have more complete information.
That said, the topic at hand :)
One question is whether the battery failures represent an unacceptable safety risk. Without knowing anything about the technical specifics you can figure the answer is yes: Boeing is the home town team and the FAA has every political interest in keeping Boeing planes in the air. Still, their overriding concern is safety and they've grounded the 787 in what must have been an unpleasant choice but one viewed as necessary.
Another question is whether there is a management failure at Boeing. Management is ultimately responsible for outcomes so whether this incident is out of line with the industry norm is one reasonable way to answer that question. In this case looking at data on whether grounding of a new model is unusual would be reasonable. NTSB or FAA probably have this in a report somewhere, my guess is it's rare (the only other similar thing I can think of is the A380 wing cracks). So my guess is yes, by a reasonable standard there was a management failure.
Those two things seem obvious and not particularly interesting to discuss. Some things that are not obvious:
What's wrong with the batteries? robomartin's list of potential causes is long and educational but (as he points out) incomplete. This is an interesting problem for which we don't have enough information to reach any kind of strong conclusion. Some really smart people who are also domain experts and who have all of the data will sort this one out. Best read the news, at some point there will be a weighty public report detailing exactly what went wrong.
What's wrong with Boeing? I'm sure this will be studied for decades to come, I don't know and I'm pretty sure the author doesn't know. The idea that markets start out favoring highly integrated designs and trend towards standardization and modularity is well accepted and probably correct but at this point he's just fitting recent events into that framework rather than finding ways to test that hypothesis.
I couldn't agree more with your post and your approach to discussing the problem. Yes, unless we are talking about something like and attempt to sabotage or worst, terrorism --both of which I would like to believe are highly unlikely-- ultimately management and engineering have to look back and see where the process dropped the ball. We are only human. We make mistakes. As you said, actual data will bubble-up to the surface eventually. From an engineering standpoint (and as a passenger!!!) I'd love to understand what happened.
A while ago I enjoyed reading "The Machine That Changed the World":
It was interesting to learn about the evolution of manufacturing and management practices that made a huge impact on the automobile industry. Lots of lessons from that book can be applied well outside that industry.
Thanks, that looks like a great recommendation. I spent some time at IMVU (online service + software) a few years back and their processes were heavily inspired by the Toyota Production System. The stuff they adopted seemed to work really well so lean manufacturing has been interesting to me since.
That is not always and engineering problem. It can be a manufacturing problem. One case[1] of this is the F-86. It was engineered fine, but a worker on the floor didn't follow instructions and installed a bolt incorrectly. This caused the plane's aileron to lock during certain maneuvers killing some pilots. This was a manufacturing problem not an engineering problem.
One is that I'm actually thankful for robomartin's comments; they break up some of the more outlandish hero-worshipping of Elon Musk (i.e., https://hackernews.hn/item?id=5138430, which is frankly just a ridiculous thing to say).
Another is that this is not new. What is new about the 787 is the sheer amount of media attention, not the fact that there are serious post-service-entry problems. For example:
Nothing says "management really, really screwed up" than a smoking pile of brand new tech, being doused with fire-retardant foam, while the world asks questions like "why is our technology failing us?" from afar
You mean, the way we all went crazy, posted huge numbers of threads and begged people who don't work on airliners for a living to come save Airbus in one fell swoop after the A380 exploding-engine accident (which, on investigation, turned out to not be a one-off problem, but a general flaw with the plane's engines)? The one where a smoking pile of brand-new tech had to be doused with fire-retardant foam?
This stuff happens. The other thing that's different about it now is that when it happens, people don't die by the hundreds, as they used to when a new airliner entered service and a problem suddenly became visible. What is not different is that it still gets solved by research and hard work, rather than by pop-entrepreneurs snarking away in major newspapers.
Actually that's inaccurate... The A380 has 2 engine types and the issue only affected one of the 2 types, specifically the Rolls Royce engine. The other problem is that the issue was not a general flaw but some of the oil valves in the engine were not cut properly and would eventually break.
A better example of how this stuff hits the A380 are the cracks in the wings.
There have been a lot more problems with the 787 than the batteries, as the article describes. And you didn't address the central thesis of the article, which is that the root of the problems lies in how the project was managed.
As I stated in another thread, I'd prefer to read commentary from someone who has actually had first-level access to the relevant design and test documentation. Everything else is nothing more than conjecture.
What are some of the failure modes of battery systems, including Li-based systems?
OK, here's an incomplete list that has nothing whatsoever to do with the 787 case because, well, no design and test data has been released. Again, this has nothing to do with the 787. Think of it as the start of a list of things to watch out for if you were going to set out to design a battery pack and related systems.
Anyhow, this is not, by a long-shot, an exhaustive list. There's a lot I am ignorant about and, of course, that stuff is missing. Feel free to expand.The point is that a proper investigation would use --at a bare minimum-- something like this and, more than likely, a book full of specs and procedures before reaching a verifiable and reproducible conclusion.
Maybe it's just me. I don't know. I just don't like it when people do some of what is being done to Boeing. Particularly when they don't have access to the reams of FAA, Boeing and airline data that is required to really understand the problem.
EDIT: Formatting