In the beginning was the plan.
And then came the assumptions.
And the assumptions were without form.
And the plan was without substance.
And darkness was upon the face of the Engineers.
And they spoke among themselves saying,
"It is a crock of shit and it stinketh."
And the Engineers went unto their supervisors and said,
"It is a pail of dung and none may abide the odor thereof."
And the supervisor went unto their managers and said,
"It is a container of excrement and it is very strong, such that none may abide by it."
And the managers went unto their directors, saying,
"It is a vessel of fertilizer, and none may abide its strength."
And the directors spoke among themselves saying to one another
"It contains that which aids plant growth and it is very strong."
And the directors spoke among themselves, saying to one another
"It contains that which aids plant growth and it is very strong."
And the directors went unto the vice presidents, saying unto them,
"It promotes growth and is very powerful."
And the vice presidents went unto the president, saying unto him,
"The new plan will promote the growth and vigor of the company, with powerful effects."
And the president looked upon the plan and saw that it was good.
When I was a newly minted engineer, this was one of the things passed around (as photocopies - kids, ask your parents). I was pretty green, and so thought it was breathlessly cynical. I had a lot to learn about how information deteriorates through an organization structure.
It looks like this happened at Boeing:
It looks like Boeing itself is in panic mode here. Every move they make to try to get out in front of this seems to be digging themselves into a deeper hole. Err, or deeper into their vessel of fertilizer.
But self-drivign cars will be totally safe. This sort of thing would never happen there. Nosiree.
It looks like this happened at Boeing:
Boeing engineers knew about the problem in 2017 – months before the fatal Lion Air and Ethiopian Airways crashes. The company only revealed this to US Federal Aviation Authority regulators after Lion Air flight JT610 crashed in October 2018, claiming in this week's statement that "the issue did not adversely impact airplane safety or operation".
"Senior company leadership was not involved in the review and first became aware of this issue in the aftermath of the Lion Air accident," added Boeing.Reading between the lines, it seems pretty clear that Boeing expects major lawsuits, and is preparing to try to throw their software vendor under the bus. I expect that this won't work - after all, it's really saying that "hey, we really don't know what our vendors are up to" - and quite frankly is shouldn't work. If Boeing's lawyers are successful and their software vendor gets sued into bankruptcy then Boeing has a whole bunch of critical software without a supplier to do changes and maintenance on it.
It looks like Boeing itself is in panic mode here. Every move they make to try to get out in front of this seems to be digging themselves into a deeper hole. Err, or deeper into their vessel of fertilizer.
But self-drivign cars will be totally safe. This sort of thing would never happen there. Nosiree.
Sadly true, and the lack of PROPER training hasn't helped...
ReplyDeleteSteve Savitzky turned it into a song, many years past: https://www.youtube.com/watch?v=qGzUTrnqEDA
ReplyDeleteReading between the lines, it seems pretty clear that Boeing expects major lawsuits, and is preparing to try to throw their software vendor under the bus.
ReplyDeleteFirst, Boeing is always expecting lawsuits. Being the deepest pockets associated with a crash means it comes with the territory. Second, Boeing does all its software in compliance with RTCA DO-178 and its own internal processes. That gives them some ability to say, "we did everything right" at least to some degree. The blame then spreads in some amount to the FAA, which means Fed.Gov.
Boeing holds all of its suppliers to those specs and goes after vendors zealously.
My rough rule of thumb was that certifying a box for Boeing, hardware and software, took longer than a generation of a consumer electronics product. After all the design, all the reviews, far more Process than the commercial world ever sees, the qualification testing alone would take a year. There could be two new generations of iPhones in that time.
And that's the thing about self-driving cars that's really scary. As messed up as the 737Max seems to be, the self-driving car business doesn't even seem to have that level of software process.
Software QA. It's really a thing.
Thing about having good Process... it guarantees that a defective spec will be implemented to perfection.
ReplyDeleteI keep wondering why nobody seems to have questioned the functional spec for MCAS. Like: "Hey, if we're trying to correct for an uncommanded pitch up, shouldn't we be looking at the pitch gyro?" Or, perhaps, "If we have two AOA inputs, they disagree, and one of them is behaving in ways inconsistent with the wings remaining attached, shouldn't we believe the other one?"
Eric, that song is awesome. And this does indeed smell of a broken spec - although the broken communication flow (if you believe the story here) suggests maybe a broken process. Or both.
ReplyDeleteGraybeard, I agree on the certification testing bit. Bing in an industry where "5 9s" is a fundamental criterion, you do NOT just "ship" the latest bits. This is actually what's scariest about Tesla's approach - they really think they will just "update the software over the air" and everything will be fine.