Complexity kills

In this case, it killed all 228 people aboard Air France flight 447 a couple years back.  Popular Mechanics has a detailed analysis of what we know from the recovered Black Box recordings, and it makes for fascinating reading.  Here are the parts most interesting to me:

Just then an alarm sounds for 2.2 seconds, indicating that the autopilot is disconnecting. The cause is the fact that the plane's pitot tubes, externally mounted sensors that determine air speed, have iced over, so the human pilots will now have to fly the plane by hand.

Note, however, that the plane has suffered no mechanical malfunction. Aside from the loss of airspeed indication, everything is working fine. Otelli reports that many airline pilots (and, indeed, he himself) subsequently flew a simulation of the flight from this point and were able to do so without any trouble. But neither Bonin nor Roberts has ever received training in how to deal with an unreliable airspeed indicator at cruise altitude, or in flying the airplane by hand under such conditions.

...

Almost as soon as Bonin pulls up into a climb, the plane's computer reacts. A warning chime alerts the cockpit to the fact that they are leaving their programmed altitude. Then the stall warning sounds. This is a synthesized human voice that repeatedly calls out, "Stall!" in English, followed by a loud and intentionally annoying sound called a "cricket." A stall is a potentially dangerous situation that can result from flying too slowly. At a critical speed, a wing suddenly becomes much less effective at generating lift, and a plane can plunge precipitously. All pilots are trained to push the controls forward when they're at risk of a stall so the plane will dive and gain speed.

The Airbus's stall alarm is designed to be impossible to ignore. Yet for the duration of the flight, none of the pilots will mention it, or acknowledge the possibility that the plane has indeed stalled—even though the word "Stall!" will blare through the cockpit 75 times. Throughout, Bonin will keep pulling back on the stick, the exact opposite of what he must do to recover from the stall.

What makes this particularly interesting is that the Airbus has a history of pilot/auto-pilot confusion.  An early crash (don't have the book here, sorry, but it was in the early 1990s) happened because the autopilot overrode the pilot's controls, preventing the plane from recovering from a stall and ultimately crashing it.  Here, the auto-pilot didn't override the pilot's controls, allowing him to basically bring the speed of the plane to zero.

The moral, from a software engineering perspective, is that there probably isn't any right way to design a system that will always work.  What is absolutely critical is to have obvious and transparent communication of critical information that the pilot needs.

Airbus has a spotty record on this.  Another crash (this one in Greece in the 1990s) was due to confusion about rate of descent vs. altitude, and the computer being incorrectly set.  That plane flew into a mountain.

Now, it's easy for me to say this, after all, I'm not a pilot and don't really know all the possible inputs that might lead to a crash.  However, there is a long history of strange "pilot error" in Airbus planes, and I quite frankly don't think that Air France hires incompetent pilots.  I think that the system complexity of their "Glass Cockpit" has exceeded the pilot's ability to understand what's going on, and in particular what is happening in emergency situations, and what the computer will do and what it won't do.